Week ending October 2nd, 2020

I don’t even know where to begin, what a crazy year we have had so far. Not having written a work journal since December, it does feel like there are so many things I could write about but also so many things I have forgotten already. Having worked on hundreds of projects, I feel I do have a high capacity for compartmentalization, and that means that once a project is finished and paid for I tend to not think about it again. Of course clients come back, sometimes years later, and I have to try and remember the details of their project as if it was yesterday. All that being said, I don’t plan on summarizing much of the work I have done this year, as I’ve had five ongoing retainer clients and eleven other projects to keep me busy.

In cuter news, a few weeks ago our family got a kitten, which is pretty exciting. Hercules likes to hang out in my office, keeping me company as I am coding away.

This week I want to dive into a project I was working on this month as a subcontract for an agency.

Salesforce and products and WordPress, oh my!

Last November an agency requested a quote for a way to pull product data down from Salesforce to a new site they were designing and developing for a client. The client had their product objects in Salesforce and they wanted Salesforce to be their “single source of truth”, and they did not want to manage the products through the WordPress admin. I came up with an estimated price for the project and didn’t hear about it again for 6 months, when they indicated that it seemed like it was going to get going soon. Eventually the contract was signed, and the agency created a timeline for my part of the development to fit into. This project was likely to take me around 6 full days of development, and for a single project that was not for a retainer client that is a pretty significant job, so scheduling ahead was crucial.

This custom plugin project had some key features the client (or I) wanted:

  • Automatic syncing of the product data on a regular basis
  • A way to see any errors and a history of the sync progress
  • Copying of the images from the remote server to the local WordPress
  • Optimizing for performance as much as possible to reduce server load and increase stability, within our budgeted time

The client provided some basic documentation about the API and what should be expected, but that was outdated and didn’t apply anymore. Thankfully we were able to connect to Salesforce (using desktop software like Paw or curl) and gather product data to start discussing the details or mapping. Mapping is how I would refer to connecting data from one source to another source, in this case specific Salesforce fields were going to be mapped to fields and taxonomies on a “product” custom post type in WordPress.

After confirming we had data we could get from the API, the next step was creating a basic plugin framework that displays a few pages for the plugin with API authentication settings, a log page, and an overview style page with key statistics. Next up was writing PHP code that would retrieve the data from the API and handle the storage of the information in WordPress. This part was pretty straightforward, as I have done similar work on multiple other projects.

Things get tricky

One of the first challenges I ran into (and ran into again later) was image handling. Copying images from the Salesforce servers was taking longer than expected, and with hundreds of products to import, the image transfers would need to happen over hours. I decided it was time to implement some fancier sync processing to make sure the processes wouldn’t time out or exhaust a server memory limit. At first, I was looking at WP Background Processing from the awesome folks at Delicious Brains, but upon further review, decided to go with the Action Scheduler. The Action Scheduler is what WooCommerce and WooCommerce Subscriptions use for queueing large blocks of jobs to be done. In our case we would get a list of products from the Salesforce API and create a job for each product to be synced (or skipped). Action Scheduler allows us a super easy way to manage the job queue and do fun things like query the remaining jobs and adjust the job processing speed (depending on available server resources). After the sync jobs are added to the queue, the site will check each product (based on a unique SKU) to see if it exists. If it doesn’t exist then the product and any associated images or taxonomy terms are generated. If the product exists we compare timestamps to see if it has changed since the last time we updated it on the WordPress site. Any product that hasn’t changed is skipped and the Action Scheduler goes to the next job.

After nailing down the mapping and performance enhancements, and improving the user interface of the plugin, I found another problem: the images from Salesforce have no file extension and WordPress was often complaining (and error-ing out) because it expects the images to have a valid extension and MIME type. The core WordPress functionality for “sideloading” an image would work on some of images but many had issues, so I had to make a slightly modified version of _wp_handle_upload (found in /wp-admin/includes/file.php) that was a little more flexible.

It is oh so satisfying to watch the products show up in WordPress as the sync is being processed. There is still a little more functionality to be added, as far as removing products that don’t show up in the Salesforce products list anymore, but I won’t implement that until we have signed off on the work done so far. I am really enjoying using the Action Scheduler library and hope I get to use it again soon, but if I don’t need to deal with copying images from remote servers again for a little while I would be very happy.

What’s next?

Between my newfound attempt to take up writing again and my ongoing work for retainer clients, I am keeping pretty busy. However, I have submitted quotes to a few fun projects that I look forward to taking on, so stay tuned to what problems I get to solve in the near future.

To read past work journals and learn more about the projects I enjoy, click here. If you are interested in working together or have questions about the work I do, contact me.

The complete Work Journal series:
1. Week ending January 25th, 2019
2. Week ending February 1st, 2019
3. Week ending February 8th, 2019
4. Week ending February 15th, 2019
5. Week ending February 22nd, 2019
6. Week ending March 1st, 2019
7. Week ending March 8th, 2019
8. Week ending March 15th, 2019
9. Week ending March 22nd, 2019
10. Week ending March 29nd, 2019
11. Week ending April 5th, 2019
12. Week ending April 12th, 2019
13. Week ending April 19th, 2019
14. Week ending August 9th, 2019
15. Week ending September 20th, 2019
16. Week ending September 27th, 2019
17. Week ending December 6th, 2019
18. Week ending October 2nd, 2020
19. Week ending April 2nd, 2021
20. Coding API integrations in Twilio Studio - Work Journal May 8, 2021
21. Trudging through a complex theme implementation - Work Journal October 29, 2021
22. Creating custom Duda widgets - Work Journal December 10, 2021