I don’t even know where to begin, what a crazy year we have had so far. Not having written a work journal since December, it does feel like there are so many things I could write about but also so many things I have forgotten already. Having worked on hundreds of projects, I feel I do have a high capacity for compartmentalization, and that means that once a project is finished and paid for I tend to not think about it again. Of course clients come back, sometimes years later, and I have to try and remember the details of their project as if it was yesterday. All that being said, I don’t plan on summarizing much of the work I have done this year, as I’ve had five ongoing retainer clients and eleven other projects to keep me busy.
In cuter news, a few weeks ago our family got a kitten, which is pretty exciting. Hercules likes to hang out in my office, keeping me company as I am coding away.
This week I want to dive into a project I was working on this month as a subcontract for an agency.
Salesforce and products and WordPress, oh my!
Last November an agency requested a quote for a way to pull product data down from Salesforce to a new site they were designing and developing for a client. The client had their product objects in Salesforce and they wanted Salesforce to be their “single source of truth”, and they did not want to manage the products through the WordPress admin. I came up with an estimated price for the project and didn’t hear about it again for 6 months, when they indicated that it seemed like it was going to get going soon. Eventually the contract was signed, and the agency created a timeline for my part of the development to fit into. This project was likely to take me around 6 full days of development, and for a single project that was not for a retainer client that is a pretty significant job, so scheduling ahead was crucial.
This custom plugin project had some key features the client (or I) wanted:
- Automatic syncing of the product data on a regular basis
- A way to see any errors and a history of the sync progress
- Copying of the images from the remote server to the local WordPress
- Optimizing for performance as much as possible to reduce server load and increase stability, within our budgeted time
The client provided some basic documentation about the API and what should be expected, but that was outdated and didn’t apply anymore. Thankfully we were able to connect to Salesforce (using desktop software like Paw or curl) and gather product data to start discussing the details or mapping. Mapping is how I would refer to connecting data from one source to another source, in this case specific Salesforce fields were going to be mapped to fields and taxonomies on a “product” custom post type in WordPress.
After confirming we had data we could get from the API, the next step was creating a basic plugin framework that displays a few pages for the plugin with API authentication settings, a log page, and an overview style page with key statistics. Next up was writing PHP code that would retrieve the data from the API and handle the storage of the information in WordPress. This part was pretty straightforward, as I have done similar work on multiple other projects.
Things get tricky
One of the first challenges I ran into (and ran into again later) was image handling. Copying images from the Salesforce servers was taking longer than expected, and with hundreds of products to import, the image transfers would need to happen over hours. I decided it was time to implement some fancier sync processing to make sure the processes wouldn’t time out or exhaust a server memory limit. At first, I was looking at WP Background Processing from the awesome folks at Delicious Brains, but upon further review, decided to go with the Action Scheduler. The Action Scheduler is what WooCommerce and WooCommerce Subscriptions use for queueing large blocks of jobs to be done. In our case we would get a list of products from the Salesforce API and create a job for each product to be synced (or skipped). Action Scheduler allows us a super easy way to manage the job queue and do fun things like query the remaining jobs and adjust the job processing speed (depending on available server resources). After the sync jobs are added to the queue, the site will check each product (based on a unique SKU) to see if it exists. If it doesn’t exist then the product and any associated images or taxonomy terms are generated. If the product exists we compare timestamps to see if it has changed since the last time we updated it on the WordPress site. Any product that hasn’t changed is skipped and the Action Scheduler goes to the next job.
After nailing down the mapping and performance enhancements, and improving the user interface of the plugin, I found another problem: the images from Salesforce have no file extension and WordPress was often complaining (and error-ing out) because it expects the images to have a valid extension and MIME type. The core WordPress functionality for “sideloading” an image would work on some of images but many had issues, so I had to make a slightly modified version of _wp_handle_upload (found in /wp-admin/includes/file.php) that was a little more flexible.
It is oh so satisfying to watch the products show up in WordPress as the sync is being processed. There is still a little more functionality to be added, as far as removing products that don’t show up in the Salesforce products list anymore, but I won’t implement that until we have signed off on the work done so far. I am really enjoying using the Action Scheduler library and hope I get to use it again soon, but if I don’t need to deal with copying images from remote servers again for a little while I would be very happy.
What’s next?
Between my newfound attempt to take up writing again and my ongoing work for retainer clients, I am keeping pretty busy. However, I have submitted quotes to a few fun projects that I look forward to taking on, so stay tuned to what problems I get to solve in the near future.
To read past work journals and learn more about the projects I enjoy, click here. If you are interested in working together or have questions about the work I do, contact me.