Do you have a Shopify store with a large number of products and their variations? And a lot of metafields? Then perhaps you have noticed how painfully slow a full import of such stores may be. The reason behind lies in the nature of Shopify GraphQL Admin API rate limits. The richer the information your integration queries, the higher its cost in terms of API limit. As a result, an integration may be obliged to slow down drastically in order not to spend all its points quota too soon.
In order to address this problem, we have introduced a couple of new features into Feedoptimise Shopify integration: now you can run partial updates (or deltas/incremental imports) in parallel with regular full imports.
How Shopify full imports can be slow on large catalogs
When using Shopify GraphQL Admin API, it is necessary to keep track of your points limit budget. Each field included in a single API call consumes a certain amount of points, according to a complex formula. Simple scalar values, enums and objects consume 1 point per call. Connections cost depends on the number of items returned. Mutations (writes) cost 10 points per call.
Each store starts with 1000 points in the pool. Points are replenished according to a constant per-second rate based on your plan tier. Shopify standard plan offers 100 points/sec replenishment rate; 200 points/sec for advanced plan and up to 1000 points/sec for Shopify Plus or Enterprise plans.
Depending on the amount of information requested from each product in a bulk query (price and stock info, metafields, images and references, translations), a single query may consume 50 or 100 points easily, sometimes even more after batching many products into one. Once the budget reaches zero, the rate becomes limited to a slow refill rate. Even though Shopify returns points at a fixed pace and regardless of the speed at which the requesting software consumes them.
Hence, for example, a 100,000 SKU store with extensive usage of metafields would require many points per query for a single product – more than the refill rate. The integration is therefore forced to wait until it gets the quota of points to be able to make another API call.
How partial updates overcome this issue
We have developed three partial update modes to bypass the mentioned problems. While the rest of your product attributes is imported during the regular weekly/monthly imports, the following partial updates allow to fetch updated price and stock data or new products much quicker, every few hours.
Three types of partial updates
Here's a brief list of what is included in each partial update mode:
- Price updates - current pricing information across all products and variants, including different currencies.
- Inventory updates - stock information for all your products.
- Import of newly added products - It is possible to combine any or all of the listed modes. Your catalog attributes remain untouched until the next scheduled full import.
It is possible to combine any or all of the listed modes. Your catalog attributes remain untouched until the next scheduled full import.
Independent scheduling for each mode
Each of the three partial update types can be independently scheduled in terms of frequency. Here's an example of optimal schedule:
- Full import - every night or during off-peak hours once a week;
- partial imports - several times a day, for example, every few hours starting early in the morning.
The time-consuming part (data extraction, translations pulling, resolving variants, image processing) will still take place in the nighttime as usual. But prices and stock changes are propagated to your feeds within a few hours. It allows you to respond quickly to the market changes and adapt your products' prices.
Also, partial updates reduce the number of API calls made, so your point limit budget is more likely to be enough for other apps and integrations connected to the same shop.
The downside
While partial updates are great for fetching updated prices and stock levels or importing newly added products, other attributes will remain as is until the next full import is executed. If you have changed a metafield value, edited a product title, swapped an image or performed any other action with the product, the change will take effect only with the next full import.
This approach suits most stores, as product details such as title, metafields and description don't change as often as prices and stock do. And whenever you need to push an urgent content update, you may still force an on-demand full import.
What are the benefits?
One of our clients was facing issues with their Shopify import taking more than 20 hours to complete. This happened to their largest store containing more than 100,000 SKUs and variants and extensively using metafields and translations in several languages. Since the new feature became available, we were able to significantly improve the situation. Partial import cycles currently take less than 30 minutes to complete.
Although catalog sizes and generated feed outputs remained unchanged, partial imports now pull smaller amounts of information about each product, thus making a greater number of requests per minute. Regular full imports occur only once a week, while prices and stock are updated in a shorter period of time.
Who should enable this feature?
The proposed functionality can be helpful especially to those who have either:
- Large stores (more than 100,000 SKUs);
- Heavy usage of metafields and translations;
- Several markets configured in their Shopify store;
- Frequent changes in prices or stock.
If your catalog is small enough to be able to import regularly within a couple of minutes, you will hardly experience any performance gain.
How to turn this feature on?
Just log in to your account in Feedoptimise. Go to your Shopify source setup page and find the toggles for each type of partial import right next to your current full import settings. Turn them on, choose the required frequencies, and the next time your import occurs, it will do so in partial mode.
And if you need help configuring the partial import schedule, just contact us via our platform's support section and our team will be happy to help.