Shopify Data Flow - really slow

  • 80Views
  • Last Post 03 November 2021
1
votes
Jo Einarsson posted this 21 October 2021

Hello 

I have built a data flow updating pricing in product variants in 2 Shopify stores using data from Dear Systems and it is incredibly slow. It is taking hours to update pricing on 64 variants and I plan on updating thousands across 5 channels I've been waiting for this to finish its run so I can report how long it is taking but it is seriously too long to wait.

I am not using any lookups or conditional split logic after the source module just an SQL query on Dear(source) retrieving products updated in the last 30 minutes directly connected to 2 targets (Shopify Stores) using a split module

I have all of my Shopify variants IDs stored in my Dear database (customs fields) so the flow doesn't require doing lookups in Shopify to get the IDs etc.

I understand the answer might be a limitation of the Shopify API but the reason for using Skyvia this way is Dear doesn't currently support bulk updating products in Shopify channels based on recently updated and requires re-listing all products in the system to the channel again as well as updating stock quantities, but even that takes a fraction of the time it takes to update just 64 variants at the moment in Skyvia.

If you could please point me in the right direction or have any suggestions for optimizing the data flow that would be great. Can Skyvia use the bulk update features of the GraphQL API? I also have the product ids available for each row is it possible to use this somehow to optimize?

 

  • Supported by
  • lang liu
Order By: Standard | Newest | Votes
0
votes
Nataliia Nikulina posted this 25 October 2021

Hello Jo,

 

Thank you for contacting Skyvia Support team.

I am sorry for the late reply, am I looking at a correct package - 

https://app.skyvia.com/#/25241/packages/144000 ?

Looking forward to hearing from you.

 

Best regards,

Nataliia,

Customer Support Engineer 

0
votes
Jo Einarsson posted this 27 October 2021

Hey Nataliia 

 

Yes, these are the different packages below that I have tried to run with different methods to optimize for speed. The platform is so great and intuitive it's just how can I get the runtime down? 

https://app.skyvia.com/#/25241/packages/144000

https://app.skyvia.com/#/25241/packages/144002

 

Thanks, Jo

 

 

0
votes
Nataliia Nikulina posted this 28 October 2021

Hello Jo,

 

Thank you for the update.

We continue to investigate your case and will contact you as soon as we have any updates here.

Feel free to contact us in case of any questions or issues.

 

Best regards,

Nataliia,

Customer Support Engineer

 

0
votes
Jo Einarsson posted this 28 October 2021

I thought i would just run the flow again to see how long it takes on: https://app.skyvia.com/#/25241/packages/144002 and now I'm getting the error

 

The network path was not found.

0
votes
Nataliia Nikulina posted this 29 October 2021

Hello Jo,

 

Thank you for the update.

 

We have passed your request about this error to our development team.

We will contact you as soon as we have any updates. 

 

Best regards,

Nataliia,

Customer Support Engineer 

0
votes
Nataliia Nikulina posted this 29 October 2021

Hello Jo,

We have received an update from our development team that the 'The network path was not found' issue has been fixed. The point about package performance is currently under investigation.

We will contact you as soon as we have any updates. 

Feel free to contact us in case of any questions or issues.

 

Best regards,

Nataliia,

Customer Support Engineer 

 

0
votes
Nataliia Nikulina posted this 03 November 2021

Hello Jo,

 

As for the speed of your package performance, filtering goes through the cache in this case.

We have investigated the performance issue in your package.

The execution takes much time because filtering is performed via the cache.

This is related to Shopify API limitations - not all filters are supported by this API natively.    

Let us know if you have any further questions.  

 

Best regards,

Nataliia,

Customer Support Engineer  

0
votes
Jo Einarsson posted this 03 November 2021

Ok, I have packages with no filters and still get the same results. Is there a roadmap to implement the Bulk update of the GraphQL API with Shopify?

0
votes
Nataliia Nikulina posted this 03 November 2021

 

Hello Jo, 

 

Thank you for the update. 

 

We don’t have any timeline for this request but we will definitely let you know as soon as we have any news.

 

Feel free to contact us in case of any questions or issues. 

 

Best regards,

Nataliia,

Customer Support Engineer

Close