Slow Imports

  • 244Views
  • Last Post 21 October 2022
0
votes
Jay Ong posted this 18 October 2022

Hi Skyvia Support,

 

How long does it normally take to import 10000 records? When I attempted to import 65000 records, it failed. I had to split the file, but it takes 3 hours to process only 10000 records. I have witnessed Azure Data Factory import 65000 records in approximately one and a half minutes. Would you be able to advise how to speed up the import process?


Regards
Jay

slow-imports
Order By: Standard | Newest | Votes
0
votes
Yevheniia Bilotserkovska posted this 19 October 2022

Hello Jay.

Thank you for contacting us.

Could you please clarify the Package ID you have issue with? 

The execution time of the Package may depend on the number of API calls.

As we can see you have several Import Packages with Lookup option.

You can try to reduce amount of API calls by using cached Lookup. For this you need to select Options - Use cache on the Mapping Definition stage. Please find the example on the screenshot below.

Also you can find more information about cached Lookup here https://docs.skyvia.com/data-integration/common-package-features/mapping/lookup-mapping-target-lookup-and-source-lookup.html#lookup-options

Please feel free to contact us if you need any additional information from our side.

Best regards,

Yevheniia Bilotserkovska.

0
votes
Jay Ong posted this 19 October 2022

Run ID 109782265 was the one with 65000 records that failed. Here are a two Run IDs with 16000 records. 

109861943

109878946

0
votes
Yevheniia Bilotserkovska posted this 21 October 2022

Hello Jay.

Thank you for the clarification.

As we can see from the log Run ID 109782265 failed due to the 'Node is offline for 30 seconds' error. Looks like it was a temporary issue on our side or temporary connection issue.

Run ID's 109861943, 109878946 are successful, but took a long time to proceed the run. To speed up the process we would recommend you try to use cached Lookup option as was mentioned above.

Please try and let us know the result.

We are looking forward to your reply.

Best regards,

Yevheniia Bilotserkovska.

 

New Ticket
Categories
Popular Tags
Close