Imports taking a long time

  • 85Views
  • Last Post 2 weeks ago
0
votes
Jay Ong posted this 4 weeks ago

Hi Skyvia,

 

I am trying to import a single csv file with 60k records. It has almost been 2 hrs and only 6k records have appeared in the DB. This is unacceptable. There are only 3 columns and 60k records should not take this long.

 

Can you please advise what the issue is? What is the largest import file you support??

 

I have had to merge multiple csv files into this large single csv files because you do not support importing files using minutes in the File Mask.


Regards
Jay

Order By: Standard | Newest | Votes
0
votes
Nataliia Nikulina posted this 3 weeks ago

Hello Jay,

 

Thank you for contacting Skyvia Support team.

 

We see that the package has failed with multiple errors.

Please review a line-by-line log and send us the errors that we can clarify the situation.

 

We look forward to hearing from you.

 

Kind regards,

Nataliia

0
votes
Jay Ong posted this 3 weeks ago

Hi Nataliia,

 

I believe I have fixed the issue with the multiple errors whic was due to the absence of a unique primary key for each record. The imports still appear to take a long time. I will advise the total time once the import completes.


Regards
Jay

1
votes
Nataliia Nikulina posted this 2 weeks ago

 Hello Jay,

 

Thank you for the update.

 

As we can see, your package has been completed successfully. 

 We noticed ‘The specified lookup is ambiguous. Lookup must return at most one row for the specified lookup condition’ error in one of the runs in your package. It means that if you want to use a Lookup mapping, you need to have a column (or a set of columns) that uniquely identifies a record, other than the primary key.

 

In case SCID is not a unique value, you can add another condition so that SCID + FullName will be a unique condition for your Lookup.

You can add one or more conditions by selecting + Add Lookup Key option.

 

 

Let us know if we may help you more.

Best regards,

Nataliia,

Customer Support Engineer

  • Supported by
  • Jay Ong
0
votes
Jay Ong posted this 2 weeks ago

Hi Nataliia,

 

I created the SCID column as the new lookup key i.e. Date + Sensor Name. The import completed without any issues, but it took a little under 7hrs to import 60,500 records. Is there any way to increase the import speed?

 

The ability to add additional lookup keys/columns is an excellent feature. I was not aware of it. Thanks for sharing. I will try it as I am using PowerShell scripts to create the additional UID (SCID) column. It will give me more options for sure.

 

Is there any way to increase the import speed?

 

Thank you & regards
Jay

0
votes
Nataliia Nikulina posted this 2 weeks ago

 Hello Jay,

 

Thank you for the update.

 

 

The package is configured for UPSERT operation that itself takes more time than, for example, INSERT. Also there is a pretty big amount of records in the package which slows down a package performance.

 

 

As a workaround we recommend that you enable a 'Use cache' option for lookup mapping to increase the performance of your package:

 

 

 

 

Please update us about the results. 

 

 

Best regards,

Nataliia,

Customer Support Engineer

 

Close