SkyVia Replication - Package Status Indicates Failure but No Details Available

  • 79Views
  • Last Post 22 March 2018
1
votes
API User Reporting posted this 12 March 2018

Greetings:

I have a replication package with a status of failed yet there are no details available regarding the failure.

The package name is Temporal Store - Salesforce Objects. The ID of the execution instance is 11152597.

This happens somewhat regularly.

Please advise.

Respectfully,

Darryll Petrancuri

 

 

  • Supported by
  • Systems Operations
Order By: Standard | Newest | Votes
0
votes
Simon Bubnov posted this 15 March 2018

Sorry for the late response.

The ID of the execution instance is 11152597.

We have studied this execution. The following error appears in this run: "The remote server returned an error: (404) Not Found."

Probably the error occurs because of high server load. We have noticed that a lot of your replication packages start at the same time, and we create a separate session for each of them. When a lot of packages access Salesforce simultaneously, the Salesforce server may fail to process some of the requests. This situation is unlikely, but possible. 

Please try scheduling your packages so that they started at different time (for example, each next package one minute later than the previous one). So that the second package started a minute later than the first one, the third one a minute later than the second, and so on. 

 

Does the error occur after this?

0
votes
API User Reporting posted this 15 March 2018

Greetings:

 

Thank you for the answer.

it would be most helpful if all the error messages were included in the log. Not having a message present is very disturbing.

This further re-enforces my request for package perhaps not rely on a package execution date but perhaps the LastModifiedDate / SystemModStamp of the individual tables.

Done properly, there should be no reason to use a LastSyncDate if instead you queried for the MAX(LastModifiedDate) or MAX(SystemModStamp) on a per target table basis.

I look forward to you reply and would greatly appreciate it from the lead engineer on SkyVia replication on these matters.

Respectfully,

Darryll Petrancuri

0
votes
Simon Bubnov posted this 20 March 2018

it would be most helpful if all the error messages were included in the log. Not having a message present is very disturbing.

When error appears there can be the following:

1. If it is a record-level error (records fail for some reason: unique constraint violation, lookup fails etc.), the information about failed records with error message texts is added to an error log. This per-record error log can be downloaded by clicking the corresponding package run in the Run History, and then clicking the number of failed rows for a table. For more information, please refer to https://skyvia.com/resources/docs/index.html?error_processing.htm

 

2. If it is a package-level error (package execution fails: connection issue, source error etc.), there will be no log file, and no successful record count or error record count. Instead the error text is displayed, which explains why the package run failed. So, in this case, you can find the corresponding run in Run History and see the error text.

This further re-enforces my request for package perhaps not rely on a package execution date but perhaps the LastModifiedDate / SystemModStamp of the individual tables.

Done properly, there should be no reason to use a LastSyncDate if instead you queried for the MAX(LastModifiedDate) or MAX(SystemModStamp) on a per target table basis.

 

We have sent your request to our lead engineer on Skyvia replication. He will investigate the possibility to implement such scenario and we will notify you when we get any results.

0
votes
API User Reporting posted this 20 March 2018

The error text is NOT in therun history when there is a package-level error.

In the event there is a package level error it seems you're updating LastSyncDate under some circumstances.This should not be the case. 

All of the issues I've reported must be addressed collectively in order to provide proper resilience and fault tolerance.

Respectfully,

Darryll Petrancuri

0
votes
Simon Bubnov posted this 20 March 2018

The error text is NOT in the run history when there is a package-level error.

 

Could you please create and send us a corresponding screenshot? You have the same error in the run 11283852 which was 8 hours ago.

The error message should be "(404) Not Found.".

 

In the event there is a package level error it seems you're updating LastSyncDate under some circumstances.This should not be the case.

 

We think that the LastSyncTime value should be updated in this case. If you use incremental updates a record can fail if source data is corrupt and can't be inserted to a database. For example your run processes 2000k records successfully and 50 records fails. In case when the LastSyncTime is not updated, if you do not change anything in these failed records in source, they will fail again during the next replication, and the successfully loaded records will be processed again. Failed records always will fail and LastSyncTime won't be updated.

However, if you fix data in failed source records, their LastModifiedDate field will be updated, and thus, they will be processed in the next replication run even if LastSyncTime is updated.

If you got some records level errors, caused not by invalid source data, please tell us the error text to investigate this case in more details.

 

All of the issues I've reported must be addressed collectively in order to provide proper resilience and fault tolerance.

 

We have sent all your reported issues to our project manager so that they were processed diligently.

0
votes
API User Reporting posted this 20 March 2018

We should discuss the proper resilience / fault tolerance in more detail.

I would be happy to discuss such with your project manager.

Record level errors should allow last sync date to be updated in the event SkyVia comes up with an appropriate mechanism to attempt to sync the failed records again. Currently, you have no mechanism to do such and the spreadsheets are entirely too cumbersome and next to impossible to use for manual intervention in such.

Package level errors should not allow the LastSyncTime to be updated because if you have multiple tables in a package you can fail to sync zero or more tables if the table was not sync'd before the failure. This is another reason why you need to stop using LastSyncTime at the package level but rather a LastSyncTime at the object level at a bare minimum, but what would be much better is to use the 'Source Object Record SystemModStamp > Target Object MAX(SystemModStamp) or similar as a filter against source objects.

 

0
votes
Simon Bubnov posted this 22 March 2018

 This further re-enforces my request for package perhaps not rely on a package execution date but perhaps the LastModifiedDate/SystemModStamp of the individual tables.

Done properly, there should be no reason to use a LastSyncDate if instead you queried for the MAX(LastModifiedDate) or MAX(SystemModStamp) on a per target table basis.

 

Our lead engineer on Skyvia replication said that your suggestion is reasonable, and they will consider its implementation. I have created the corresponding request. We will post here when this feature is implemented.

 

We should discuss the proper resilience / fault tolerance in more detail.

I would be happy to discuss such with your project manager.

 

Our business process is organized so that our users communicate with the technical support team. Supporters give the information to developers and then tell users about results. Project manager is very busy, and he does not communicate with users directly. 

I addressed all your request to our project manager personally. As we wrote before, as for your suggestion about using MAX(LastModifiedDate) or MAX(SystemModStamp), our team decided to consider the implementation of such functionality. 

 

As for other questions, we will reply to you in https://support.skyvia.com/thread/skyvia-replication-breaks-when-column-is-removed-from-salesforce-object-being-replicated/

 

Record level errors should allow last sync date to be updated in the event SkyVia comes up with an appropriate mechanism to attempt to sync the failed records again. Currently, you have no mechanism to do such and the spreadsheets are entirely too cumbersome and next to impossible to use for manual intervention in such.

 

I have sent this information to our project manager. He told me that he will study this issue, and I will post here when we get any results.

 

Package level errors should not allow the LastSyncTime to be updated because if you have multiple tables in a package you can fail to sync zero or more tables if the table was not sync'd before the failure. This is another reason why you need to stop using LastSyncTime at the package level but rather a LastSyncTime at the object level at a bare minimum, but what would be much better is to use the 'Source Object Record SystemModStamp > Target Object MAX(SystemModStamp) or similar as a filter against source objects.

 

LastSyncTime is not updated if there is a package level error. In this case already processed records are not counted in the subscription counter. Next run will process these records again. And if a record was inserted during a failed run, we will correctly process this situation and update this record instead of creating a duplicate. Thus, we think that we correctly handle such situation.

Close