22 March 2018
This further re-enforces my request for package perhaps not rely on a package execution date but perhaps the LastModifiedDate/SystemModStamp of the individual tables.
Done properly, there should be no reason to use a LastSyncDate if instead you queried for the MAX(LastModifiedDate) or MAX(SystemModStamp) on a per target table basis.
Our lead engineer on Skyvia replication said that your suggestion is reasonable, and they will consider its implementation. I have created the corresponding request. We will post here when this feature is implemented.
We should discuss the proper resilience / fault tolerance in more detail.
I would be happy to discuss such with your project manager.
Our business process is organized so that our users communicate with the technical support team. Supporters give the information to developers and then tell users about results. Project manager is very busy, and he does not communicate with users directly.
I addressed all your request to our project manager personally. As we wrote before, as for your suggestion about using MAX(LastModifiedDate) or MAX(SystemModStamp), our team decided to consider the implementation of such functionality.
As for other questions, we will reply to you in https://support.skyvia.com/thread/skyvia-replication-breaks-when-column-is-removed-from-salesforce-object-being-replicated/
Record level errors should allow last sync date to be updated in the event SkyVia comes up with an appropriate mechanism to attempt to sync the failed records again. Currently, you have no mechanism to do such and the spreadsheets are entirely too cumbersome and next to impossible to use for manual intervention in such.
I have sent this information to our project manager. He told me that he will study this issue, and I will post here when we get any results.
Package level errors should not allow the LastSyncTime to be updated because if you have multiple tables in a package you can fail to sync zero or more tables if the table was not sync'd before the failure. This is another reason why you need to stop using LastSyncTime at the package level but rather a LastSyncTime at the object level at a bare minimum, but what would be much better is to use the 'Source Object Record SystemModStamp > Target Object MAX(SystemModStamp) or similar as a filter against source objects.
LastSyncTime is not updated if there is a package level error. In this case already processed records are not counted in the subscription counter. Next run will process these records again. And if a record was inserted during a failed run, we will correctly process this situation and update this record instead of creating a duplicate. Thus, we think that we correctly handle such situation.