How does Incremental loading works for NetSuite as source?

  • 310Views
  • Last Post 22 December 2017
0
votes
Mark Kim posted this 22 December 2017

Hi 

I set up Azure Data Warehouse with a bob storage and created test replication packages with NetSuite as a source.

I have two questions regarding the replication process

1. Incremental loading ('Create table' is not ticked')

How does it work? We'd like to replicate the entire data in all the tables. No filtering will be applied to any of tables. 

Does it always extract all the records from NetSuite?  Or does it extrat only records modified from the last loading? If so, how?  LastModifiedDate is used?  If no such field is available, then what happens?

 

2. Add or remove fields after the first load 

What happens after fields are removed or added after the initial load? (Incremental load is ticked. Create table is not ticked).

I got an error .. then I ticked 'create table' then it went through. It looks like it drops the table and re-create when a field is added or removed. Is that right?

I am asking because we will have a case where a column(s) on Invoice object are added and needed to be replicated. 

 

 

 

0
votes
Simon Bubnov posted this 22 December 2017

> 1. Incremental loading ('Create table' is not ticked')

 

Skyvia replication package has the "Incremental Updates" check box, which allows processing only the records, changed or added in a cloud source since last package run. For more information, please refer to https://skyvia.com/resources/docs/index.html?replication_package.htm

 

If you select the "Create Tables" and "Drop Tables" check boxes together with "Incremental Updates", the tables are created/dropped only on the first run of the package. They are not (re)created during next runs.

 

> 2. Add or remove fields after the first load

 

Skyvia caches metadata of available objects for cloud sources. So, if a field is added, removed or changed in a NetSuite object or you add or delete an object, the cache should be refreshed. To do this, you need to open your NetSuite connection and click 'clear' for the Metadata Cache parameter. For more information, please refer to https://skyvia.com/resources/docs/index.html?netsuite_connections.htm

 

After this you need to edit your package, then edit your task and click Refresh. You also need to re-create tables in Azure Data Warehouse, so that they don't have these columns. If you use incremental updates, then clear the LastSyncDate parameter and select "Create Tables" and "Drop Tables". You package will delete table from Azure Data Warehouse, create a new table with actual NetSuite schema, and copy all records from NetSuite to Azure Data Warehouse.

 

If you do not clear metadata cache or refresh replication package, SQL queries that access non-existing (deleted) fields may be generated, and this will cause an error. It doesn't matter if you edit the Azure Data Warehouse schema manually. For any NetSuite schema changes, you need to clear the metadata cache, edit the replication package and re-create tables in Azure Data Warehouse.

 

Whenever you modify the package schema, the LastSyncDate parameter is reset, and the next replication is performed as in the first time.

 

You need to note that Skyvia support for NetSuite is somewhat limited. Skyvia cannot extract data from NetSuite array fields, and thus, cannot replicate these data.

 

For example, Skyvia cannot extract items from NetSuite transactions - CashRefund, CashSale, Check, CreditMemo, Estimate, Invoice, Opportunity, PurchaseOrder, ReturnAuthorization, SalesOrder, etc. There also are other tables, not transactions, which also are not integrated completely. When you replicate such tables, certain data from them is not replicated.

 

Data from other tables, like Customer, Contact, Campaign, etc. can be replicated completely.

Close