Hi,
Let me introduce our connections and then our problem. We have an integration between Skyvia and our CRM, which is Hubspot and then another integration with our data lake which is Big Query. We have dashboards that we have created in Google Sheets bringing the information from Big Query that has all our data from our CRM.
Recently, we received a comment that our dashboards in Google Sheets were not updated, so we have analyzed why we were not bringing the most updated results from Hubspot.
We have been checking Skyvia to identify if there has been a failure in the last 7 days, but all the logs have been successful. However, something caught my attention while checking the package: "Hubspot to BigQuery - Incremental Updates 3 [every 02h]", as I saw that the logs going from 12/12 - 10:25 am have taken only a few seconds to run.
So today, I cleared the Metadata Cache and, in turn, reset LastSync. This caused the data to update correctly.
We have programmed to receive a notification if something was not working properly, however, all the logs were successful. My question is: how can we detect that the information is not being updated correctly to prevent our dashboards from showing incorrect or outdated information?
I look forward to hearing your comments,
Thanks in advance.
Iván.
-14.43.34.jpg?width=690&upscale=false)
Data is not being triggered correctly
- 25Views
- Last Post 14 December 2022
Hi,
Thank you for contacting Skyvia Support.
As we see, your situation caused due to the Metadata changes in your HubSpot.
If metadata of your cloud app changes, by default, Skyvia knows nothing about it. If some fields or objects in your data source that are used in your integration packages, backups, or endpoints, are deleted, this will cause errors next time when the integration package or backup runs, or endpoint data are queried. If an object is added in the data source, existing backups, integration packages, etc. will not fail; however, Skyvia will not know about these changes until metadata cache is cleared.
By default, the Metadata Cache parameter is set to Infinite, and cache is reset only manually. If metadata of your cloud application changes often, you may need to change this setting. We also see that you have already changed the setting. It will help you to avoid this issue further.
Please see more about the Metadata Cache here: https://docs.skyvia.com/connections/metadata-cache.html
In other possible problematic situations, you will get notification about it on your email if the Email notification setting is enabled.
Should you have other questions, don't hesitate to reach out.
Best regards,
Serhii
- Supported by
-
