Skyvia Data Flow Logs - Why Errors Aren't Shown by Default

Skyvia Data Flow Logs: Why Errors Are Not Shown by Default

Overview

In Data Flow, tracking execution success or failure can be challenging due to the nature of row processing. Rows that fail in one component may succeed in another, complicating error detection. By default, Skyvia logs do not provide detailed error or success information, which can lead to confusion.

To enhance visibility, implement the following components in your Data Flow:

Components for Enhanced Logging

  1. Row Count Component — counts the records passing through specific points in your Data Flow. It helps track the number of rows processed successfully or failed at each step. For more details, see the Row Count documentation.
  2. Log Component — records specific information during execution. You can configure it to log essential details like errors, names, row statuses, or specific data values, which is crucial for deeper analysis.

How to Add Components

  • Row Count: Drag and drop the Row Count component at key points of your Data Flow to count processed records.
  • Log: Insert a Target Log Component to capture additional details like errors or IDs.

For example:

By setting up these components, you can effectively monitor each step, simplifying troubleshooting and tracking.

Useful Links

    • Related Articles

    • Skyvia Query: Verify Data in Your Connection

      Overview Skyvia Query allows you to browse and verify data in any connected data source directly from the Skyvia interface, without writing complex scripts or using third-party tools. You can use the visual Builder view or write custom SQL ...

    • Integration Failure Types in Skyvia Data Integration

      Overview When using Skyvia Data Integration, you may encounter two distinct types of integration failures: 1. Record-Level Errors These occur when one or more individual records fail to process during an integration task. For example, in operations ...

    • Understanding Records in Skyvia Data Integration

      What Is a Record? A record in Skyvia Data Integration represents a single row in a database table or a single object in a cloud source (e.g., one contact in Salesforce, one row in a Google Sheet). Every integration operation - Import, Export, ...

    • Data Integration: You have reached the limit of records

      Overview The error "You have reached the limit of records" means your Skyvia account has consumed all records included in your current Data Integration plan for the billing period. No further records will be processed until the limit resets or you ...

    • Backing Up Association Data with Skyvia

      Overview Association data refers to the relationships between records in a data source. For example, in a CRM system, association data might include the connections between customers, contacts, and accounts. Losing this data can lead to data ...