Create A Data Pipeline

Understand the concepts and configuration details to create a data pipeline in DataStori

📄️ Configure Ingestion

Data pipeline setup in DataStori is a three-step process:

📄️ Scheduling and Data Load

Data pipeline setup in DataStori is a three-step process:

📄️ Destination Setup

Data pipeline setup in DataStori is a three-step process:

📄️ Historical Load and Pipeline Concurrency

Once the data pipelines have been setup, they will continue to run as scheduled, fetching fresh source data with each run. There may also be a requirement for historical data, which means additional pipeline parameters and executions. DataStori maintains data integrity through all stages and modes of the data load process.

📄️ Data Deduplication

When DataStori fetches and loads source data, it carries out the following transformations:

📄️ Data Flattening and Schema Management

Data Flattening and Normalization