Create Data Pipelines from ServiceTitan to Azure SQL

This tutorial details the Steps to fetch data from two ServiceTitan tables and load it into Azure SQL. The two tables are BusinessUnits and Invoices and we will fetch data by connecting to their API endpoints.

The below articles provide the sequence of steps to create the above pipelines, along with the key concepts covered in each step.

1. Connect to ServiceTitan: Concepts - OAuth2 authentication and connecting to a cloud application.

2. Connect to Azure SQL: Concepts - Connecting to Azure Blob and Azure SQL, data security, data privacy.

3. Create a Connection Between ServiceTitan and Azure SQL: Concepts - Creating a relationship between source and destination

4. Set up BusinessUnits Pipeline: Concepts - Full Refresh dedupe logic

5. Set up Invoices Pipeline: Concepts - Incremental dedupe logic, configuring Azure Blob destination, pipeline variables

6. Load Invoices Historical Data: Concepts - Load historical data, schema adjustment, schema tracking and actions

7. Pipeline Monitoring and Documentation: Concepts - Alerting and monitoring of pipeline runs and pipeline concurrency, handling pipeline failures

8. Pipeline Infrastructure and Cloud Costs: Concepts - Understand how DataStori executes pipelines in the customer's cloud and how their data never leaves their IT environment