What is a Query Pipeline?
A query pipeline enables you to create a chain of dependencies between queries, where:- Each query in the pipeline can depend on the results of previous queries
- The pipeline executes queries in the correct order based on their dependencies
- All queries are executed as a single atomic operation
Use Cases
Query pipelines are ideal for:- Complex Data Transformations: Breaking down complex analysis into smaller, manageable queries
- Data Lineage: Maintaining clear dependencies between related queries
- Coordinated Updates: Ensuring multiple queries are executed together
- Incremental Processing: Building data pipelines where each step depends on the previous one
Response
The endpoint returns apipeline_execution_id which you can use to:
- Check the status of the pipeline execution using the Get Pipeline Execution Status endpoint
- Monitor the progress of individual nodes in the pipeline
- Retrieve results from each query in the pipeline once execution completes
- Credits are consumed based on actual compute resources used for each query in the pipeline
- Use the
performanceparameter to specify execution tier (medium or large) - Query parameters can be passed to parameterize queries in the pipeline