Pipeline Cost Analysis Automation

This cookbook demonstrates how to create, deploy, and integrate the Pipeline Cost Analysis Agent into a data engineering workflow.

The agent attributes BigQuery query costs to dbt models and pipeline stages, aggregates costs at pipeline level, and stores them in a centralized BigQuery table for historical analysis and visualization.

Quick Start

  1. Set up BigQuery billing export - Enable detailed cost data export and create the cost history table

  2. Create the Chicory agent - Build and deploy your cost analysis agent with the provided template

  3. Configure Airflow integration - Add the post-hook task to trigger cost analysis after pipeline runs

  4. Run and visualize - Execute your pipeline and view cost analytics in the dashboard

Contents

  1. Introduction

  2. BigQuery Billing Setup

  3. Agent Creation

  4. Airflow Post-hook

  5. Run Pipeline + Dashboard

  6. Sample Analysis Report

  7. Troubleshooting

PreviousTroubleshootingNextIntroduction

Last updated