Pipeline Cost Analysis Automation

This cookbook demonstrates how to create, deploy, and integrate the Pipeline Cost Analysis Agent into a data engineering workflow.

The agent attributes BigQuery query costs to dbt models and pipeline stages, aggregates costs at pipeline level, and stores them in a centralized BigQuery table for historical analysis and visualization.

---

What You’ll Build

An agent that:

• Collects BigQuery job costs from billing exports

• Attributes costs to dbt models/stages

• Aggregates total per pipeline run

• Stores history in a BigQuery table

• Publishes results to a Redash dashboard

Pipeline Cost Analysis Workflow

---

Contents

1. Introduction

2. BigQuery Billing Setup

3. Agent Creation

4. Airflow Post-hook

5. Run Pipeline + Dashboard

6. Sample Analysis Report

7. Troubleshooting

---