🤖

The Autonomous Data Engineer

When AI becomes your data engineering colleague

A family of open source products — and a YouTube series — for bringing AI agents into analytics engineering across Databricks, Microsoft Fabric, and Power BI.

The ADE family

Modern analytics is fragmented: notebooks on one platform, semantic models on another, pipelines everywhere. Two open source products tackle the two questions every team keeps asking — what exists, and how do we operate it — and they are built to be driven by AI agents, not just read by humans.

🧬

ade-catalog

Cross-platform analytics metadata catalog

Maps your entire analytics landscape — what exists and how it's connected. It extracts metadata across platforms, builds cross-platform lineage that native tools don't provide, and exposes everything through a navigable web UI and an MCP server for AI agents. (Formerly ade-core.)

📊 Multi-platform metadata

Extracts and consolidates metadata from Databricks, Power BI, and Microsoft Fabric into a single queryable catalog.

🔗 Cross-platform lineage

Builds the data-flow graph across systems — Databricks → Fabric → Power BI — for real impact analysis, not platform-locked lineage.

🤖 MCP server for agents

Lets AI agents query the catalog, lineage, and impact directly — search, object detail, lineage, and stats over MCP.

🖥️ Web UI

A clean interface to browse and explore the catalog visually, with deployment both as a hosted service and on your own infrastructure.

Python Databricks Power BI Fabric MCP Server Open source
View on GitHub →
⚙️

ade-ops

Operations framework for multi-platform analytics teams

Most data-platform tools hand you a sample and walk away. ade-ops asks what you have, then scaffolds the workflow to operate it — keeping notebooks, semantic models, and reports in sync across dev, cert, and prod from a single source of truth.

📦 Single source of truth

One canonical src/ with overlay-based transforms for each environment — no hand-maintained per-environment copies.

🛡️ Diff before push

Remote workspaces are authoritative. Every promotion is visible — pull, diff, confirm, push. No silent overrides.

🧭 Scenario-aware onboarding

Asks about your setup — Databricks-only, Databricks → Power BI, or Databricks → Fabric → Power BI — and scaffolds what fits.

👥 Human + agent operation

Usable from a plain CLI, but reaches its potential paired with an AI coding assistant that drives its skills.

Python Databricks Fabric Power BI CLI + Skills Apache 2.0
View on GitHub →

The YouTube series

The Autonomous Data Engineer is a video series documenting the journey of building an AI-based autonomous data engineer. Each episode explores a different aspect: from metadata extraction to autonomous pipeline navigation, from lineage tracking to code generation.

It's not a tutorial. It's a logbook — showing what works, what doesn't, and what happens when you give an AI agent real access to a data ecosystem.

Watch on YouTube →

The vision

Each analytics system has its own metadata, its own interface, its own language. ADE was born to close that gap — not with another dashboard, but with products an AI agent can drive: one that understands what exists across the stack, and one that operates it safely.

Together, ade-catalog and ade-ops form a continuous operational intelligence layer over an organization's analytics estate. Already in production with enterprise data teams.

The goal isn't to replace the data engineer. It's to give them a colleague that never sleeps, never forgets, and can traverse the entire stack in seconds.

← Back to projects