Agentic Data Engineering

Introduction

Agentic data engineering is all about using AI in your existing workflows. It's mainly aimed at integrating AI into tools you already use through MCPs. While it's focused on the Coalesce product suite (Transform, Catalog, and Quality), most of the lessons apply regardless of which tools you use.

We focus on the workflows you most often find yourself doing: planning a change and checking its downstream impact, getting to the root cause of an anomaly, implementing data governance processes, or generating the same report every week.

This guide covers everything from a single ad hoc prompt to a production playbook. Whether you are exploring on your own, building skills for your team, or putting workflows into production, you will find practical tips and real examples throughout.

What an MCP is

MCP stands for Model Context Protocol. It is an open standard that enables a client such as Claude or Cursor to provide a large language model (LLM) with structured context from other tools and data sources. This gives the model context and metadata that would otherwise be unavailable to it.

At Coalesce, we have an MCP for Transform, Catalog, and Quality. Each one exposes many of the capabilities you'd otherwise reach through the UI. For example, with the Catalog MCP, you can programmatically extract the column-level lineage to determine the exact impact before making a change. With the Transform MCP, you can ask it to implement those changes. And with the Quality MCP, you can have it run a full root cause analysis, extracting metadata around similar issues, recent code changes, logs, errors, etc.

How to understand MCPs through tools

You can get a long way using the MCPs out of the box, but it's often worth understanding the platform's inner workings and the tools it offers. A tool is a function an MCP server exposes to the agent, typically a single action like fetching data, changing state, or triggering a workflow. The agent decides when to call a tool and the server runs it. The set of available tools defines the boundary of what the MCP can and cannot do.

Every Coalesce MCP tool ships with a description. It tells you not just what each tool does, but where its limits lie. The more specific the instructions baked into a tool, the more deterministic the output. Open-ended API calls leave all the interpretation to the LLM, which can quickly take things off the rails.

Tools also map onto the steps a person would take. If you ask the agent to investigate potential root causes of an issue using the Quality MCP, it works through the same metadata a data engineer would, one tool at a time, choosing each call from the tool descriptions and what it has learned so far:

• It reviews historical issues to see how similar problems were resolved in the past.
• It reads upstream code changes across the affected models.
• It detects schema changes anywhere across the stack.
• It runs queries to analyze the actual data and what happened.
• It retrieves data profiles to compare before-and-after distributions.
• It samples rows from the affected tables to find the exact records that failed.

Each of those is a separate tool with a defined scope. The agent chains them, feeding what one returns into the next, and it branches or stops early depending on what it finds. Reading the tool descriptions tells you which of these the MCP can do, and where it will stop.

If you haven't specified the steps, there's no guarantee that you'll get the same result the next time you ask the LLM the same question. A useful trick is to ask the LLM to specify exactly which tools it used and in which order. With that at hand, you can create reproducible prompts that produce more deterministic results (more on this in chapter 3: Skills).

Some MCP servers also help here by shipping their own routing context, telling the agent which tool to reach for before it starts guessing. It's worth checking whether an MCP provides this, since it does some of the sequencing work for you, though your own skills are still what pin down the exact path you want to run every time.

Three levels of working with the MCPs

You can work with MCPs through one-off prompts, but there are also cases where it makes sense to productionize workflows through skills or playbooks.

• Level 1, prompts. Ad hoc interactions to speed up data engineering and governance. For example, an ad hoc prompt to understand the potential root causes of an issue.
• Level 2, skills. Key workflows packaged as reproducible skills. For example, a way to generate and triage data quality reporting across the team.
• Level 3, playbooks and agents. Structured playbooks, often with governance tooling and orchestration. For example, a background agent running workspace analytics on customer issues each evening.

Choosing your LLM tool

You're likely already using prompts in tools like ChatGPT and Claude Desktop, and coding tools such as VS Code or Cursor.

There are two main interfaces you'll typically use to engage with MCPs: code editors and chat interfaces.

For use cases such as data modeling, you're most likely using a code editor, while for ad hoc exploration, a chat interface often works best. It's worth considering the user experience for both, especially if you start developing tools and skills that a wider audience will use.

Writing good prompts

What a good prompt includes

There's no one-size-fit-all answer to writing good prompts, but here are a few tips we've seen help

• A role lets the model answer at the right level. For example "You are a data governance lead working in Coalesce" can help make sure the answer you get is not too technical
• A goal works best stated as the outcome you want rather than the steps to reach it
• Constraints are the rules the model must follow, such as leaving passthrough columns untested or keeping Tier-1 coverage capped
• An output format makes the result land in a shape you can use, whether that is a model file, a set of tests, or a short table

Finding the right balance

Below are three examples of prompts to add tests to a data source - one that's too specific for most use cases, one that's too vague, and one that strikes a better balance.

Give the model room to ask and disagree

For anything non-trivial, tell the model to ask questions before it writes anything, and to push back if your instruction will not hold up.

This helps confirm the model has the right level of detail before adding tests.

Finally, you can help the model avoid context overload. For example, parsing an entire project or every schema risks burying the request, and the model focuses on material it does not need. Point it at the one model and the handful of upstream tables that matter. It's also recommended that you start a new conversation when you're moving to a different task or feature to reset the context window.

Extend and chain MCPs to automate workflows

There's no shortage of questions you can ask. To learn where you can expect good answers, start by understanding what tools each MCP offers, and check the docs for example questions.

You don't have to stop here. MCPs become more useful when you combine them with the other MCPs your team already uses. Most clients let you connect several at once, and the model will call across them in a single prompt.

Here are a few examples of how to use Coalesce MCPs alongside other tools.

• Investigate and resolve an issue. Have the Quality MCP investigate the root cause, then have the Transform MCP implement a fix based on what it finds.
• Data quality report in Slack. Ask for a summary of open issues older than 7 days using the Quality MCP, have it posted to a specific Slack channel, and automatically open issues in Linear.

Limitations of prompting

Starting with basic prompting is a good way to get familiar with MCP capabilities and see results right away. It is worth keeping an eye on a few common limitations.

• Results vary from run to run. Ask the same question twice, and you will sometimes get slightly different answers. For exploration, this is fine, but for other workflows it can lead to inconsistencies.
• Models may answer even when the tool is not available. If the MCP does not expose the capability you are asking about, the model will sometimes improvise rather than stop. Knowing how to understand tools is the best guardrail.
• Everyone starts from scratch. Each person figures out their own prompts, workflows, and patterns. That makes it hard to scale, and it tends to favor people already proficient with MCPs.

Managing permissions and data access

If you are using the Coalesce MCPs in production workflows, it is worth being deliberate about permissions, especially around read versus write access.

Below are a few recommendations to consider as a starting point:

• Default to read-only, managed at the token level, and keep it that way for most users.
• Treat MCP write tokens the way you treat production database credentials. The same person who approves one should approve the other.
• Gate raw warehouse tools behind per-call approval. An unbounded SELECT can run up warehouse costs or pull customer PII into a context window where you did not expect it.

From prompt to skill

Introduction to skills

A skill is a Markdown file (SKILL.md) plus optional supporting files that provide instructions about how to perform a task. This helps you encode workflows so they're performed consistently each time and share standardized ways of working across your team. For the rest of this chapter, we'll focus on skills as used through Anthropic's products, although they're starting to get popular in other tools as well.

To use a cooking reference, the best way to think about a skill is as a recipe. A good skill spells out four things, just as you would when writing an instruction doc for a colleague.

• What the task is
• What tools to use
• What good output looks like
• What to avoid

When a new conversation starts, Claude automatically checks your request against the available skill descriptions to see if one fits. You can also force it by typing / to bring up the available skills.

Designing a good skill

The first part of a skill is the description. This tells Claude when to load the skill and is important to get right. A good description has three parts: what it does, when to use it, and its key capabilities.

Skills also rely on progressive disclosure. The header is always loaded; the body loads when the skill appears relevant; and any linked files load only as needed. That keeps token use low while keeping the expertise available when it matters.

The better the description, the better the first iteration of the skill.

Below is an example of a good description that's specific, actionable, and with a clear trigger.

A good description

The two below are weaker starting points.

Descriptions that need work

When designing the core of a skill, you should start with the use case in mind and write the steps as you would to a colleague. Here you specify what the user wants to accomplish, which best practices to encode, and which MCP tools to use.

With this, you get repeatable results, unlike ad hoc prompts, which can vary from one run to the next.

Below is an example of a good use case description.

Below is an example of a complete skill designed with best practices in mind:

Creating a real-world skill

The steps above – integrating MCPs, writing clear descriptions and use cases, and specifying the expected result can take some work to get right. Luckily, there are creative tools to help you get started. Claude's skill-creator is excellent for kickstarting the process and only requires a description. With this, you can iterate fast and fine-tune your skill so it produces the output you expect.

Back to the data quality report example, this is the result we ended up with.

The process we went through to create this shows what an iterative process like this often looks like. Here are some considerations we had to give feedback on only after seeing the first version of the report.

• What the structure of the report should look like. What happens when there are hundreds of errors
• What counts as a data issue, all severities, or only errors
• Whether it should be actionable, tagging owners, linking data products, and suggesting next steps.
• Whether it posts directly or goes for review, which Slack channel and the fallback, and whether the window is adjustable, say monthly instead of weekly. The first version was too dense, so it was replaced with a brief summary, with the evidence in the Slack thread.

Planning a skill suite

For data roles specifically, think of data skills as jobs to be done. These can be grouped into areas of responsibility such as data quality, data modeling, analytics, and data governance. Each job category contains specific jobs.

This is powerful and, for example, lets us encode our testing philosophy as a skill, so each time data engineers want to add new tests, they can do so using the add-tests-to-node skill that follows our standards.

As a rule of thumb, the more often you find yourself doing a workflow, and the more time it takes, the more likely it is to benefit from being converted into a skill.

Each skill includes a description that helps Claude quickly find the right one for the job. So when someone asks to "triage this issue," the model consistently points them to that skill, and they get all the right context, instead of the model chaining LLM calls and reinventing the workflow each time. That's the benefit of building out a deliberate skill suite and a good way to figure out which skills you should build.

Extending skills with other assets

You can extend skills with assets beyond the main content of the SKILL.md file. This can be anything from specific scripts you want to execute as part of the skill to other references, templates, etc.

your-skill-name/
├── SKILL.md            # Required - main skill file
├── scripts/            # Optional - executable code
│   ├── process_data.py # Example
│   └── validate.sh     # Example
├── references/         # Optional - documentation
│   ├── api-guide.md    # Example
│   └── examples/       # Example
└── assets/             # Optional - templates, etc.
    └── report-template.md # Example

Evaluating our skill

In many cases, eyeballing the results tells you how well a skill works. Sometimes it makes sense to be more systematic, through three lenses:

• Does it trigger on the right requests. If you run 20 sample prompts that should trigger the skill, for how many of those is the skill actually triggered?
• Does it produce what we expect? Are all elements there, and does it look as we expected?
• How does it handle edge cases? What if there are more than 20 errors? What if there's no owner set?

Claude's skill creator comes with built-in testing. You shouldn't use this 100% as a replacement for manual testing, though it does give you a more systematic way of assessing skill quality, especially as you start growing your library of skills.

We ran a small benchmark: three test cases, each run twice, once with the skill loaded and once without. Each run was graded against five objective assertions, such as whether it contains a TL;DR, deduplicates the noisy monitor, calls out P1 downstream impact, and is Slack-formatted.

In short, the skill provides us with better output (more of the assertions we're running pass), but on average also adds to the time it takes to complete the task. The extra tokens are the cost of the skill pulling more data (get_issue_impact per top issue and deeper execution history) and following a stricter template. For a once-a-week report, that's fine.

With Skills, the variance also drops significantly as they make each task execution much more predictable. Without one, Claude makes structural decisions fresh every time. Sometimes it writes a great report; sometimes it skips the TL;DR or forgets to deduplicate noise. With a skill, the template is enforced. Although we shouldn't read too much into these numbers with such a small sample, they still illustrate the point.

Sharing skills

As you build out a suite of skills, you'll want to share them with your team. There are several ways to this:

• In the repo. Commit the skill folder to version control, and anyone who pulls it has a copy. This fits teams already working in a repo, and it gives you history, review, and rollback for free.
• For one person. Upload the skill folder to Claude's settings to use it yourself; it is handy for trying a skill before the team adopts it.
• Across a workspace. Upload via the Skills API so every member of the workspace receives it, with versioning and centralized management. This is the production route, where you pin a version and roll out updates deliberately.

For applications, agents, or scheduled workflows, skills are invoked via the API and require a code execution environment. The same skills work in the Agent SDK when you build custom agents. Treat a shared skill like software you install. Pin versions, evaluate before each rollout, and adopt skills only from sources you trust, since a skill can run code and call tools against your data.

Productionizing MCP workflows

There are several ways to productionize MCP workflows: as playbooks or as built-in AI agents in the products. Below are a few practical ways we're doing this at Coalesce. Some workflows are core enough to build directly into the product. Others are domain-specific, and we've built production-grade playbooks and skills on top of our MCPs.

Built-in agents: Copilot and Scout

Built-in agents are workflows that are part of the core product, with guardrails already in place. They run where the work already happens, so you do not have to assemble the context yourself. Copilot from Coalesce Transform and Scout from Coalesce Quality are two examples of this.

Copilot, in Transform. Lives in the development workspace. You describe what you want in natural language, and it turns that into governed transformations, writing SQL and building staging, dimension, fact, and view layers, keeping lineage intact under the roles and audit already in place. It captures context from the subgraph, and a read/write toggle controls whether it can change anything or only propose.

Scout, in Quality. Coalesce Quality's Scout works in the background as an always-on data SRE, running asynchronous agents that kick off their own jobs, working through every open issue around the clock, triaging each by importance, and surfacing a recommended action with the evidence alongside the issue. By the time you open an issue, the investigation is usually already done, so you stay in the loop by reviewing conclusions rather than chasing every alert. In one example, an anomaly monitor fired on a sharp drop in row count, and within seconds Scout had traced it to an intentional code change, linked the relevant commit, and proposed marking the issue as expected.

While much of the same analysis is available through the MCP, running the investigations in the background as an agent enables new workflows, such as presenting users with a sorted list of recommended actions across all issues.

Production-grade playbooks and skills

Not every production workflow belongs in the product. When something is specific to how your team works, it is often better to build it on top of the MCPs as a playbook, running on a set schedule.

Customer issue triage. Runs automatically each morning against the workspace that monitors our production systems: the ClickHouse tables, Postgres, and job queues. It groups related issues, queries the data, reads logs, drafts likely root causes ready to share with customers, and tags the person on rotation that week. It replaces an otherwise lengthy manual investigation.

Governance rollout. Takes a team from zero to a governed catalog in 8-12 weeks. It writes descriptions before assigning owners, classifies assets into tiers, caps Tier-1 at roughly 5% based on real usage signals, and assigns an owner, an effort estimate, and a measurable exit criterion to each phase that the agent can check itself.

When to use which

We start from a prompt for exploration, then build skills for repeatable tasks, playbooks for repeatable workflows, and finally a built-in agent for the workflows that fit better into the product.

• Built-in agents. Make sense when they capture context from the UI and underlying platforms, the subgraph you are editing, or the full state of the monitoring environment. This way, you are not feeding it context by hand.
• Building on the MCPs. More flexible and better when the workflow is specific to a team or needs to span tools. You encode your own sequencing, anti-patterns, and exit criteria. The governance rollout is a good example since the phases, tiers, and order vary by organization.
• Do not overbuild. Building a playbook for a low-frequency, low-stakes task is usually a waste of effort. A prompt or a skill is enough.

Coalesce MCPs

You can download and install the Coalesce MCPs in your workspace using the following links

• Coalesce Catalog MCP
• Coalesce Transform MCP (link TBD)
• Coalesce Quality MCP (link TBD)

Coalesce is the only data transformation and governance platform designed for the AI era. Built on a metadata-driven framework, Coalesce gives data teams the speed to build and deploy transformations 10× faster—while enforcing the standards, structure, and governance needed to scale sustainably. With Coalesce Catalog, transformation and metadata management come together in a single solution, enabling discovery, trust, and collaboration across the business. Whether accelerating AI-assisted migrations from legacy tools or future-proofing enterprise data architectures, Coalesce provides the guardrails and efficiency to keep data teams AI-ready.

Book a demo