July 21, 2026
Data quality tooling has always fallen into three broad categories: profiling (understanding what you’ve got), testing (checking it meets expectations), and observability (watching how it behaves over time). None of these are new. What’s changed is that GenAI is making each of them dramatically more useful — and more accessible to teams who previously lacked the bandwidth to use them well.
Each of these three areas has been transformed by AI in ways that are practical and available now — not theoretical. What follows covers what’s actually changed, which tools are worth knowing about, and how to get started without overcomplicating it.
Data profiling has traditionally meant running statistical summaries across columns and tables. Libraries like ydata-profiling — that your data scientists probably already use — check distributions, nulls, cardinality and patterns, but then rely on a human to interpret the results. This doesn’t scale when you have hundreds of datasets and a small team.
AI changes that. You can point a large language model at your metadata — column names, descriptions, data models, system documentation, even pipeline code — and get surprisingly strong inferences about data quality risks without exposing a single row of actual data. An LLM can look at a column called customer_nino in a table documented as “UK applicants” and flag that it likely contains National Insurance Numbers, suggest validation patterns, and recommend it be classified as PII — all from metadata alone.
This matters particularly for data governance. Many organisations are rightly cautious about feeding production data into AI systems. Metadata-driven profiling sidesteps that concern entirely.
If you want to go further, MCP (Model Context Protocol) lets you give AI agents controlled, read-only access to aggregate statistics — value distributions or null rates, for example — without ever exposing the underlying records. This isn’t hypothetical: tools like Elementary Data and Datafold already offer MCP servers that expose data quality metadata, lineage, and test coverage to AI agents in exactly this way. Databricks has taken a similar approach with managed MCP servers for Unity Catalog, ensuring agents can only access data they’re authorised for. When governance-approved, anonymised subsets of data are available, results get even richer.
Frameworks like Great Expectations, Soda, and Deequ have given data teams a way to codify quality checks — “this column should never be null,” “row counts should be within 10% of yesterday.” The problem has always been that someone has to decide which tests matter, and then actually write and maintain them.
AI is changing both sides of that equation. Soda’s AI suite can now translate natural language into check definitions, suggest regex patterns, and generate SQL validations, which meaningfully accelerates test implementation. Great Expectations’ ExpectAI analyses data patterns and proposes validation rules automatically. DataKitchen’s TestGen — open source under Apache 2.0 — goes further still: it profiles your tables, builds a catalogue, and auto-generates thousands of data quality tests from what it finds.
More broadly, you can use frontier models via GitHub Copilot, Claude, or your own agent setup to analyse a dataset’s profile and propose a sensible starting set of expectations. The human still reviews and approves, but the heavy lifting of “what should we even be testing?” is handled automatically.
Equally valuable: AI can help identify tests that are no longer pulling their weight. If you’ve accumulated hundreds of quality checks over the years, some will be redundant, outdated, or testing for conditions that never occur. Pruning these saves compute costs and, more importantly, reduces noise — so your team can focus on the alerts that actually matter.
Data observability platforms like Monte Carlo, Anomalo, and Bigeye have moved beyond simple rule-based monitoring. They build statistical baselines of your data’s normal behaviour — freshness, volume, distributions — and flag deviations automatically. Under the hood, this is time-series anomaly detection: the platform learns that your orders table grows by roughly 50,000 rows each weekday and alerts you when Tuesday’s load comes in at 12,000.
The more interesting recent development is bringing agentic AI into the observability loop. Monte Carlo now uses AI agents for automated monitor development and root cause analysis: correlating a volume drop with an upstream schema change, checking whether a known deployment happened around the same time, and presenting a probable root cause alongside the alert. Bigeye’s bigAI does something similar — not just detecting issues but recommending specific remediation steps. This is genuinely useful for triage. Teams with extensive data quality tests often find themselves suffering from alert fatigue, where the twentieth “anomaly detected” notification of the week gets ignored regardless of severity.
AI also helps with prioritisation: given what we know about downstream consumers of this dataset, how urgent is this issue? Is anyone actually querying this table, or is it an orphan that nobody would notice?
Start where the risk is. You don’t need to instrument every dataset on day one. Identify your highest-value, highest-risk data — the tables that feed regulatory reporting, customer-facing products, or executive dashboards — and begin there.
Start where you are. If your team already uses dbt, Elementary gives you observability and an MCP server out of the box. If you’re on Spark, try Deequ. If you want to explore AI-assisted rule generation without a procurement conversation, DataKitchen TestGen and DQOps are both open source and capable. Soda Core remains a solid choice for checks-as-code in pipelines. You can prove the value of this approach with free tools and a small team before anyone needs to talk about budget.
Don’t be afraid of custom. Your data teams can use the agent frameworks they are probably already reaching for — GitHub Copilot, Claude with tool access — to connect frontier models directly to your profiling and testing tools. Building agent-orchestrated tooling is increasingly a core data engineering skill, and it lets you use the best available models rather than whichever one a vendor happened to integrate eighteen months ago. Don’t shy away from this because it feels improvised — it often isn’t.
Know when to adopt a platform. As your needs mature, specialist platforms like Monte Carlo and Ataccama bring real value: machine learning pipelines for baseline modelling, correlation engines, lineage-aware alerting. Ataccama’s agentic platform, for instance, automates profiling, rule suggestion and data cleanup as a single autonomous workflow — capabilities that are genuinely hard to replicate from scratch. But the friction of budget approval and InfoSec governance means it makes sense to scale up when the complexity justifies it, not from the outset.
The tools have never been better. The barrier to entry has never been lower.
Thorben is a data and software engineering specialist with over 18 years’ experience delivering scalable, pragmatic data and machine learning products across cloud platforms. He helps teams adopt modern data practices to rapidly build, test, and deliver value from their data.
All business models have their pros and cons. But, when you consider the type of problems we help our clients to solve at Equal Experts, it’s worth thinking about the level of experience and the best consultancy approach to solve them.
If you’d like to find out more about working with us – get in touch. We’d love to hear from you.
