Why your Dataform incremental table keeps adding duplicates
Your Dataform incremental table keeps adding duplicates? A NULL in the merge key could be why. Here’s how it happens with GA4 and how to stop it.
Elisa Liu3 Nov 2026
Blog
Mark Rochefort7 April 20265 min read

Companion piece to Measure Pod #139 with Colin Zima and #98 with David Jayatillake
Your warehouse is full of tables. Your agents can query them. The question is whether they'll get the same answer.
Ask three different agents "what's our monthly revenue?" and you'll get three different numbers. Not because the data is wrong - because nobody defined what "revenue" means in a way machines can consistently resolve. Is it gross or net? Recognised or invoiced? Including refunds or not? Each agent makes its own assumption, and none of them tell you which one they picked.
This is the problem a semantic layer solves.
At its simplest, a semantic layer sits between your data warehouse and the things that query it - dashboards, models, agents, analysts. It defines:
None of this is new. Good analytics teams have always maintained metric definitions somewhere - a spreadsheet, a wiki, a Dataform model with comments. The difference is that a semantic layer makes those definitions machine-readable and enforceable.
Two years ago, the path from data to decision ran through a human. An analyst wrote a query, checked the output, and built a dashboard. The definitions lived in their head, and that was fine - they'd catch inconsistencies before anyone saw them.
That path has collapsed. AI agents now query your warehouse directly. They generate SQL, execute it, and return answers in seconds. The analyst isn't in the loop. And the definitions that lived in their head? The agent doesn't have them.
This is what David Jayatillake described on the podcast as "the governance gap." The warehouse was never designed to be self-describing. It stores data, not meaning. When the access layer was a human analyst, that didn't matter. When the access layer is an autonomous agent, it matters enormously.
The failure modes are predictable:
Conflicting numbers. We worked with Springer Nature where Marketing, Product and Finance each produced different revenue figures from the same underlying data. Different query logic, different filters, different assumptions. It took a unified Dataform architecture to get them within 2% of each other. Now imagine three agents with the same problem - except nobody notices the discrepancy because the numbers arrive instantly and confidently.
Invisible errors. An agent that hallucinates is obvious. An agent that returns a plausible but subtly wrong number is dangerous. Without governed definitions, there's no way to distinguish a correct answer from a confident one.
Ungovernable access. As organisations deploy more agents across more data sources, the question isn't "can the agent access the data?" - it's "should it, and does it know what it's looking at?" A semantic layer provides the guardrails.
In practice, it's a set of definition files - typically YAML or SQL - that encode your business logic as infrastructure. At Measurelab, we've built SEAM (Semantic Engine for Agent Mediation) to do this specifically for AI agents. The architecture is:
The key principle: definitions are infrastructure, not documentation. They're version-controlled, tested, and deployed like code. When someone changes the definition of "active user," that change is reviewed, approved, and propagated to every agent and dashboard simultaneously.
Here's where it gets interesting. Traditional semantic layers only govern structured data - warehouse tables with defined schemas. But agents increasingly query unstructured sources: Slack messages, meeting transcripts, email threads, support tickets, internal documents.
No warehouse has ever governed this kind of data. SEAM does - by extending the same definition-and-resolution pattern to unstructured sources. The agent's access to a Slack channel is mediated through the same governance layer as its access to your revenue table. Same audit trail, same validation, same control.
You don't need to boil the ocean. Start with your five most-queried metrics - the ones that show up in every board deck and every campaign report. Define them precisely. Encode the definitions. Route your most-used dashboard through them.
Once those five metrics are governed, extending to ten, then fifty, then to agent access becomes incremental. The hard part isn't the technology - it's getting agreement on what the metrics mean in the first place. If you're not sure where your organisation stands, our agent readiness assessment is designed to help you figure that out.
If you want to hear more, Colin Zima from Omni joins us on Measure Pod #139 to talk about AI and semantic layers in modern BI, and David Jayatillake on #98 covers the foundational thinking. And if you want to see how SEAM applies this to AI agents specifically, take a look at the product page.
Update — September 2026: SEAM continues to evolve as an approach to shared meaning and context for people and AI. The runtime described in this article was an earlier implementation of that concept.
Book a free discovery call and tell us what you're working on.
An automated first look at collection and consent signals visible on your public site. No account access needed.
Your Dataform incremental table keeps adding duplicates? A NULL in the merge key could be why. Here’s how it happens with GA4 and how to stop it.
Elisa Liu3 Nov 2026
TimesFM spots unusual days in BigQuery; ARIMA_PLUS explains why. We tested both on a year of GA360 data to turn traffic drops into clear answers.
Elisa Liu3 Nov 2026
A year ago, I wrote about getting a BigQuery warehouse ready for AI ( Easy ways to prepare your BigQuery warehouse for AI) , but didn't look properly into Knowledge Catalog (known as Dataplex at the time). We decided to go back and look through Google's Knowledge Catalog properly: what's actually in there, how it works and what it relates to. What Knowledge Catalog is Knowledge Catalog is Google's metadata layer for BigQuery data (and a few other sources). It sits alongside your tables rather
Katie Kaczmarek15 Sept 2026