Why your Dataform incremental table keeps adding duplicates
Your Dataform incremental table keeps adding duplicates? A NULL in the merge key could be why. Here’s how it happens with GA4 and how to stop it.
Elisa Liu3 Nov 2026
Blog
Katie Kaczmarek24 July 20264 min read

At some point, most BigQuery users run a query they instantly regret. A missing WHERE clause on a table that turned out to be enormous. A JOIN that multiplied rows in a way nobody intended. A curiosity query on an unfamiliar dataset that scanned several terabytes before you could cancel it.
By the time the query finishes you already know something went wrong. And you don't find out the cost until the billing report lands.
There is a setting in BigQuery that sits between you writing a query and BigQuery running it. It is called maximum bytes billed, and it stops the query before it executes if the estimated scan size exceeds a limit you set.
Open the BigQuery console and look at the Explorer panel on the left. Scroll to the bottom and you will see settings, also called the query settings panel. This is your personal set of defaults for how your queries run, it covers things like your default location, job priority, whether to use cached results and your SQL dialect.
“Limit the number of query bytes processed billed” is near the bottom of that list. Set a number here and BigQuery will apply it as a cap on every query you run from the console.
This is the part worth understanding properly. BigQuery estimates how many bytes a query will scan before it runs. If that estimate exceeds your cap, the query fails immediately, before any data is scanned and before any cost is incurred.
The error message will tell you the query would have exceeded your limit and how many bytes it estimated. At that point you can look at the query, work out why it is scanning so much and fix it. Maybe you are missing a partition filter. Maybe you are selecting columns you do not need. Maybe the table is larger than you expected and you want to preview it with a TABLE SAMPLE first.
The key point is that no money leaves your account. The check happens before the execution, not after.
Join marketing, customer and product data with tested models and documentation your team can use. Start with the sources needed for your question.
I use 15 GB as my personal default. That is comfortably above what a well-written analytical query on a reasonably sized table should need, but low enough that a runaway query on a large table will hit the limit rather than scan the whole thing.
If you work with consistently large datasets you might want to push that higher. If you are mostly exploring, experimenting with new tables or writing ad-hoc queries, 15 GB gives you plenty of room. Start there and adjust based on whether you are hitting the cap on queries you expected to be fine.
One useful habit: when you open a new or unfamiliar table, write a TABLESAMPLE SYSTEM (10 PERCENT) query first to understand its scale before running anything broader. Maximum bytes billed is a safety net, not a substitute for understanding what you are querying.
SELECT *
FROM dataset.my_table TABLESAMPLE SYSTEM (10 PERCENT)
While you are in settings, also look at the job timeout option. Set it to something like 30 seconds. If a query runs for longer than expected (even if it is within the bytes cap) it will automatically be cancelled.
These two settings together cover the two most common causes of unpleasant billing surprises: queries that scan too much data and queries that run for far longer than expected. Neither setting costs anything to enable and both take about 30 seconds to configure.
Maximum bytes billed only applies to queries you run yourself from the console, or queries in code where you explicitly pass the parameter. It does not apply to other users in your project, to service accounts running scheduled queries or to queries fired by connected tools like Looker Studio. Those all need their own controls.
It also only covers query compute. Storage costs, streaming inserts and other BigQuery features are unaffected.
For personal query safety, it is one of the most effective single settings you can enable. Go and set it now before you forget.
Join marketing, customer and product data with tested models and documentation your team can use. Start with the sources needed for your question.
Bring data to connect, a question to investigate or something to fix or build. We can agree a focused scope directly.
Your Dataform incremental table keeps adding duplicates? A NULL in the merge key could be why. Here’s how it happens with GA4 and how to stop it.
Elisa Liu3 Nov 2026
TimesFM spots unusual days in BigQuery; ARIMA_PLUS explains why. We tested both on a year of GA360 data to turn traffic drops into clear answers.
Elisa Liu3 Nov 2026
A year ago, I wrote about getting a BigQuery warehouse ready for AI ( Easy ways to prepare your BigQuery warehouse for AI) , but didn't look properly into Knowledge Catalog (known as Dataplex at the time). We decided to go back and look through Google's Knowledge Catalog properly: what's actually in there, how it works and what it relates to. What Knowledge Catalog is Knowledge Catalog is Google's metadata layer for BigQuery data (and a few other sources). It sits alongside your tables rather
Katie Kaczmarek15 Sept 2026