Why your Dataform incremental table keeps adding duplicates
Your Dataform incremental table keeps adding duplicates? A NULL in the merge key could be why. Here’s how it happens with GA4 and how to stop it.
Elisa Liu3 Nov 2026
Blog
Liam Grant5 August 20214 min read

At Measurelab we’ve heard some horror stories about the deletion of critical datasets. Even though Google BigQuery has the BigQuery Backup Recovery of up to 7 days, it’s often better to err on the side of caution when it comes to allowing access to your data warehouse.
Those with rose-tinted glasses glued to their head might have a hard time lauding the usability of GCP IAM (the Google Cloud Platform’s Identity and Access Management). Clients find it to be one of the less understandable parts of the platform.
Originally the GCP’s security settings were three-fold. Viewer, Editor and Owner. Three primitive basic roles for access to everything within one GCP. These roles are still around but really shouldn’t be used unless there is no alternative. To illustrate the problem, if you use the basic roles, that member of your team you granted Editor permissions for Cloud Storage, also now has Editor permissions on your Machine Learning Pipeline. Not the best.
Security exploded into IAM and now there are 4,167 (and counting) different permissions at Organisation, Folder, Project and Service levels. With some services like BigQuery allowing for Dataset, Table, Column and Row security. A true web of security, especially now that there is a hierarchy to permissions. This is great, but it often means you need permissions at two or even three different hierarchical levels in a mix and match that can leave your head spinning.
One of the most common dilemmas our clients bring to us is: how can I only allow a group of people to see a single dataset in my BigQuery data warehouse, rather than all the tables?
Join marketing, customer and product data with tested models and documentation your team can use. Start with the sources needed for your question.
Here’s a guide on how to allow access to just one dataset on your Google Cloud Project.
This preliminary step is optional but will allow for easier management of the selected group of people from outside the Google Cloud Platform environment. If the group that you wish to give access to is large, or changes regularly requiring updates to permissions, this method will save you a lot of manual work.
Google Groups can be used instead of unique Google accounts. For the above-mentioned reasons, you might want to create Google Groups for more effective management of your GCP IAM roles.
To edit a user's Roles and Permissions within a project the Role Administrator Role is required.
To edit a user's Roles and Permissions within a project at dataset level the Role Administrator Role is required.
Understanding the roles you can select and what each means: BigQuery IAM Roles.
Join marketing, customer and product data with tested models and documentation your team can use. Start with the sources needed for your question.
Bring data to connect, a question to investigate or something to fix or build. We can agree a focused scope directly.
Your Dataform incremental table keeps adding duplicates? A NULL in the merge key could be why. Here’s how it happens with GA4 and how to stop it.
Elisa Liu3 Nov 2026
TimesFM spots unusual days in BigQuery; ARIMA_PLUS explains why. We tested both on a year of GA360 data to turn traffic drops into clear answers.
Elisa Liu3 Nov 2026
A year ago, I wrote about getting a BigQuery warehouse ready for AI ( Easy ways to prepare your BigQuery warehouse for AI) , but didn't look properly into Knowledge Catalog (known as Dataplex at the time). We decided to go back and look through Google's Knowledge Catalog properly: what's actually in there, how it works and what it relates to. What Knowledge Catalog is Knowledge Catalog is Google's metadata layer for BigQuery data (and a few other sources). It sits alongside your tables rather
Katie Kaczmarek15 Sept 2026