Insights

BigQuery and Dataform

187 posts on BigQuery and Dataform

Need help with BigQuery? See intelligence platforms

Browse the archive

Explore a topic across our insights

BigQuery (185)Google Analytics (144)SQL (117)Looker Studio (102)Privacy (89)Data Analysis (88)Google Cloud (84)AI & ML (83)Marketing (80)Google Ads (71)
More topics
Google Tag Manager (66)How-to (64)Testing (64)Conversion Tracking (49)Ecommerce (40)Firebase Analytics (37)React (36)Server-side Tagging (24)Dataform (23)Debugging (18)Shopify (10)Wordpress (6)Measurelab (3)MarTech (1)

187 results · Page 3 of 11

Behind the Cloud – Releases and scheduling in Dataform

In this episode of Behind the Cloud, Matthew dives into the details of releases and scheduling in Dataform. He breaks down how to manage different versions of your codebase in GitHub. From taking snapshots, to scheduling executions at various intervals daily, hourly, or monthly. By the end of the episode, you’ll have the know-how to confidently release and schedule your code, making it easier to build robust tables and models with Dataform. Video transcript Introduction to releases and sche

Matthew Hooson-

How to extract GA4's event sequencing in BigQuery using the new batch fields

Google Analytics 4 (GA4) exports event data to BigQuery, enabling detailed user behavior analysis. However, GA4 batches events before sending them, making GA4 event sequencing in BigQuery more complex. Fortunately, three fields—batch_event_index, batch_ordering_id, and batch_page_id—help provide precise sequencing information. For a full schema of the GA4 export, head over to google documentation. This article breaks down these fields in a clear, practical way and shows how to use them togethe

Katie Kaczmarek-

Automating BigQuery workflows with conditional logic and data readiness checks

BigQuery offers powerful scripting capabilities that can simplify complex workflows. In this post, we’ll explore two essential techniques: using IF...THEN...END IF for procedural control and leveraging the metadata in project.dataset.__TABLES__ to ensure your Google Analytics 4 (GA4) data is ready before running queries. These strategies help you avoid unnecessary processing, reduce costs, and improve efficiency in your data pipelines. 1. Leveraging IF...THEN...END IF for procedural logic Whi

Katie Kaczmarek-

Mastering data loading in BigQuery using Dataform

Efficient data loading is crucial for managing and updating tables in Dataform. Various strategies exist to handle different use cases, including truncate and load, appending data, and leveraging incremental tables with unique keys. This blog explores these primary methods and more: Truncate and Load In this method, all existing records in the target table are deleted and replaced with a fresh table. This approach works well when a full table refresh is necessary or if managing slowly changin

Prasanna Venkatesan-

A step-by-step guide to migrating scheduled queries to Dataform

Managing scheduled queries in BigQuery often feels limiting — there’s no version control, no easy collaboration, and scaling can be difficult. If you’ve ever wondered how to make SQL workflows smoother, Dataform is your answer. In this post, I’ll show you how I migrated a BigQuery scheduled query to Dataform and how it transformed the way I manage my data pipelines. After all, we all want to know who’s been touching our queries, don’t we? Getting started in Dataform First thing you need to

Katie Kaczmarek-

How to set up a Dataform repository with GitHub & Google Cloud integration

Setting up a Dataform repository can be challenging without the right steps. Whether you’re new to Dataform or want to optimise your workflow, this guide will show you how to seamlessly connect it with GitHub and Google Cloud (GC). What is Dataform and why use it? Dataform is a powerful tool for managing version-controlled SQL workflows in a collaborative way. GC incorporates BigQuery and GitHub integration, providing an efficient way to organise and maintain complex data pipelines. Let’s bre

Katie Kaczmarek-

Behind the Cloud – IAM roles in Google Cloud

In this episode of Behind the Cloud, Matthew dives into Identity and Access Management (IAM) on Google Cloud. He discusses the common frustrations users face, and the significance of assigning specific, granular permissions to mitigate potential risks. Video transcript Introduction to IAM roles [00:00:00] Matt: Hello and welcome to another episode of Behind the Cloud. I’m afraid today isn’t going to be the most glamorous of episodes because what we’re here to talk about today is IAMs on GCP

Matthew Hooson-