#112 Product analytics tool assessment

[00:00:00] Dan: Welcome back to The Measure Pod. This is episode 112, where it is me and Bhav this week. And we are discussing, essentially my problem or my issue that came up this week, which is everything to do with, product analytics, assessments, requirements, understanding the vendor landscape and things like that.

[00:00:32] Dan: my name is Daniel Perry Reed. I’m a principal analytics consultant at MeasureLab. And of course, I’m, as always, joined by Bhavik Patel. Director of analytics and experimentation at LeanConvert. So Bhav, I have just spent the last 40 odd minutes grilling you on all questions that I need to do for a living.

[00:00:47] Dan: So you’ve helped me do my job a little bit easier. First of all, thank you. Secondly, what do you think? 

[00:00:54] Bhav: You’re welcome. it’s kind of weird doing a, kind of like an intro to an episode. That’s just you and I can feel it feels a little bit self serving. I think it was, it’s the whole podcast though.

[00:01:05] Bhav: It’s a, it was an interesting episode in the sense that you are some, you are some very good questions. And actually the questions that I imagine a lot of people have had to think about some point, or we’ll have to think about at some point, if they work in this kind of like marketing and product analytics, you know, Crossover ecosystem, because they, you know, the lines are getting more and more blurred every day.

[00:01:23] Bhav: I think marketing teams are more and more interested in what’s happening to their customers. Once they reach the website, like on the flip side, product teams, more and more interested in like, what is the quality of the traffic and where it’s coming from? And how does that. Drive user behavior and changes the behavior.

[00:01:37] Bhav: So I think, I think at some point, more, more and more companies are going to have these types of questions, especially given that, you know, as you, as you’ve, as you’ll hear in the episode GA4, when it comes to product analytics is still at this moment in time, November, 2024, still very limited as a product analytics tool.

[00:01:56] Bhav: The data is there if you set it up properly, but it requires a platform like BigQuery to really. pull it out and, and query in a way that’s quite, you know, and, and customise in a way that you would want. So I think it’s an interesting episode. I hope, I hope listeners find it useful and not too confused by it.

[00:02:13] Bhav: I tried my best. There’s so much like information blurring around in the, in my head right now that I just, I sometimes fail to find the right words. Words. And you say tracking when you mean tagging. And when you say tagging, when you mean tracking and it gets confusing, even my voice is going there for sure.

[00:02:29] Dan: Well, we appreciate it. Thank you so much. Before we jump into the episode, is there anything coming up that you want to plug or share or shout about? 

[00:02:36] Bhav: There’s nothing coming up. I do want to talk about, I’ve my recent transition away from Twitter. it, the platform, they come an absolute success. Pool of misinformation and stuff.

[00:02:47] Bhav: So I’ve now moved over to blue sky. If anyone wants to catch me, I’ve been able to secure Bhav, the handle. So Bhav blue sky or be sky or social, whatever the common one is.That’s BHAV. Yeah, so if you want, if you want to find a new place where I’m actually more active than I, I stopped being active, like properly active on Twitter a while ago. so if you want to see sort of like a the flurry of thoughts that are completely irrational and the ramblings of a madman like, come follow me and hit me up on, on blue sky. 

[00:03:21] Dan: Awesome. All right. Well, we stick a link in the show notes for that. I have a Twitter profile that I have fairly used occasionally. I do something, but I never, I never go on it.

[00:03:31] Dan: I really struggle with all social media, to be fair. I think that the closest I’ve ever got is, so I’m a, I’m one of these guys that doesn’t have a Facebook. I don’t use WhatsApp, but I don’t use any of this stuff. So, LinkedIn is the only real place that I can class a social media in that feels pretty, pretty lame out of all the things to use, but I use it for business.

[00:03:47] Dan: And I, I find a level of enjoyment keeping up with analytics and I feel like it’s useful at the same time. It’s kind of like that. My hobby is running, you know, it’s like, you know, one of my things I like to do is exercise. It’s like, yeah, get over yourself. I know I sound like that. But anyway, there’s nothing to promote.

[00:04:00] Dan: The only thing I will say is that if you will listen to this on the first weekend that this episode comes out next week, we are doing our first ever crap talks, Brighton meetup, which as they say, The name suggests it’s in Brighton, central Brighton, a couple of minutes from the train station. If you are coming in via train, same by bus as well.

[00:04:17] Dan: I’ll put a link to this in the show notes. It’s on the meetup community. It’s completely free. We’ve got a couple of speakers talking about analytics, products, conversion rate optimization. we’ve got, we’re doing it in a Gail’s bakery for God’s sakes. It’s going to be a really interesting one. we’ve got an after party, a local pub.

[00:04:31] Dan: So please come along if you’re local, or if you want to travel down for it, I’m sure it will be fun. that’s it really. So my self serving plugging thing is over. With that, I hope you enjoy the episode and we’ll catch you in blue sky. Blue sky. Is that how you say it? Are you going to blue sky us around? Is that how you say, how do you tweet in blue sky? I don’t know. 

[00:04:50] Bhav: I just say, I post them. Well, I mean, I will plug it one last time and sense that the data community on blue sky is really thriving. And there are a lot of, so blue sky has this, this cool feature called starter packs, and you can find the analytics and data starts back and just follow a whole bunch of analysts and data professionals in the industry from multiple different parts of the world.

[00:05:12] Bhav: And because there is no. Self serving algorithm. It’s what your feed is basically what you want it to be, which is just data. If you’ve signed up to the data starter pack and followed all the data people. 

[00:05:24] Dan: Awesome. Well, blue sky away. Enjoy the episode. All right, we’re back. We are, solo. Why say solo? There’s the two of us. And, I wanted to use this opportunity to dive into the wonderful world of product analytics and as, maybe frequent listeners may understand. I work in the world of marketing analytics. I have done for well over 10 years and the world of product analytics may as well be a foreign language because I know nothing.

[00:05:52] Dan: So I hope you’re ready because I’ve got a bunch of questions for you. And, it may, may seem as very self serving, but I’m here to learn. I want to know a bit more about product analytics. Are you ready? 

[00:06:03] Bhav: I hope so. 

[00:06:06] Dan: Let’s go. You know what? That’s as good. That’s as good as we’re going to get. And, I want to jump straight in. So, so I, I wanted to preface that with the whole, my background being in marketing analytics. And if it comes into, if the question comes up, like, I’ve got a website, I’m running advertising. How the hell do I track this? What are the different measurement opportunities? what analytics tools are there?

[00:06:24] Dan: What should I be thinking about? I feel confident enough. Maybe we can even do this for another episode. I feel confident enough that I’ve got that nailed. I know what I’m doing, or at least I’m aware enough of the landscape to be able to advise and make a good recommendation to the companies that I’m working for.

[00:06:38] Dan: However, a question came up in my work life the other week, whereby a client of mine, we’re doing all of the lovely, web and marketing and digital analytics stuff, and then they said to me. We’ve got this product, we’ve got this digital product and they’re, they’re a big, like SAS platform and they log in, all we’re doing is generating leads and I’m working with them on that at the moment, but then once they, once they sell the software, they log in and they’re like, what do we do?

[00:07:01] Dan: Should we put GA4 on it? And I was just like, and it kind of stumped me because, I think we’ll get into a bit of the GA4 stuff later on, but I think starting at the very beginning, I had no, I had no clue where to even start that thought process. So. If someone comes to you and they’re like, we’ve got this login pods.

[00:07:18] Dan: We’ve got this digital product. We want to do analytics. What I suppose if you were in my shoes in that situation, what would, what would you say, like, where would you start that conversation? 

[00:07:28] Bhav: I probably wouldn’t start it with GA4 if I’m being honest. I think GA4 serves as a good platform, but I, I, I do think GA4 is more around when you’ve got a slightly uncomplex product. And you’re more concerned with things like ROI CPA, performance of how well your landing pages are doing, how effective your, your, maybe your e-commerce funnel is. I think GA4 in those instances serves. As a wonderful platform, it does what you need to do. It’s not without its limitations as I’m finding more and more week after week, and I’m realising that actually the complexity of what I potentially want to do doesn’t, is not enabled within the platform.

[00:08:10] Bhav: And I constantly have to try and either get back, get access to BigQuery, or. I have to consider the fact that you might just, I just might not be able to do it. So it’s usually big. If I would need to do anything more complex, it would usually be a big quote. So I think for when you, when you’re in the instance, you talk, and I work for a SaaS company, keeping in mind, before I joined link convert, we used, yeah, we used amplitude as our, as our, as our platform for analytics and amplitude was really good for understanding how user behavior, it can be monitored, measured and tracked and reported on.

[00:08:43] Bhav: If I had to do what we were doing in GA4, I probably would have shot myself. So if I know, I mean, genuinely, if, when it comes to understanding usage, product analytics platforms far exceed. The capabilities of GA4 when you want to understand the performance of your ad campaigns and, you know, checkout funnels and things like that.

[00:09:05] Bhav: And, you know, there isn’t complexity in the product. I think GA4 serves well. So other examples, alternatives, you know, like, e-commerce. are typically very good, GA4 is typically very good for that because you don’t really need to do too much on it. if you’ve got something like, maybe a gaming site, like a betting gaming site, the complexity of the product isn’t, it’s not too in depth.

[00:09:24] Bhav: You really just want to know how many people are registering to your site, how many people are then depositing, and then how many people are then potentially placing a bet on or doing some games. So things like that is, I think it’s very, fairly straightforward. But when you, when you’ve got a platform, which is more around user behavior, usage, apps, I think a proper product analysis platform is where I would start.

[00:09:47] Dan: Okay. And that’s, that’s, I think what I had in the back of my brain through years of doing this podcast and chatting to you and, and, and, and riffing on this kind of stuff for a long time. My brain somehow said, no, let’s take a step back before we just run ahead and do Google tag manager, Google analytics for, et cetera, et cetera.

[00:10:05] Dan: Because. I’m then just treating it like a website and I’m doing web analytics and acquisition based analytics, which is what the platform is designed to do. So, so put, put it back slightly. So if we establish that Google analytics for out of the box, especially if it’s, if it’s slightly more complex than a, than a basic product, isn’t that useful, or at least there’s better alternatives.

[00:10:26] Dan: Like, what are you thinking about? What are the considerations like in that marketing analytics tool? I’m thinking about how do they handle attribution modeling? What does sessionization look like? What kind of, other sort of, how do I import media spend? And is there any output connectors to the different platforms to share things like that?

[00:10:43] Dan: So when it comes to marketing analytics tools, I’m like, okay, that for me is a, a checklist of things that I would go through and say, Hey, look, how do you do attribution modeling? Cause that’ll affect how I measure marketing campaigns when it comes to product analytics tools. Like, what is the criteria?

[00:10:57] Dan: What are you thinking about in the back of your mind when you. For example, you have like the five biggest products, analytics stores come and pitch to you. Like, what are you thinking about as you go through that process? 

[00:11:05] Bhav: So I think there’s going to be a few things I’m going to be thinking about. First of all, I really don’t want sample data in the platform.

[00:11:13] Bhav: I’m going to be working with, cause with GA4, I’m largely working with sort of like everyone who comes to the site after a certain point. When you, you know, when you, when you’ve got the registered users, they become your customers effectively before they register. They’re still prospects, right? And if you don’t get that a hundred percent accurate, it’s probably okay.

[00:11:33] Bhav: But when they become customers, they registered with your platform. You probably don’t want to have a probabilistic view of what’s happening on your site. You try, you want to try and have it as close to a hundred percent as you possibly can. So that’s not to say, I don’t think it’s not to say you can’t do a GA4.

[00:11:48] Bhav: If you set up server side tagging, you probably are in a better position. situation. But with the product analytics platform, what you want to try and do is you want to try and have a deeper integration with your backend services, because you want to understand when they’ve logged in. You want to keep that, that user stitched across everything they do.

[00:12:06] Bhav: You probably want to ingest data into the platform as well. So for example, if you’re a SAS company, you might want to ingest actually what type of user are they? Are they a free user enterprise user business user? So having all of that stuff put being pushed within. GA4, within, so not within GA4, within your product analytics platform with each event.

[00:12:27] Bhav: So you might want to do some type of enrichment services, which you can’t do with GA4, at least I don’t think you can do with GA4, but effectively with product, some type of product analytics platform, you probably, you might want to do some type of enrichment of who’s the customer, what type of plan are they on?

[00:12:41] Bhav: Are they, you know, in, in, in many cases for SAS platforms, you have kind of like a two sided marketplace. Are they the person who registered or are they the consumer of the person who registered? Right? So you might have like a customer, your customers, customers, and you want to identify them. You might want to build a view that is purely within the SAS organization.

[00:13:02] Bhav: So for example, you know, if you’ve, if your client is acquiring a new customer who is made up of multiple people, you want to, you might want to have an organizational view of that user, right? So not what is one person doing, but what is this organization doing and all the users within that. So you then want to create different tiers from users to organizations and, and you know, subgroups within that.

[00:13:23] Bhav: You don’t want to think about potentially like some type of AB opportunity. That’s not to say you can’t do it with GA4, but in many cases, GA4, you’re doing some type of integration. So, you know, if you can, if you, if the platform does allow you to do some type of A B testing, Amplitude does A B testing, PostHog does A B testing.

[00:13:40] Bhav: Now, I think you’ve now got like smaller pools of users to work with, but you’re, you’re doing it all within the self contained platform. There’s probably loads of other things you want to think about. but they’re the kind of the immediate ones that come from top of mind enrichment, non sampling. Different types of cohorts and customer usage, AB testing. 

[00:13:56] Dan: So, so let’s zoom into two of those. So, one thing is, so yeah, I’m going to ignore GA4 for the time being. I want to come back to that in a moment. So let’s pin that for a bit in terms of the analysis that you do inside these products.

[00:14:07] Dan: Right. I mean. But there’s one aspect to this, which is feeder, feeder, data lake, or a data warehouse. And we can do whatever the hell we want to do using SQL Python R over there. Sure. Inside the product itself. what I know a lot of the product analytics is around cohorting funnels, segmentation.

[00:14:24] Dan: So like, what are the kinds of features from, from the kind of analysis and reporting aspect that you’re looking for? Like, what are, what are the kind of common questions and, and approaches that you need when it comes to analyzing this kind of data? 

[00:14:35] Bhav: Oh, you’re really putting me on the spot here, Dan. If we think again, if we think about sort of like what you’re doing on GA4, which is all around ROI acquisition, those type of things, your product analysis platform is really trying to measure lifetime value or retention effectively.

[00:14:49] Bhav: Right. So you’ve, you, you, you’ve mentioned in sort of like cohorts. And then I think it’s, it’s kind of, I’m going to be talking about this in the abstract. If, if we can think, if we can like. arbitrarily pick an example from one made up customer, we can start to like maybe put a bit more depth into the discussion.

[00:15:03] Bhav: But in an arbitrary sense, if we were thinking of the types of things that the types of cohort analysis you’d want to do, the type of lifetime value usage analysis, you want to look at, you’ll be, you’re probably going to want to do a few things. You want to look at the depth of the usage and breadth of the usage.

[00:15:21] Bhav: So how frequently do people use your platform? And then also to what depth and degree do they use your platform? Right? Because if you’re a SAS vendor or someone where you, where you probably need more product analytics, I guess that was a good example. They’re not a SAS vendor actually, but they are a.

[00:15:37] Bhav: Platform, which has a kind of complex buying cycle because every week customers come back, they pick different recipes. The recipes themselves could be vegetarian. They could be cuisines. They could be healthy recipes. They could be me only. So you want to try and capture all of that information within your product analysis platform.

[00:15:57] Bhav: So then you can cut it. So you can say, actually, what is the likelihood of like, what is the cohort of vegetarian users look like? What does that conversion rate look like? What is your. You know, pescetarians, what do they look like? if you’ve got users who have a weekly subscription, how does their behavior differ to people who have one off subscriptions, or not one off subscriptions, but make one off purchases or don’t really have a subscription?

[00:16:20] Bhav: Are they buying as frequently? If we think about Something like PlayStation. That’s again, like I know we’ve had Gabor on recently. And so PlayStation is currently top of mind for me. they probably want to use some type of product analytics S platform, which requires a lot of custom integration and custom event tracking GA4 in this instance is quite limited unless you’re on like the 360 platform, but even then, I’m not quite sure how deep you can go with the event tracking, in GA4 with PlayStation, you’d probably want to try to understand what game they’re playing, what levels are sending, you know, users are playing on how long they’ve been playing.

[00:16:57] Bhav: Are they playing single player versus double, you know, multiplayer? Are they playing online versus offline? All of these types of things that they’re, they’re collecting are going to be quite bespoke. And I think when you, when you really think about the power of product analytics, if you’re in a very bespoke organization, I think that’s going to be the thing where, where is your, it’s really going to separate itself from a platform like GA4.

[00:17:19] Bhav: So, again, I’m doing a swing and some roundabouts. You answer. This was going to be a messy episode, by the way. I hope people can follow. I hope people are following my train of thought, but your question was like, what are the types of analysis you’re doing? You’ve got cohorts, your segments, your LTV, your frequency analysis, how often people are playing your breadth analysis.

[00:17:36] Bhav: So, how often do people come back? What are the different things they do? are there relationships between two posts? Talk is a really good platform because, one of the things it does is it automatically builds correlations between metrics, which you can’t do in GA4. So if you’re trying to understand like, hey, this metric is really going up or going down, it will tell you that there is a diametrically opposite or a parallel metric that is going up or down in line with that metric.

[00:18:07] Bhav: So you might be able to try and find sort of like correlations and do more advanced things that GA4 just doesn’t do. Doesn’t or won’t do. And I think that’s largely down to the developers. The people who built GA4 are very different in the sense that people who build things like amplitude and postdoc and heap and, you know, whatever the other ones are.

[00:18:27] Dan: So let’s unpin the GA4 things. Let’s come back to this. So there’s a, there’s an argument there, which would be, well, which I’m going to, I’m going to voice, product analytics is all about, event based data, event schemas, and a lot of the transition from universal analytics to GA4 is moving from a sessionized view to a, an event schema.

[00:18:44] Dan: And, and a lot of product analytics, people, including yourself, was like goddamn time. Everyone else is doing this. And now Google analytics is doing this. So there is an element there that Google analytics has moved to a more product tracking mentality. They also can ingest data from lots of different sources.

[00:18:58] Dan: Now it may be more limited than other sort of more well established product analytics tools, but it’s moving that direction. The other big thing, what were two big things is that it syncs to BigQuery and it’s free, right? Or at least the majority of it’s free. If you’re within the kind of limitations of the free license.

[00:19:13] Dan: Now, all of that considered is Google Analytics for good enough. Is it a good enough product analytics tool to get. Event data into a data warehouse, or just to track the top level view. If you don’t have, or have yet to have ever ventured into this world, you don’t have budget, you haven’t considered a vendor selection.

[00:19:32] Dan: You don’t have that internal awareness or knowledge or adoption of a tool just yet. Is it good enough or is it actually worth doing the work before you go far into this, because I feel like once you’re embedded tool, it kind of is kind of sticky. Right. 

[00:19:46] Bhav: Yeah, I mean, tools aren’t something that I would recommend people changing. so like on a yearly basis, I think they’re one of those things that, you know, my philosophy on this, you squeeze the platform till you’ve got every inch of value out of it. And then when you then feel like you’re approaching the platform’s limitations and capacity, you then think about potentially moving or doing something different.

[00:20:08] Bhav: So to answer your question, it depends on who the user is now for people who can work within a SQL environment. And if you have got 360 and you’re capturing every single event and you’ve got loads of different, events being captured and being sent, and three, six, remember in order to connect to BigQuery, you can still do it, but you’re not pushing all your data into big query.

[00:20:32] Bhav: You will hit limits quite, quite soon. So let’s assume that you are on 360. As an end user like myself. I can work in BigQuery. So for me, when I say about bloody time, for GA to move towards, towards an event based schema. I mean, this in the sense that actually offers people like me, more flexibility to create views that are more bespoke to what I need.

[00:20:55] Bhav: I can, I can manipulate the data how I want, but I’m not the GA4. So when you think about your clients and You or, you know, whoever, who might not be a natural user of BigQuery, is GA4 good enough to answer, you know, to, to, to quote, to paraphrase you. And in that instance, my answer is a categoric, no. The platform, when you’ve used platforms like Amplitude, Heap, Postdoc, you actually come to realize how limited a platform GA4 is.

[00:21:25] Bhav: I mean, it’s very, very, very limited. The platform just doesn’t do or have the power or the same type of analytical capabilities that these other platforms have. And this is where it then starts to become very subpar. Does that makes sense? 

[00:21:43] Dan: Yeah, no, it does. It does completely. So I mean, I suppose, let’s go slightly, slightly off, not off topic, but let’s say let’s take that and zoom in a little bit.

[00:21:51] Dan: How do you think Google analytics has got to the place it is? Do you think it is just because it’s free, but if it is a subpar product and there are plenty of better alternatives out there that are either free or next to nothing in the grand scheme of things, like why does everyone continue to use a subpar? 

[00:22:07] Dan: Or sorry, this is your opinion here. 

[00:22:09] Bhav: No, no, I let me, when I say it’s subpar, I mean, subpar from a product analytics perspective as a marketing analytics platform, it’s just, you know, it does what it needs to do. Right. So from a marketing analyst perspective, it meets or meets a criteria, but let’s to the point, to your point, most organisations are e-commerce organisations, right?

[00:22:30] Bhav: So majority, I think majority of the people who have adopted GA4 are e-commerce organizations. I think as we’ve transitioned to more web based products and apps and phone apps, obviously we’ve been around for a while, the requirements to analyze how users use those platforms. Is has changed to, to, to like traditional e-commerce, e-commerce brands and e-commerce models.

[00:22:54] Bhav: So even though, like when I was at Gousto, we did use a product analytics platform. We still had GA4. GA4 was largely used by the marketing team, but the product team, and, and, and the product analytics team, we used the product analytics platform. I forgot the name of it was, it’s, it’s not one you would’ve heard of, but it was, it was.

[00:23:12] Bhav: Very different in what it did in terms of, in terms of product analysis. And it just made it easier for us to analyze data and user behavior. It wasn’t the best one, but it was still miles better than GA4. 

[00:23:24] Dan: I understand that. But then inevitably you’re going to get the whole, my report says this, your report says that, like, is that, is that like a. Does this multi tooled approach only work if you do have like an independent data team or a single source of truth, a database or a data warehouse that kind of has the business word reporting. I can imagine for the day to day activities, jumping into a platform and seeing some numbers is fine. And if they don’t match, it doesn’t matter.

[00:23:48] Dan: But I’m just thinking of someone let’s put back into the perspective of this, this scenario I was put in, in terms of, this client’s request saying, Hey, look, we’ve got no analytics on our products. We’re looking at what we’re doing. We’ve used GTM. We’ve used Google analytics. Should we just do the same?

[00:24:03] Dan: Like, should we just roll it out? And, and I’m just like, I don’t feel, I feel like I have a better answer of saying no, but like, no, I don’t think so. But actually maybe I like, it’s a really hard thing to say when. You know, like, it’s the same people using this product. So like there’s one analytics person at the organization.

[00:24:22] Dan: So it’s going to be on them. And if I, because now in my brain, in the moment of receiving this email, I’m going through, okay, well, if we have two tools, then there’s the technical debt. if we have one tool, it minimizes that. Okay. But if we have two tools, they can be better suited for the things. But then they might come with additional costs, right.

[00:24:39] Dan: And understanding like the kind of parity between all these things is there. And I know I’m putting you on the spot here and I’m not looking for every single answer within seconds, but is there, is there a best approach to this? Like, does it matter depending on the team size or say the maturity of the organization or the product management structure?

[00:24:55] Dan: where this may change your advice or how you would approach this. If you set up a Google BigQuery export from Google Analytics 4, and you’re not really sure what to do with it. It’s your first time using the GCP, then give us a shout. We’re here to help. We’re GCP certified, GMP certified. And of course I can’t stop talking about Google Analytics 4.

[00:25:13] Dan: So we’re here to help you make the most out of that data and show you exactly how powerful it can be. The world of cloud databases in the Google cloud platform can be complex and overwhelming at times, but ultimately it’s there to serve a purpose and to make the most out of the data you have, you can reach out to us by scanning this QR code, or you can click the link in the show notes to find out more, or you can just head over to measurelab.co.uk and find out a bit more of what we’re about.

[00:25:39] Bhav: Well, I mean, let’s talk about why the data would be different, right? Because you’re right it will be it won’t be the same if you have an instance of GTM that’s been installed on your site And you’ve got another instance of I don’t know amplitude installed the way it captures data is going to be slightly different So and then on top of that the way amplitude models data is different to the way ga4 models data So ga is sessionizing things if you want it to it’s creating rules and it’s creating concepts of bounce rates and Engaged users and then it’s modeling on top of that and all of those type of things And he and amplitude and heap and you know all these other platforms That during They’re doing their own things.

[00:26:21] Bhav: They’re doing something very similar where they’re creating concepts of session. They’re creating concepts of users and events and engagements, you know, whatever that might be, and they’re modeling in a way that makes sense to them. So, and you know, if you have got two very different instances of, you know, of your platforms being installed, you’re going to see different numbers.

[00:26:38] Bhav: I think most people will, I hope, be okay with it, provided they’re roughly in line now. It’s never, there’s always going to be some margin of error. That’s never, you know, that will occasionally cause one number to go down in one platform and others go up in another platform. I think that’s just, it’s just one of those things you can’t avoid.

[00:26:54] Bhav: So I think the way I would do this is think about the data collection priority and I, I, unfortunately I’ve come from a, I’ve come from a world where I prefer having a data pipeline that collects my data and then uses different end points. So the data I’m collecting. is the same and where I put it is the same.

[00:27:16] Bhav: So it could be going into my data warehouse. It could be going into amplitude or another product analytics platform. It could be going into my CRM platform. It could be pushing into Adobe analytics. It could be pushing into GA4. You’ll have to, I’ll have to, I need correction on this because I actually don’t know the answers.

[00:27:33] Bhav: Can you connect? Send GA4 data through platforms like Snowplow. That’s a really good question. We’ve had people from Snowplow on this podcast as well. We should know this. Anyway, regardless. So my, the way that I think about is the data collection side of things. And if you’re doing it in parallel, let’s say for example, you do have to have two separate setups, right?

[00:27:52] Bhav: The event is being emitted to GTM and then to amplitude. If the event being emitted is exactly the same. then it doesn’t really matter, if you see some discrepancies at the end, because that’s just down to the platforms. And I think as, I think as long as people can get comfortable with the fact that your marketing analytics platform is different to your product analysis platform.

[00:28:14] Bhav: They’re not designed to do the same thing. Your product analysis platforms is designed to tell you how users are using the site and your marketing analysis platform is designed to tell you how effective your marketing channels are. So if, as if you then try to cross over, you’re going to see some weird things, right?

[00:28:31] Bhav: But you’re never going to go to a product analysis platform and say, Hey, what’s my CPA on my, on my marketing campaigns, because you probably haven’t got your, UTM parameters set up for, for amplitude or heap when you, you have probably haven’t, you’ve probably ticked the auto tag tracking in, in Google ads or something like that, that allows you to automatically capture that information so. 

[00:28:52] Dan: This is a really good point because it does work both ways. Like the street is a two way street. Cause, you know, even like things like post hoc and other tools that we did a podcast episode on this a little while back of the The evolution of product analytics and tools and these kinds of things and how they expand, because a couple of these products literally just rolled out, Hey, look, we do marketing analytics now.

[00:29:10] Dan: And they released one dashboard that has like sessions and bounce rates and they pull in UTMs and I’m like, that’s not. That’s not it. Like I think it’s so narrow visioned from both sides. If you go into the Google analytics tool, then you think, oh, I can do a product analytics. It’s just events, right?

[00:29:24] Dan: Easy, done. And you go into the product analytics tool. It’s like, Oh, look, UTMs, sweet, bounce rates. There you go. There’s your marketing analytics done. And actually there’s so much more nuance than that. And I think these dedicated tools are definitely a worthwhile investment. One, one thing I think we’re getting towards a kind of resolution is like.

[00:29:40] Dan: Realistically, unless you have zero other choice. So Google Analytics 4 is not a good product analytics tool. And actually, I mean, we, we mentioned GTM a couple of times that obviously they’re not the same products, but they’re often talked together of using Tag Manager to deploy Google Analytics and things like that.

[00:29:55] Dan: Is there, is Google Tag Manager quite a common tool to be using within a product landscape or, or is actually natively coding and hosting these tools more common than using a tag management platform, much like you would on a marketing website? 

[00:30:09] Bhav: Even if you’re not using GA4 and you’re using a platform like Amplitude, you’re probably not going to do a direct native integration onto your code base.

[00:30:17] Bhav: You’re probably going to be doing it either through some type of tag manager or more common, certainly in my experience, I don’t want to speak for the entire industry. We’re doing it through some type of data pipeline. So something like snowplow segment, one of those, one of those like CDPs and, they’re the ones that are collecting that information and then pushing it to, the product analyst platform, because why would you want to send?

[00:30:40] Bhav: Why would you want to create a world where you’re doing a native implementation of amplitude or heap, and then you want to send that same, because remember it’s event based, right? So at the end of the day, you still want that data in your data warehouse. So you can stitch it together with your customer database.

[00:30:54] Bhav: So your, your customer data tables, or your, you know, your CRM tables, you know, whatever the other tables you might have. So then to have to capture it again, to send to your data warehouse doesn’t really make sense. So you’re better off. Using some type of data pipeline CDP to be able to push the data into multiple different endpoints.

[00:31:15] Bhav: And then that way you’re using the same, you’re still using the same event, but you’re using it. Let’s say, and I’ll give you a good use case example for it. Let’s say you want to send an email to someone who has performed some action on your website. You don’t want to have to set that up all over again for your CRM platform.

[00:31:32] Bhav: Ideally, what you want to do is you want to take that event that’s already been demitted. You want to push it into amplitude. You want to send that same event into your data warehouse. And then you also want to use that same event to trigger an email campaign. So any user who has done this, but who hasn’t made a purchase, send them an email.

[00:31:48] Bhav: You want to do that once as opposed to three different times. So I think the whole concept of product analytics is it’s actually more natively integrated with the rest of your ecosystem than something like GA4. GA4 actually stands as a bit more of a standalone product. Whereas product analytics, you have to think about your entire customers.

[00:32:07] Bhav: user experience. It’s not just the experience they have on their site. It’s the experience they have when they contact customer service. What have they been doing? Why did they contact customer service? Can we see what they’ve been up to that CRM platform? You know, what, what do we, we want to send them emails based on what they’ve done on this side, what they viewed, what they browsed.

[00:32:27] Bhav: And you don’t want to have to keep setting up something new every time. So in my world, actually, GA4 is the kind of, Alien entity that sits on the website. Everything else has much more native integration with everything else 

[00:32:40] Dan: Yeah, no, I like that. And, you know, we’ve talked about this with a couple of guests over the last couple of months, but I wonder if GA4 is the ambition of GA4 is to move more that way, because we’re seeing more connectors, more ingestion methodologies, more, more things like that.

[00:32:54] Dan: You know, there’s even this rumored sort of Facebook. Facebook advertising sort of meta pixel integration with GA4 coming and GA4 is integrating with Salesforce and, you know, AB tasty and Optimizely and things like this. I think it’s an interesting thing. And anyway, sorry, I’m just riffing on that really, because I’m, I’m, I’m doing, like you said, GA4 is the odd one out. But it’s such a highly adopted tool that, it’s, it’s kind of the most well known outsider. 

[00:33:21] Bhav: Yeah, exactly. Let’s think about the platform again. if we think about GA4 as a platform, even though it sends events and captures events in the same way heap and amplitude do, forgetting that it’s front end events or back end tagging, you know, or server side tagging, whatever you’ve got, the problem then still comes down to the platform.

[00:33:40] Bhav: So you’re probably right. At some point in the future. GA4 will be integrated with all of your kind of like your CRM platforms, everything like that. So you’ll be able to create a nice, if then, if this, then that type situation, that you can do with, with, with platforms like Amplitude and heap and, and other product analytics platforms right now, for me, the biggest challenge with GA4 is the platform itself.

[00:34:00] Bhav: And, and I’ll give you an example. If you go on to measure Slack for those, if you’ve, for any listeners, by the way, if you have any GA4 questions, go to measure Slack, they, there’s just absolute GA4 geniuses on there. I was trying to create it GA4 explore, a view that allowed me to create a segment of users who had done some action and just see those users sessions where they did the action.

[00:34:24] Bhav: I can’t remember exactly if I’m getting this right, but it physically was not possible. It’s like, it’s not, it was, it’s physically at this moment in time, not possible to be able to do that analysis. In order to do this, what you have to do is you have to go to BigQuery. You have to create a common table expression of all the users who have done this action.

[00:34:45] Bhav: Then you take those lists of users, and then you look up, then you write your next set of query. And then you look up the users against that first common table expression, where you’ve got your list of users. That’s how you do it in something like a product analytics platform. You can just do that in the interface.

[00:35:01] Bhav: Now think about if you’re a non technical analyst or you’re a non technical marketer, or, you know, you’re someone who can’t write SQL. But you want to do this analysis, you can do it in these platforms already. It’s just, you can’t do it in GA4. So right now, for me, the biggest challenge, regardless of data accuracy and sampling and all that kind of, all of those other types of things, the biggest limitation for me right now for GA4 is the platform. It still looks like a very childlike platform when it comes to doing product analytics. 

[00:35:34] Dan: Hmm. Yeah, no, no, it’s very Google, isn’t it? In that way where you’ve got one button and you kind of need five. so, okay, there’s, there’s one, there’s one last aspect to this, but, again, I’m not necessarily looking for answers, but came up and I didn’t know how to address this.

[00:35:49] Dan: So, and this is the idea that, I was actually chatting to someone today about this, that about the old concept of like being an analyst or being in analytics, and I kind of didn’t expect 10 years ago that my, most of my life will be spent talking about cookie banners and consent management yet. It is.

[00:36:04] Dan: And so, the question comes up is when you’re inside of a product, do you have the same, do you have to consider the same things? Like, do you still have to serve a cookie banner? Are you still, beholden to the same restrictions, even though they are a customer of yours and they have logged in or. Does it depend?

[00:36:18] Dan: And if so, what does it depend on? Like, how does, how does that, how does, how do companies that had a product teams address the whole concept of cookies, non essential cookies and tracking and things like that? 

[00:36:30] Bhav: So in our instances, a lot of the stuff we do is not sort of like third party base, so it’s not going to some type of ad network. So I think, and again don’t quote me on this, if you know, if you work in this space, I’m sorry, I’m not an expert when it comes to cookie laws and privacy. I just avoided this topic with a 10 foot, you know, like I’ve just avoided this topic completely. So I think as long as the data being captured is being used for the purposes of creating the experience, I think it’s okay.

[00:37:00] Bhav: In some instances, you, you physically can’t use the platform because the platform needs to know who you are, what you’ve purchased, all of those, what you’ve done to be able to create that view. So I think in those instances, it’s okay. I need to look a bit more into this. So I’m not going to give a, a, like a very comprehensive answer on this one, but I think generally for the purposes of.

[00:37:20] Bhav: If as a customer, what you’re capturing, if it’s being captured as a front, as a, as a first party data, as if it’s been captured as first party data, I think it’s okay. 

[00:37:32] Dan: This is the thing that’s, that’s tripping me up. Like I say, if it comes to a marketing website and it’s all used for marketing purposes or analytics or whatever, I’m kind of all over it when it comes to in product.

[00:37:41] Dan: I feel like, you know, if part of your license for signing up to a product or logging into an account is that’s all covered already. then. It could be a non issue, right? It could just be like, just roll in, do your thing. like you said, if you’re using these pipelining tools, if you’re using like a, a rudder stack or a segment or a whatever to kind of pull this data in, then, you know, it kind of depends on the use cases, right where you’re sending that data. Yeah. Like pulling in logs, isn’t an issue. 

[00:38:06] Bhav: Exactly. Like, let’s say for example, Going back to my time at Hopin, Hopin is a virtual conferencing platform. If you’re a first time listener and. We used to capture events. Let’s say for example, you’re an event organizer and you were building an event.

[00:38:22] Bhav: In this instance, this is going to get confusing. I mean, event as in a conference. So you’re building a conference and you want to run a virtual conference. You have to go onto the, you know, new conference page, new event page, create conference, add in the conference name, set your criteria. How many attendees does it have to be RSVP?

[00:38:40] Bhav: Do they need tickets? Is it free? Did all of these things are essential data points for us to capture, to help understand? if something goes wrong, how the product is being used. So we can, it’s not being used for marketing purposes, right? So we’re not using it to then send, you know, sell this information.

[00:39:00] Bhav: It’s information we’re keeping to understand how the product plot. And I think if I can’t, it’s been a while, but I think for those types of things, We were able to get away with it because we were capturing that information anyway. We’re just not pushing it to some third party ad network. We’re just keeping it for ourselves to understand what users are doing because we need a record of that, of that event being created with the details of what the, the setup of it was like.

[00:39:23] Bhav: We needed to do it. I used to work with a client who, they’re a pension tech company. Once you’ve registered, it’s all within the app, right? Now, let’s say you wanted to view your pension. Let’s say you wanted to add money to your pension pot. Let’s say you wanted to pull in your pots from other, providers, that’s information they have to, they had to capture.

[00:39:42] Bhav: It wasn’t, it wasn’t optional information that, you know, the customer could say, well, I don’t want you to know what my pots are and how much money is in there. It’s because, well, we have to know, right? We have, or not we, the company, they had to know, they have to collect that information to be able to understand how people are using the pots and what features of the pots are they using with, are they on high, you know, high risk.

[00:40:06] Bhav: Investments or are they low risk or medium? And all of this information is part of the entire end to end user experience. It’s not something that they were then going to push to some ad network saying, Hey, you can now retarget these users because they have a lot of money in their pension, but, or, you know, they, they’ve only got one pension pot.

[00:40:20] Bhav: So I think I’ll, I’m going to look into this, but I think when you’re talking about information that’s being captured to improve the user experience. Genuinely, I generally think that’s okay. 

[00:40:34] Dan: Okay obviously this is legal advice. You can take it as such. No, of course I’m joking. This is the complete opposite of that. This isn’t legal advice. Seek out legal advice. If you are uncertain, this is just our opinions and riffing on what’s going on. And it’s always important to state that just, just in case there’s anything that comes back to bite us in the ass about this. I don’t, I don’t think there’s any, anything else I’m.

[00:40:55] Dan: I have lined up or at least on the top of my head around this kind of like vetting process for product analytics. And my question, I suppose, if I just pass it back to you for like final thoughts on this, like if you’re in my shoes and, and someone comes up to you and says, Hey, look, we’ve got this product, the product team alike, we should do analytics and they’ve got nothing on there right now, where would you go?

[00:41:14] Bhav: I would start, I mean, this is an easy one for me, because this is what I’ve done in the past. I would start off with an RFP, right? So a request for proposal. That sounds very formal and document. Build a list of requirements of what you need, right? So does it need to be 100 percent accurate? It’s the same as if you were doing an A B testing platform.

[00:41:32] Bhav: Say you’re thinking about buying an A/B testing platform. Don’t go out and speak to the vendors first. Build a list of requirements first. This is the same with your product analytics, because you might find that actually GA4 is perfectly fine for, for your product. It’s just, you don’t know that yet until you’ve asked the right question.

[00:41:50] Bhav: And those questions are going to be around things like is sampling an issue. If it’s an issue, then, okay, we probably should, we need to consider that. and that consideration could be a new platform or it could be three, BigQuery. Then you want to look at how important are going to be things like segments and cohorts and LTV and usage and number of events you’re tracking, how much you want to track.

[00:42:13] Bhav: Do we need to push that data into big, into a data warehouse? That’s not big query. Cause it might be that actually you might have GA4, but your entire data warehousing ecosystem and infrastructure is on AWS. In which case you might think, actually, no, we don’t want it in BigQuery for, because we’re not, we’re not on a Google, we’re not on the Google ecosystem.

[00:42:36] Bhav: You might want to think about, do we need A B testing? Are we going to be connected? Do we need to send this data downstream to other places? What do we need to do? Whatever it might be the million different things asking those questions will then allow you to start off in the right place. You know things like are we going to have resources going to be a big one?

[00:42:54] Bhav: how much resources do we have to rebuild because if you’ve already got GA for chances are a lot of the Core events are already built in. If you’re thinking about product analytics platform, it’s likely that there are a bunch of custom events you need to build, and you might need resources for that.

[00:43:12] Bhav: Also, then it’s a case of, okay, well, if we’re creating a new platform, we’re going to be capturing this data from zero. GA4 has like three years worth of history in it. Are we okay with starting from zero all over again? So it’s, I think an RFP type. Approach is the best place to start. Think about your requirements, what you need.

[00:43:31] Bhav: And, you know, you might not have all of the questions down, but if you’ve got most of the questions, the types of analysis you want to do off the back of that, you know, all of these things should feed into your, Into your requirements and then you can make a more informed decision. 

[00:43:44] Dan: Like anything, figure out what the purpose is before you start running and charging ahead with anything. I like that. It’s, it’s, it’s kind of like, it’s important to state this. Cause often, including myself, I forget, you know, just take a step back, take a breath and actually think about why we’re doing something, not just what the shiny new object in the distance is. And can we go play with it? You know, I think that’s really, that’s really good.

[00:44:04] Dan: Is there anything else like. But, if, okay, rather than advising me, which this podcast has been about thus far, if, if you are right now able to speak to this person that asked me this question saying, Hey, look, I’m the only analytics person in this organisation. People are looking to me to recommend and implement and resource this.

[00:44:21] Dan: And I understand your point of view that Google analytics isn’t a best case solution, at least for this, or it might be, but we need to go do some digging. Is there anything else that you would advise them on or suggest to them in terms of going down this journey or path? 

[00:44:35] Bhav: If the person you’re speaking to is a marketing person, I really think it’s going to come down to, do they have buy in from the organization to not just purchase a new platform because chances are the marketing person, assuming they’re a marketing person is using their own budget, but do they have buy in from the rest of the organization to support the integration of a new platform?

[00:44:57] Bhav: That is the biggest, that is the biggest hurdle I’ve seen because the amount of times I’ve worked in organizations where marketing team have run off and purchased the platform and then turned around to the product team, the data engineering team, the analytics team saying, Hey, can you go in, can you make this work?

[00:45:12] Bhav: It’s like, well, no. Right. Because they’ve, what they’ve, what you’ve done is you’ve gone and signed up to a 12 month subscription without realizing that it’s probably going to take you six months to get to a point where the platform is even operational. So I would think about like speed versus coverage.

[00:45:30] Bhav: If they want something fast, they’re not going to get high coverage. If they want high coverage, they’re not going to get something fast. Right. So those are kind of like the two way ups that I would have when they’re thinking about this. I don’t know if it’s the best advice. It’s a, it’s a, it’s, it’s such a complex question is there’s so much gray area into this.

[00:45:47] Bhav: I budgets is going to be a massive thing about how much budget do they have to play with? Because each one of these platforms is very different. also if they start off with something like postdoc, which is a great platform, but it’s inevitably free, will they have, you know, if they invest in art, you know, a whole bunch of time Building out that platform and then they hit the, kind of like the, the proverbial wall of like what the limitation of the free platform can provide, are they going to get the resources financially to then go in, buy out by the, you know, the rest of the platform or will they have to move?

[00:46:17] Bhav: If they have to move, did they consider it? Doing the event tracking through some type of CDP, or did they use posthogs to implement? Did they implement postdoc on the site? Because then they have to do it all over again with a new platform. So honestly, I don’t think there’s a perfect answer to this question.

[00:46:32] Bhav: I really think it starts off with what are your requirements? Do you have the metrics you want to measure in place? I think I, we, you and I have talked about measurement frameworks a lot. I think. Understanding your requirements is obviously important, but understanding your measurement framework, your metric tree, your business model in great depth is the most important thing, because you might build out that metric tree and you might find you get to like the lowest common, you know, lowest level denominator in terms of event tracking and you realize you have it all in GA4.

[00:46:59] Bhav: What you don’t need, you don’t need a new platform, you just need someone who can query it in BigQuery. Because there are other platforms which are warehouse native product analysis right? Let’s say you don’t want to commit to a an ecosystem, you could go and set up your data pipeline and push your events and then get just a tool that just takes that data and analyzes it, right?

[00:47:18] Bhav: It doesn’t do anything shiny, but it takes it so that it makes the data available to the everyday person, who can’t, who can’t code in it. and, and I forgot the name of the platform, but it’s a warehouse native product Is it’s really cool. You know what they do. And I think this is where Amplitude are moving to Amplitude are moving to this warehouse native approach as well, because I think they realize that a lot of companies want to own their own data.

[00:47:43] Bhav: And what they need is a tool to visualize that data, which is why platforms like Amplitude are so great for, you know, Using that data and GA4 is so bad. 

[00:47:52] Dan: Yeah, for sure. And I think this is really interesting because it brings me back to that. actually the first episode we had with Adam Greco, who was our last guest.

[00:47:59] Dan: The first episode was about warehouse native analytics and exactly the same situation here, which is that, you know, we don’t need 15 different SDKs and trackers on something. Actually, we just need like a warehouse native approach to something and we can feed that data out. So, yeah, I’ll put a link to all these episodes and some of the other ones that we’ve talked about, product analytics, if this is something you’re, then it could be a good way, place to start.

[00:48:20] Dan: and I’m your kind of, you can, you can learn vicariously through my dumb questions around, product analytics coming from the world of web and marketing and digital and not really understanding, even though it’s the same stuff when using often the same tools and the same applications, sorry, different applications with all this stuff.

[00:48:35] Dan: So, I think that’s it. I think that’s kind of it. I think I know what I’m doing now in my situation. I hope the other people that are listening, have a bit of a flavor of how these, you know, tools and their approaches and these, you know, still working with analytics, but in a different context is actually quite a different sort of thought process and mentality.

[00:48:53] Dan: the landscape of tools, different, the use cases are different. but I think there’s an under fundamental kind of commonality there around, you know, think before you run, do you have a framework in place before you start jumping in? Do you have an RFP before you start spending money? Is everyone aligned?

[00:49:09] Dan: You know, what is it you want to measure before you start tracking stuff? Like, what is it? And I think these are things that are underpinning all of us analysts in a way. And I think that’s really good advice, just even if it’s stuff you already know, just to say out loud and hear it back again. So, appreciate that.

[00:49:23] Bhav: That’s a very nice summary. Thanks, Dan, of what we discussed. I, I do, I do agree with, analytics is not something you can. Leap before you look into. You really need to look and understand and craft and build. I’m also, that’s not to say you need to take two years designing and implementing, but a lot of prethinking ahead of time will save you a lot of pain further down the line.

[00:49:48] Bhav: So yeah, just give it some thought ahead of time. And then I think you’d be on to the right track. And also if anyone’s listening, I think you’re going to either be very, like, you’ll understand what I’m saying in this episode, or you can be very confused either way. if you have any further questions to reach out to me, I’m more than happy to talk about things.

[00:50:03] Bhav: I think. This episode could probably do with one of those like mini PDF write up books, you know, like people publish those eBooks. I think I genuinely think this episode could be an eBook. 

[00:50:15] Dan: I tell you what we do is we’ll use the notebook LM, the Google’s notebook LM, and we’ll put this episode in and we’ll get it to generate a five minute episode.

[00:50:22] Dan: You know, with those, the American hosts, you know, the kind of generative AI podcast, have you seen those before? 

[00:50:27] Bhav: I’ve not, but I’m, I’m fascinated. 

[00:50:29] Dan: Oh my God, notebooklm.google.com. I think that’s the URL. I’ll put a link in the show notes as well for us listeners as well. So, you upload files, URLs, documents, clips, audio, whatever you want to do.

[00:50:42] Dan: And it, and essentially it works as a generative AI chatbot that you’ve just trained and it’s just learned on all the content you’ve given to it. But one of the nice little side features is that it can auto generate a podcast episode based on your content. and it’s got two hosts that just have a.

[00:50:57] Dan: Backwards and forwards conversation, discussing your information. And, what I find this fascinating is that we’re doing a podcast of this. I wonder if it can just summarize and redo our podcast in a more coherent way. I think that could be an interesting exercise. So, yeah, let’s see all right, let’s end it there.

[00:51:12] Dan: Thanks everyone, we’ll be back in a couple of weeks, maybe with a guest, maybe without, who knows? It depends, how quickly me and Bhav can figure out, figure that stuff out. All right. reach us on, measure slack, reach us on the CRAP Slack community, reach us out on LinkedIn as well. 

[00:51:27] Dan: These are all good ways to connect with us. otherwise I’ll see you on the next one. That’s it for this week. Thank you for listening. We’ll be back soon with another episode of the measure pod. You can subscribe on whatever platform you’re listening to this on to make sure you never miss an episode.

[00:51:39] Dan: You can also leave us a review if you can on any of these platforms. Forms. We’re also over on YouTube. If you want to see our lovely faces and our lovely guest faces while we do this as well, make sure to subscribe to the measure lab channel to make sure you never miss an episode as they come out, if you’ll leave us a review, that’ll be hugely appreciated.

[00:51:55] Dan: You can do that on most of the podcast applications, or there is a form in the show notes, you can leave feedback directly to me and Bhav. Thank you for listening. And we’ll see you on the next one.

• Spotify
• YouTube
• RSS Feed