#91 Warehouse native analytics (with Adam Greco @ Amplitude)
In this week’s episode of The Measure Pod we spoke to Adam Greco from Amplitude. We talked specifically around warehouse native analytics, and whether it’s a passing trend or an integral part of the future. We also go his thoughts around marketing and product analytics, as well as analytics vendor consolidation.
🎥 The podcast is now available in vodcast (video) format! Watch the episode below, or over on YouTube.
Show note links:
- Please fill out our Feedback Form to let us know what you’d like us to talk about next season, or email email@example.com to drop Dan, Dara and Bhav a message directly.
Find out when the next CRAP Talks event is happening on LinkedIn.
Music composed by Confidential – check out their lo-fi beats on Spotify.
Master Google Analytics 4 with Daniel Perry-Reed on the next GA4 Immersion 6-week cohort training course. Early bird, charity and group discounts available!
Quotes of the Episode:
- “…I call it the last mile problem where you can have the best data quality, best implementation but then if you can’t get people in the last mile to use the data then everything you did is for nothing.” – Adam
- “…I think the problem with you know, being a complete one vendor marketing specialist or analyst, whatever it’s going to be, it means that you kind of alienate yourself from everything else that’s out there and you never become fully adaptable.” – Bhav
The full transcript is below, or you can view it in a Google Doc.
[00:00:00] Dara: Hello, and welcome back to The Measure Pod. We’re into a new season. Good to be back, Dan.
[00:00:21] Daniel: Yes, yes. It’s been well, in terms of recording, it’s been a minute. So it’s nice to be back in the, back in the hot seat, back in the chair, doing these kinds of things. This one was an incredible conversation with Adam Greco from Amplitude. One of Bhav’s favourites, I think he’s been wanting to talk to Adam for a long time on the podcast, ever since we actually were talking to Bhav about this and Bhav’s not here as you can maybe tell, he had to drop off halfway through or just towards the end through other commitments as his life nowadays on remote working.
[00:00:46] Daniel: So you’ll hear Bhav introduce Adam properly in a moment. But yeah, it was a great conversation and I felt like we covered a lot of ground, basically talking about Warehouse, Native analytics, vendor consolidation, product and marketing analytics. I think it’s, it’s all kind of, it felt weird, doesn’t it? It was all kind of the same conversation yet it felt like we covered a lot of ground.
[00:01:04] Dara: Yeah and Adam’s such a knowledgeable and experienced guy. And actually one of the things I thought was the most interesting, I mean, obviously there’s a lot of detail around, you know, his views and our views around marketing analytics, product analytics, warehousing. But I completely agree with what he said about how the industry hasn’t actually really changed much in 20 years and all of a sudden, and not, not just within the industry itself, but I guess with AI coming along and shake in every tree, if that’s the right thing to say, something. It’s created this kind of sense of like, it’s a mixture of excitement and probably a bit of fear as well. And fear is, you know, sometimes a good motivator. I’m not sure if you’re supposed to say that, but it is.
[00:01:43] Dara: So I, I completely agreed with him on that. You know, we’ve got so used to things being the way they are. And now it’s actually quite an exciting time to be in this space because it’s all up for change really. And you know, how AI is going to fit into this as a whole other conversation. I’m surprised it didn’t come up more than it did in this, in this conversation actually.
[00:02:02] Daniel: Well, let’s not give too much away. Let’s jump straight into the episode. Just before I do just to say, yeah, we’re back for a new season of The Measure Pod. We’ve got 10 episodes coming up. We’re leading up to the magical or the milestone 100, which yes, we do have some fun stuff planned, which Dara has no idea about, so look forward to that one. Just to let you know that we also have started doing a mini series on every Wednesday now. So these main episodes drop on Fridays. We’ve got Wednesdays with our soundbites short form clips from our previous guests talking about something that’s come back into the light in terms of the industry changes that are happening so stick around and subscribe to get those in your feed.
[00:02:36] Daniel: And lastly, join us over on the CRAP Talks Slack community. If you want to get in touch with me, Bhav, Dara, or anyone else and join the CRAP meetups in London, Manchester, and so on and so forth there’ll be a link in the show notes, use whatever app you’re using to access those and we’ll see you there. So enjoy the episode.
[00:02:53] Dara: Enjoy.
[00:02:54] Bhav: Hi all, and today I’m really thrilled to introduce Adam, who is not only a former Crap Talk speaker, I’ve seen him speak multiple times by the way, he’s a fantastic speaker. If you do get a chance to go and see Adam talk, his talks are wonderfully engaging, very funny, and filled with pearls of wisdom. I’ve actually been following Adam for many, many years. When I started, I pretty much read everything that he wrote on Analytics Demystified. So if you haven’t had a chance, I’m not sure if they still rank high on the Google SEO rankings, but I pretty much read most things on Analytics Demystified.
[00:03:29] Bhav: Adam currently works as a product evangelist for Amplitude. I won’t go and do all of his intros for him, but I just wanted to say it’s really great to have Adam on the show. I love speaking to him. I know Dan and Dara have heard him speak in the past. You know, we’re all big fans. So hi Adam, welcome to the show.
[00:03:45] Adam: Oh, thanks for having me. And I’m a big fan of this show. I’ve listened to most of the past episodes, so really excited to be here.
[00:03:52] Daniel: One thing I saw that I want to jump straight in and pick your brains a little bit on is I saw you do a webinar for Search Discovery or now as called Further. They have a nice webinar series they do all sorts of interesting talks and God, it feels like a lifetime ago now, but this is back in July this year, you talked about Warehouse Native analytics, fad or future. For everyone that may not have come across that we’ll put the link in the show notes to the video on YouTube so they can watch that. But give us a quick highlight, because I’d love for you to talk to us about this idea of native analytics, first of all, and whether or not you, your opinion has changed since July and whether it is a fad or the future.
[00:04:26] Adam: Yeah, absolutely. So I think in order to understand Warehouse Native analytics, you have to take a little bit of a step back. And, you know, for better for worse, I’ve been in this industry for about 25 years. And so I’ve seen a lot of fads and stuff come. But if you think about way back in the early days of digital analytics, most of the analytics happened in log files. You know, you actually used to store all your web traffic at your own company.
[00:04:53] Adam: Then we went through this whole SaaS revolution where, you know, Urchin, web trends, Omniture, where I used to work, basically send all of the data to the cloud and you would just log in and run your reports against a server that you didn’t own that you basically just rented. Now that’s kind of been fine for the last 20 years or so, but you may have heard that there’s some things going on with privacy and there’s a lot of other things going on in terms of data security and privacy and so on. But now, most companies that use analytics tools, whether it’s Adobe or Amplitude or Google because data retention is narrowed, a lot of people are basically backing up their data into warehouses like Snowflake, Redshift, and that’s pretty common. So these warehouses are getting more powerful and they’re getting faster.
[00:05:44] Adam: And so the idea of Warehouse Native is what if we didn’t have to send data to a SaaS provider and what if we bypassed that and just sent it directly to our warehouse where, by the way, we have all of our other customer data that not doesn’t include just web and app. It might have store data, it might have call centre data, and what if we could run all of the reports and dashboards and queries that we’re used to in our favourite marketing or product analytics tool directly on the warehouse?
[00:06:18] Adam: And if those reports could come back fast enough, then there’s a whole host of benefits. You don’t have to pay for storage on a SaaS provider. You don’t have to worry about if your SaaS provider is GDPR compliant. And Imagine being able to build a report or dashboard that used all of your customer data from call centre in store, then you would truly have customer journey analytics. And imagine if you could build a segment or cohort of users that say, I want to look at people who did this on the website, this in the call centre, this in the store and in traditional marketing and product analytics tools, that’s a little bit tricky because you don’t have all of the data.
[00:07:01] Adam: And if you were to use your marketing or product analytics tool as a data warehouse, that becomes one of the most expensive data warehouses in the world because Snowflake is built for that purpose. So that’s the premise behind here is just, do you want to have another silo of data, which is your marketing or product analytics silo, or do you want to have it all in one place? And I’m sure some of you have, you’ve all experienced this where your BI team runs a report. And then you go into like, say, a Google Analytics, you run a report and those numbers don’t match up because really Google Analytics is a subset of what’s in your warehouse. But if you’re querying right on the warehouse, unless you really screw up the data should be exactly the same. And that way everyone’s kind of singing from the same dance sheet or song sheet, if you will.
[00:07:52] Adam: So that’s the first part. I’ll pause there before I give you my updated opinion, but any questions on that, does that all make sense?
[00:07:58] Dara: Just listening to you describe that, I’m just wondering like, what’s not to like about that? Why, why would you. Yeah, why would you not do it that way? Why would you continue to have this siloed set of data that’s going to create those headaches when it doesn’t match the other systems that you’ve got, the other data sources that you’ve got? Apart from tradition, I guess, and habit.
[00:09:32] Adam: Obviously speed is another downside. So products like, you know, all the big ones, Amplitude, Adobe, Google, they’re, I think about Mixpanel, like all of those databases are optimised so that when you do a query, you get results in like milliseconds. Like you blink and you’re already seeing the data. Warehouse Native is not going to be that fast, now I think it will be eventually, but what’s really happening behind the scenes of Warehouse Native is you’re running a report. The Warehouse Native is changing that into a SQL query that matches the SQL language of that warehouse. It’s querying it and then it’s reformatting it back into a report that you’re used to in an analytics tool.
[00:10:12] Adam: And as you know, some of the analytics tools which have really complex reports where you’ve got a conversion funnel with a lot of cohorts and you’re doing, you know, hold this constant, you know, that could be a long query that takes a while to come back. So I think long term there are more pros than cons. But initially you might be annoyed that something that used to take you five minutes to go get the developer to just say, hey, I need to track how many people complete a form, and then you have to wait six months to get that. But I think at a small company, you could just do that pretty quickly. But at a large company, which are the ones where I think Warehouse Native makes the most sense because they have the biggest investment in the warehouse, you could get in this like in queue for months that you’d be annoyed, and that’s, I think the biggest hold back right now.
[00:10:58] Bhav: Adam, you mentioned that one of the benefits of going Warehouse Native is to de-silo data and make it available to be, you know, used joined with other data sets. In your experience, do you find that people, you know, conceptually, do people want to do that anyway? So for example, would a product manager who’s working predominantly with behavioural data even want to join it with that kind of like data that comes from the CRM platform or something like that. So whilst, and for the record, if anyone’s listening, I am pro Warehouse Native, I’m just trying to play devil’s advocate here as well so I’m not too biassed towards what I really, really think the world should look like. So Adam, yeah so from your point of view, do you think that there is an appetite for de-siloing data?
[00:11:44] Adam: I do think there is, but I think that in a weird way, we’ve all been conditioned that to believe that that’s not possible. And so we’ve lowered our expectations, but I’ll give you an example. Let’s say that you’re a retailer and you work on the product team or marketing team, a retailer. And let’s say that Adam goes in and buys a thousand euros of product. And you have your Google keyword, you know what led to it, you could see the flow, you could see everything, you’re all good.
[00:12:13] Adam: But now, Adam returns 900 Euros of it. Most organisations, unless you’re really advanced, are not ingesting the return data into a marketing or product analytics tool, it’s possible, you know, in Adobe, there’s this thing called transaction ID and Google, I think it’s measurement protocol. Like there’s lots of different ways to do that, but most people don’t because it’s not, it’s more of a warehouse function. But if you were doing the Warehouse Native, you would be able to see much more accurate data to say, hey, this campaign code looked awesome, but it really isn’t when you take into account returns.
[00:12:47] Adam: Another example, my favourite example is imagine someone goes to the call centre. And they just have an awful experience. They hate your brand right now, but they need to come back to you, classic like cable provider. So they come back to you. You could then, if you had the call centre information, you could have a promotion on the homepage that says, hey, sorry, we sucked last time, let us make it up to you, maybe running through an A/B test. And that’s something that’s really difficult to do if you’re not looking at the whole customer picture, but if you do, you just made a better customer experience, which is really what all of us are in the business to do.
[00:13:24] Daniel: So Adam, there’s a thing that I was thinking of as you were saying that. And I think this is, you said about like enterprise companies are going to be the best place to invest in this. They’ve got the kind of the budget, they’ve got the people, the teams, maybe. But is there a chance that someone can be successful without going down this path? Like, can they stay within the product specific tools, the marketing analytics tools, the product analytics tools, can they still be successful there? Or does that naturally come up to barriers should we say that they just can’t kind of progress or become more advanced without going down this road here?
[00:13:55] Adam: No, I certainly think you can be successful, but I think at scale where it becomes tricky is that you end up, if you really want to see everything about a customer and you send all of that data into your marketing or product analytics vendor, I mean, they will love it. I mean, as a shareholder of Amplitude, I would love it if you would send all that information in because we get paid by events. But I think that it just does get a little bit expensive and I think that there’s other groups at the organisation that will laugh at you if you say that our enterprise data warehouse, where we see everything about a customer is in a marketing or product analytics tool, like that’s not really what they’re meant for. That you could do it and it’s possible, but I think that’s why so many organisations have invested in warehouses. And even if you’re a small company, you could rent a partition of like a Snowflake or, you know BigQuery.
[00:14:52] Adam: And I think that what I am actually predicting is going to happen is I think that marketing and product analytics tools are ultimately going to become the front end of warehouses, and I think that there is just going to continue to be consolidation. If you look at Google Analytics right now, GA4 is kind of in a weird way, just the data collection and reporting mechanism of BigQuery, and Google ads like that’s really what GA4 is now, in my opinion, but I’m a little bit biassed.
[00:15:25] Adam: But I think that that’s what we’re seeing. And I think that if you look at what Adobe is doing, Adobe with their AEP, they’re basically saying, don’t use Snowflake. We want you to put everything into Adobe, but that’s really, really expensive. And I think Amplitude, like we’re trying to be a little bit more in the middle. Like, hey, if you want, you could use us if you’re a small company, but if you’re a large company and you have a warehouse, then you should be able to use your warehouse, pick which provider you want to use, and we believe that we should not dictate that it should be up to the customer.
[00:15:54] Bhav: I completely share your view on this one, where I think certainly for GA4 it’s just the holding for the raw data to be pushed into BigQuery so that people can access it. And, you know, Google can start to build up their own sort of like cloud offering and increase revenues I think that’s where it’s going. You mentioned time, you mentioned speed of results, performance issues, and getting services.
[00:16:14] Bhav: But there’s also other factors that out of the box platforms already do for you, such as aggregating data and creating models that make it easy for people who aren’t SQL experts to do. Like, do you think that we’re entering a future where people will have to become SQL experts?
[00:16:33] Adam: Well, I think the whole point of the Warehouse Native is that you don’t have to be a SQL expert and that the interfaces are masking the SQL for the masses and so that you could provide self service. And I know I don’t want to trigger you, Bhav, because I went through a whole episode of you talking about your failures of self service which weren’t really failures it was kind of a humble brag. But I think the thing is, is that if you, the packaged tools basically give you a structure, they give you a data model, you know, events and properties with a Warehouse Native, you actually don’t have any limits, like you could have whatever structure you want, which is a blessing and a curse, because if you’re a small company, you’re an immature data team, that could be really intimidating.
[00:17:18] Adam: But if you’re a sophisticated data team, you’re like, listen, I don’t want to worry about user properties or event properties. I just want to have columns and rows and I could do whatever I want. So I think it’s a double edged sword. And that’s why I think the most advanced companies, and I can tell you at Amplitude, the most advanced customers that we have are the ones that are really intrigued about this Warehouse Native because they’re like, listen we have our own data models. We have our own people who know how to do queries and they’re comfortable doing SQL, but they also have people that want to self service. And so they love the idea of this hybrid model that they could do SQL, but that people who don’t know SQL can do that. So, yeah, it is a little tricky, but I think that most Warehouse Native vendors that I’ve seen, the whole point is that you build a report in a very easy GUI and it basically masks that and gets you the results without you having to know SQL, which is cool.
[00:18:10] Bhav: Do You think this is kind of like the final wave of people needing to learn SQL? Because it’s, we are 18 minutes and three seconds in, and I’m going to drop the word AI into this conversation because it’s you know, I’ve been, I’ve been trying to challenge myself to think about what the future of analytics looks like, what analytics and experimentation, and I really feel like, you know, when one of the, one of the biggest use cases of AI or large language, you know, LLMs is going to be the ability to ask questions of the data in human format and get answers.
[00:18:47] Bhav: I guess where I’m going with this is all those front end interfaces that we currently need right now to do that number crunching going to be needed you know, three years from now, four years from now, five years from now, or will AI and machine learning be able to handle all of those stuff because we’re already starting to put all this data into, into a data warehouse. There are already tables there, we just need to ask it in a very easy to understand way to say, hey, extract this data for me and put it into a nice format and even find something, you know, tell me some insights about it.
[00:19:18] Adam: Yep, exactly. And that’s why I said earlier that I do believe that these products are going to end up being a front end to the warehouse. And I think if you can just ask a question, you know, the warehouse has all the data. If your question can be translated by AI, in language models into SQL, then why do you need an interface? And I think that one of the things that Bhav you talked about is your struggles with self service. And I’ve seen this because I’ve trained people on analytics tools is I call it the last mile problem where you can have the best data quality, best implementation but then if you can’t get people in the last mile to use the data then everything you did is for nothing. And so I think that it is interesting, and I think one of the things that, like, we’ve spent a lot of time thinking about at Amplitude is, like, what if no one had to run reports, you know, in the future?
[00:20:14] Adam: What is our offering? What is our value that we provide? And that’s why we focus so much on data quality and stuff like that, but it is interesting to see. I do think AI is going to make it so that anyone can ask any question and just get the correct chart and graph and report that they need. And I do think that someone is going to come up with a way to easily translate english, you know, typed language or even voice into SQL. But as you know, SQL is a little bit different for Snowflake than it is for, you know, Redshift. There’s like different flavours of SQL. So you’re going to have to have like a, it’s almost like a Google Translate for SQL that will have to know you’re using this warehouse. So when I say something, you know, my SQL needs to compute in something that will basically report back to Snowflake if that’s what I’m using.
[00:21:03] Adam: So, yeah, it’ll be interesting to see. And I think that a lot of the UI that people have been spending time on you know, might go away, which would be really good for companies like GA4, which, you know, as you know, has a horrible UI. So if they can solve the UI problem, then that would benefit them.
[00:21:21] Daniel: For sure, I completely agree. And we’ve been playing around with GPT or chat GPT and using the connectors to things like Zapier. We’ve plugged that into BigQuery directly and created custom GPTs around giving it the context of the SQL for BigQuery right. And obviously that’s not going to work on other warehouses there, but it’s absolutely fascinating.
[00:21:38] Daniel: I’m so curious as to see where UIs will go down the line with this, because if you can just sit in MS teams or Slack, query the data that’s been pre trained or prompt with by analysts to get the right kind of results as much as possible, I think the, the access there, that last mile problem, as you said, Adam, I think he’s going to do so much to address that, which is kind of what Bhav was humble bragging about last season, about the self serve nature, right of kind of like getting to that, getting to that point.
[00:22:03] Daniel: But all that said, Adam, I find it’s a fascinating topic, but I feel like I’m, I need an answer. We’re 20 minutes into this and you said about where you got to with the Warehouse Native analytics whether it’s fad or future and where you are now, so let’s go back. Where were you thinking it was at in July and what has changed since July? What are you thinking about it at the moment?
[00:23:28] Adam: So I think it’s going to be a little bit more of a progression where you could have certain data sets that are in the warehouse and then certain that are in SaaS and then maybe five years from now, we look back and say, I can’t believe we used to send data to a SaaS product. So time will tell, but I think that I was hoping that it was going to be faster, that it was going to just be like one day everyone was going to say, I’m chucking my SaaS provider but I also have questions about whether, if you send your data right to the warehouse, I don’t think a lot of the GDPR things go away, but I do think that you don’t have a vendor that has to be, you know, kind of proven to be GDPR compliant because obviously if you’re not sending it to that vendor.
[00:24:17] Adam: Like if you were sending data to Amplitude, you got to make sure that Amplitude ticks all the boxes, but if you’re basically just collecting the data and sending it to snowflake and you’re querying it in Amplitude, I think there’s an open question of what are the GDPR ramifications of that? And I think that has yet to be tested.
[00:24:31] Adam: So, I wish I had a better answer, but I think that long term, I think that Warehouse Native is either going to fizzle out. People would be like, that was stupid. Or I think five years from now, everyone is going to be using a warehouse. And if you go to like a GA, you’re basically using BigQuery with just a reporting front end and so on. So it’ll be interesting to see. I wish I knew I’d, you know, I have a lot more money if I could predict the future perfectly, but, but that’s the best I can tell you today.
[00:25:00] Bhav: It’s interesting because I grapple with this I think over the last few years, you know, we’ve seen things come into fruition and then slowly fizzle out, so, you know, when people talk about big, and I know these were more concepts than actual physical artefacts like a, it’s an actual artefact, whereas the concept of big data was, you know, it was very abstract and that kind of fizzled out. And then we saw data science become a thing again, but data science is very abstract. And that kind of fizzled out into more like arms of data science around machine learning and, and also, you know, AI and econometrics and things like that.
[00:25:31] Bhav: And like you, if Warehouse Native is going to suffer the same fate. And I wonder what the cause of that is like for me, the biggest problem that we face for something like Warehouse Native data to not become a thing is a lack of resources and lack of engineers and lack of people who can make this future reality. I think that’s what happened with big data. I think that’s what’s what largely happened with, well, maybe data science was too broad a concept that needed more narrowed functions, but there was a, there was a skills element and there was a skills gap that played a big part in those. And I wonder if skills and the lack of available resources to do that is, is going to be a big hindrance and I’d love to hear what, you know, what you think is going to be the cause if it does fizzle out, why do you think it might fizzle out?
[00:26:18] Adam: Yeah, and what I’ll say will sound a little crazy, but I actually think the fate of Warehouse Native has nothing to do with technology. I think what it has to do with is the collaboration between marketing teams, product teams, and data teams, because I think that these groups have to decide if they’re going to all come together and share one data model and that’s really difficult to do. And I think a lot of times why people went with a SaaS provider is they just wanted to do their own thing. And I think there’s such, I see a company’s a lack of collaboration between data science teams, data teams, product teams, marketing teams, and especially at large companies, these are different departments.
[00:27:04] Adam: And if you could get everyone to agree that this is the, these are the data points we need to collect. These are the dashboards, this is the full customer journey that we want to track. I think it’s more of a people in process thing than a technology because once you get on board with all those teams collaborating, then if the marketing team says, hey, it’s really important for us to track when someone starts a lead form but the data team doesn’t think that’s really important because they only care when someone completes a lead form, but they don’t understand that the marketer wants to understand which Google keywords are at least getting them to start the form. And that’s a different problem than, than, you know, completing the form once they hug it out and they agree that like, we’re all on the same team and we should get this to work, then I don’t think it takes you six months to get data into the warehouse.
[00:27:52] Adam: And so I think, and again, that’s what I think is going to be the biggest barrier is, you know, just people afraid that their warehouse is going to just explode. And, you know, if they get every data point marketers want or product teams want. So I think it’s, I think that’s what’s going to be the biggest barrier and I think that in a weird way, Warehouse Native is going to force these teams to either work together or push them back into their own corners. And that’s what I’m not sure is going to, what’s going to happen when that really, when that really comes to pass.
[00:28:20] Daniel: Are we kind of talking the same kind of thing as like a CDP or customer data platform or a composable CDP? I feel like there’s so many acronyms and so many kind of like skews and approaches to this, that on the same thing when we’re talking about Warehouse Native analytics, we’re kind of talking about unifying data sets from different places and activating it right and doing something with it maybe just not just beyond kind of analysis or reporting, but then kind of connecting that back in through whether you’re doing it off the shelf, like through reverse ETL tools, or whether you’re kind of building these pipelines in the clouds to push it back into these tools for whatever the purpose. I don’t really have a question there, but is this the same thing we’re talking about when it comes to CDPs?
[00:28:56] Adam: Yeah and I really think that the reason why Warehouse Native started was because of composable CDPs. And I kind of joke that I call Warehouse Native composable analytics. Because I really think it’s just, I mean, you’ve got composable happening everywhere. Message Gears is composable, you know, Brazen, iterable. You know, so I think that, and I think philosophically, there’s this packaged versus native slash composable mindset. And I think that companies are going to make a decision whether they want the warehouse to be the centre of the universe and everything else is composable off of that. Or do they want to go packaged route?
[00:29:34] Adam: And for example, if you’re in the package world, you know, Adobe is the ultimate going the other direction. Like Adobe is the anti composable vendor, where basically they’re saying we want everything in Adobe. We want your CDP, your analytics, your Adobe campaign, your testing, everything. And it’s kind of like buying an Apple product, if you buy all Apple products, they all work together like that’s great. But if you want to build your own, then you’re, you might go to the other extreme, which is, I want to do everything composable. I don’t want segment, I don’t want any package vendor. And I basically want to be able to plug and play any vendor I want.
[00:30:09] Adam: And I don’t know if we’ll end up on the right, the left or the middle, but I think that the industry seems to be veering a little bit more towards the composable, because it can be more cost effective. And ultimately, it depends on whether the data team is the one that runs the universe, or is it more of the marketing team that runs the universe and for the last 20 years, marketing has really run the show. You know, they’re the ones who decide what analytics vendors to use, but five years from now, it could be the data team that says, you know, whatever vendor you used to your reporting has to work on this warehouse and I don’t really care what vendor you use. You know, you could use Google sheets or equals for all I care, as long as the data is in the warehouse so that may be the world we go to. But again, it’ll be interesting to see, that’s why this industry is fun to be in, because just when you think you know everything, it changes.
[00:31:01] Adam: But what’s funny is, I actually feel like for 20 years, our industry has not changed that much, and we’re finally at the point where everything is changing, you know, GA4, Adobe, you know, has a new product, you know, Amplitude kind of came out of nowhere. You’ve got Composable, Warehouse Native. I feel like we’re finally in this place where like, it’s kind of fun to have things be shaken up a little bit. It’s a little bit scary, but it is kind of fun at the same time.
[00:31:26] Daniel: I love where like the, the whole thing, Adam, and I think first of all, if, if you’re not happy or comfortable with change and ambiguity, then it’s a, it’s a hard industry to be in for a long time right? And so I completely get that. The thing that I think infuriates me sometimes is this idea that you can, you can have the best intentions and go down the route of being like, I’m going to go composable. I’m going to have this modular ecosystem that’s beautiful, I’m going to set it up, it’s got documentation governance.
[00:31:53] Daniel: And the whole idea is that it’s kind of a modular enough so that we can swap in and swap out. But then you go down the route of being like, okay, cool. Well, I’m going to be using Google Cloud for the storage and then I’m going to use data form and then, oh look, Google just purchased data form and it’s rolled into its ecosystem and this kind of vendor consolidation aspect whereby you can start with that. But eventually, I mean, look at dbt and other, other tools like that within that same ecosystem. I mean, eventually they’re getting sucked up into individual, these kind of bayer moths. And it’s happening across different industries at the moment. Maybe there’s something in the water of the gaming industry specifically that I’m thinking of where Microsoft keeps buying up everyone and so does Sony. But how does that, how does a composable world work when these things keep getting purchased up? Or is it kind of like, you’ve just got to keep, keep on top of it and keep vendor swapping using that modular nature of the system?
[00:32:39] Adam: Yeah, and the joke of it is that I see a lot of organisations that say we want to be able to swap out vendors, but then they never do swap out vendors. So that’s kind of an interesting dynamic, but I think it’s a chicken and the egg thing because I think the reason that these acquisitions are happening is because people are integrating these tools together. And if you start to see that, you know, 30 percent of your customers are integrating with this one vendor that you can afford to buy, like, why wouldn’t you buy that yeah and just get the economies of scale.
[00:33:08] Adam: So it’ll, it’ll be interesting to see. I think there’s always these like plateaus where there’s a lot of change and then it plateaus for a while, and then there’s a lot of change. So I think my prediction is in about six months, you’ll start to see some of this kind of settle down a little bit. But then, you know, when the economy kind of gets back on track, you might see even more consolidation. So yeah, you just have to kind of roll with it and just determine what is the right, you know, function that you need. And don’t think of it as much as products. Think of it as like, I need this job to be done and what is the vendor that I want to do this job? And there may be some cases where having one vendor to do multiple jobs, isn’t the worst thing in the world. And like, one of the things that we focused on at Amplitude is we’ve picked what we call the big five. Where we have marketing analytics, product analytics, CDP, experimentation, and we just added session replay because we believe that those five all naturally go together in terms of building better digital products and experiences.
[00:34:06] Adam: So maybe there’s more in the future, but those are the ones that we focused on. But we also make it swappable. So if you want to use a different session replay vendor, you can, but you have the option of using one platform for everything. And I think that’s what you’re going to see it’s just lots of the ability of some vendors that can do a lot of things and some vendors that you plug and play. So it’s really up to you which vendors you want to work with.
[00:34:28] Bhav: I think for me remaining vendor agnostic has always been something I’ve prided myself in. I think it’s far too often and this speaks maybe to who I am as a person and, you know, who a lot of people are. And I really think the world should become more like this, I think the problem with you know, being a complete one vendor marketing specialist or analyst, whatever it’s going to be, it means that you kind of alienate yourself from everything else that’s out there and you never become fully adaptable. And I like the idea of data professionals and analytical professionals and, you know, marketing and product professionals becoming agnostic.
[00:34:59] Bhav: And I really think that in order for people and companies to develop their skill sets and develop the people within their companies they need to adopt this mindset of we need to be flexible. The amount of times I’ve joined a company or someone’s joined a company and they bought in their full suite of tools because they refuse to you know to integrate with what’s already in the organisation. I always found for me, that’s my kind of like bugbear is people’s inability to be flexible.
[00:35:23] Bhav: Adam, I have a question that I wanted to ask you and I’ve been meaning to ask you for about 10 minutes now and I want to, I want to get in there before we run out of time. Is, I have a tonne of respect for you, the fact that, you know, you try not to focus too much on Amplitude, the company you work for, what, you know, you could easily take this conversation down a route of Amplitude this and Amplitude that, but I do want to ask a question about Amplitude. I’ve worked with the product extensively over the last two or three years. I’m a big fan of it, but do you see as a competitor Warehouse Native being the primary risk to a company like Amplitude or do you see it more as a, you know, there are going to be up and coming players who are going to come like try to eat into Amplitude’s lunch and it’s going to be a question about cost effectiveness for certainly for new you know, startup companies.
[00:36:06] Adam: Yeah, I think that Warehouse Native is we look at it as complimentary. We look at it as, as I mentioned before, there’s some people that are that invested a lot in a warehouse, but they really like Amplitude and they should be able to use Amplitude with the warehouse state, they already invested. Other companies don’t have a warehouse and they kind of use Amplitude as their warehouse.
[00:36:29] Adam: But I think one of the things that’s really, I like about Amplitude is they’re pretty nimble and they see what’s going on in the industry and see different vendors popping up with different ideas like Warehouse Native. And what, what they do is they say, listen, this doesn’t have to be this or this, it could be, and, and we could do this and this and let the customer decide. And that’s one of the core tenets of Amplitude is just being kind of vendor agnostic. It drives me crazy because we’ve just invested a huge amount of money where you could take GA4 data and send it into Amplitude. Or you could take Amplitude data, send it to BigQuery. Like we literally integrate with our biggest competitors.
[00:37:08] Adam: And I’m like, why are we like, guys, why are we doing this? And they’re like, listen, you know, we are vendor agnostic and, and that’s part of being a CDP. So I think that it’s not, it is a threat, I think in general, but I think everything is a threat. I think GA4 is a threat, Adobe is a threat, Mixpanel I mean, there’s always, and I think threats are good it keeps you nimble and it keeps you innovating. And one of the biggest complaints that I’ve had for the last 20 years in our industry is that there hasn’t been a lot of innovation and ironically, one of the reasons I left the Adobe world after 20 years, not because I didn’t like Adobe, adobe is awesome. I wrote the book on the Adobe analytics product, but I just was a little bit bored of doing the same thing and I wanted to learn more.
[00:37:49] Adam: I was a marketing analytics person and I had heard about product analytics and I heard about Amplitude. So I just wanted to learn more about product analytics, and I thought those two could be brought together. And I think that you have to always be learning new things, and I think that that’s just, I don’t know, it’s just something that I believe in strongly is just learning and figuring out the best of each technology. And as an Adobe person for many years, I found myself doing what I would call product analytics in a marketing analytics tool, and since I wrote the book on it, I was so good at it that people would have come to me as the last resort and basically say, we can’t get this to work in Adobe, and I would find a way to make it work. But I basically was doing things like I was bending it so much that sometimes it became brittle.
[00:38:35] Adam: And now, of course, Adobe is getting into product analytics, I think, for that reason. But I think that, you know, at some point, you just got to be intellectually curious. And as an industry, I saw this convergence coming, and that’s just why I wanted to learn. And I wanted to play with new tools, and so on. So, I probably skewed off of your original question but.
[00:38:55] Dara: What about on the other side of the fence Adam? If you’re a business and you’re a little bit behind the curve, because a lot of this is focusing on us and we work in this industry day in, day out, but if you’re a business who maybe has traditionally been slightly behind the curve and is maybe still relying on using a tool like Google Analytics and they go into the interface every day, but they’re aware that maybe there are other things to consider and there’s other ways of getting their different data sources together. What would be your advice to a company like that? How would they start to, you know, use their curiosity and maybe start to move themselves along that, that maturity path?
[00:39:31] Adam: Yeah, I mean, I think the thing that I always recommend that people do is take a look at their tech stack that they have today, and they may not realise it, like your marketing team may not realise that your product team is using another vendor. You know, they might using an Amplitude or base panel or even like Splunk and then your UX team might be using like a full story or hot jar. And I think if you could just get a picture of all the places where you’re storing data about customers, and then make a list of what are the questions that we really want to answer. And this has been my lifelong complaint with the digital analytics industry is that we always focus on collecting data and the tagging and the technology first. And I’ve written a lot about just, you need to think, what are the business questions you’re trying to answer? And the technology is really just an enabler.
[00:40:21] Adam: And if the questions that you have are just around marketing, then Google Analytics, you know, maybe enough. But if you want to understand what they did after they came from a marketing campaign, once they’re using the digital products, then if they go to a store, that leads you to a different solution that may need a bigger, you know, a bigger stack or a warehouse or something like that.
[00:40:40] Adam: So I think starting with the technology is the wrong approach. And that’s why, like, I always get annoyed when there’s people who are like, I’m an Adobe person, I’m a Google person. I’m like, that’s like saying I’m a hammer or I’m a screwdriver. Like those are just the tools, like you, what you really want to focus on is what you’re building and those are just tools to get you there. And I think my biggest gripe with GA4 just to rant on that for a second is not a problem with GA4, even though there’s a lot I could go there with, but it’s more that I’ve heard from years for years from hundreds of people being like, I am tool agnostic. I don’t care what tool I use and then GA drops this bomb on them and everyone agrees that GA4 is not awesome. But they do whatever they can to bend themselves into knots to make GA4 work instead of looking at other tools. I kind of joke and I’m like, hey, if you said that the tool doesn’t matter and you hate your current tool, why 95 percent of them are sticking with it?
[00:41:38] Adam: And so I think that a lot of people are full of crap. I don’t know what I can, what I can say here, but I think that a lot of people say that they’re tool agnostic, but really their career is dependent on them knowing a particular tool and I don’t think that’s a good place to be. I think that you should be, you should know many tools and that you should be vendor agnostic, but I think people say they are more than they really are.
[00:41:59] Bhav: Testify, I agree. I Really, really feel like analysts would be in a much better position, not just, I mean, forgetting from a technical perspective, you know, the ability to adapt to your environment is a life skill, you know, not just the technology stack, but, you know, when you change company and you go from, say you worked in betting gaming and you move into e commerce, that ability to sort of like flex those skills that allow you to adapt are, they’re priceless and I just, I can’t, it really bothers me when people are really rigid. Guys, I mean the, I, I really hate to do this, but I actually have a call I need to jump on and listeners, I’m really sorry, but this is, I guess, the nature of working remotely and not being in a nice environment, but I will leave you all to carry on.
[00:42:43] Daniel: There was something that came to mind as you were going through that last piece around the kind of, the specialisms, and especially around the product and marketing analytics set. So you mentioned like Google Analytics 4 kind of like boldly quote unquote, boldly entering the product analytics space by going into the event schema, which is the thing I think people are underlying the most upset with, even though it’s just coming in line with every other tool out there. And you mentioned Amplitude kind of covering the marketing and product analytics or part of that big five set. And we’ve seen a lot of that, we had an episode in the last series where we talked about the future of product analytics tools and understanding how they’re all kind of diversifying into the world of marketing analytics as well.
[00:43:19] Daniel: And so I’d love to get your thoughts around can product tool truly be a marketing analytics tool? Can a marketing analytics tool truly kind of like disrupt that space or are they always going to specialise themselves and kind of do enough of the other thing to kind of satisfy people that want to kind of dip their toe?
[00:43:36] Adam: Yeah, it’s really tricky because it gets into this what is the big difference between marketing and product analytics? And I’ve written a little bit about this. In general, I think that once you get to events and properties, any tool can do anything, but there are different specialisations that tools make that make the difference.
[00:43:55] Adam: So, for example, if I use GA4 as an example, there’s not a lot of reporting in GA4 around, say, user retention. There’s one or two reports in Amplitude, not just all Amplitude, but you know, there’s like 20 plus reports around user retention. That’s an area where Amplitude was driven deeper by our product user base. Now, there’s probably 20 reports in GA around marketing and we probably have five or six. So I think ultimately they’re all going to end up in the same place. I do know that a lot of product teams don’t think of GA as a product analytics tool. And that’s partly because Google did such a good job of branding itself as a marketing analytics tool.
[00:44:40] Adam: It’s actually like you know, the brand tissue and Kleenex where people say Kleenex, but they really mean tissue. Like GA is synonymous with marketing or web analytics. So I think their own branding is going to get in their way. And I don’t know if Google wants to be in the product analytics space. And I wrote a blog post that said, basically, just because you’ve have event based models does not mean you’re a product analytics tool. You could do some things, but not everything. And all indications that I’ve seen from Google is that most of the features they’ve come out with GA4 are very heavily skewed towards either getting data into BigQuery or being like a front end to Google Ads. So I don’t know that they want to even be in that space.
[00:45:25] Adam: Now you never know because Google goes, they zig and zag all the time. But like in Adobe, you know, they’re clearly marketing analytics, but they just launched a separate product analytics. And that’s you know, two years ago when I joined Amplitude, one of the reasons I joined was because I do believe that that marketing and product have to work together, and that you just want to understand the funnel, and the marketing is really just the beginning of the product funnel, and everything is now a product.
[00:45:50] Adam: I think of websites as products, and so I think that marketing and product teams are going to merge. I think it’s just going to be like the customer experience team. And I think that you’re going to have to pick a vendor, you know, that does both eventually. But I think if I were a betting person right now, and I know this is, I don’t want to like offend you because I know you guys are marketing analytics people, but I think that product analytics is becoming more important than marketing analytics.
[00:46:19] Adam: And the reason I think that is because there’s just so much scatter around marketing between SEO and social and TikTok. I just think marketing is becoming really difficult because there’s so many channels and with cookie deletion and privacy, it’s really difficult to prove that the advertising that you’re doing is actually having an impact because you almost have to take a little bit of a leap of faith and hopefully, you know, mixed modelling will help but I think that attribution has become really difficult in today’s day and age. And if I was a CEO and I’m saying I have a million pounds to spend, if my marketing group is saying, listen, I want your money, but I can’t a hundred percent tell you the impact of it. And then the product team is saying, listen if you give me the money, I’m going to make, I can prove to you that the people who are loyal customers are coming back over and over again, and we need to have a better experience in our product. I think that I would probably give 30 to 40 percent of the money to marketing and 60 to 70 percent to product.
[00:47:25] Adam: And as I think marketing becomes more and more difficult because of more privacy, more cookie deletion, maybe two years from now, that’s 80, 20. And so I think that more companies are leading with, we need the best product analytics tool, but we want it to be able to do marketing analytics, or they’re going to continue to have a GA and then a product analytics tool and find a way for those two to work together. But I think that’s what’s really driving it. And what’s the last thing I’ll say is a lot of people in my LinkedIn network, I watch as they change jobs and all these people that used to be digital marketers are now changing their titles to like, digital product team. And so that is another, like, kind of thing that I notice happening out there. But I’m curious for your thoughts, because you guys are more kind of traditionally in the marketing space.
[00:48:10] Daniel: It’s really interesting because I completely agree which is a really boring way to wrap up a podcast episode, just agreeing with you. So what I’ll say is ironically, I find that the marketing analytics tools are being marketed so that people using those tools and I would say, quote unquote, traditional marketers are not understanding that the things like cookie deprecations, different browsers, handling things differently, how nuanced and hard app analytics is or app attribution is especially on the iOS devices. I think the, the tools are doing a good job of masking the complexity and saying, no, no, no, don’t worry, everything’s the same, we can still do attribution.
[00:48:44] Daniel: I mean, look at GA4, as we mentioned before, it’s just modelling data. It’s guessing the bits that it can’t see and it’s just pretending like nothing’s wrong. It doesn’t explicitly tell you necessarily that it’s guessing numbers and they’re not exact numbers, but it’s like, don’t worry, don’t worry we’ve still got numbers, we’re still doing the same thing. And I think this is the interesting thing is that I think as people become more mature or wise or woke to the idea that data is diminishing or trackable data, observable data is diminishing and, you know, understanding how to adjust budgets between teams like that. I think I completely agree, I’m just wondering that maybe you have the less savvy business owners or kind of budget holders that are like, oh, but my marketing agency or my marketing team are saying that they can still do attribution, they’ve still got their reports, they’ve still got their numbers. And I’m wondering how long that’s going to take for the, the average person, the average marketer to realise that there is a shift like that happening and they are using estimated data without really knowing it.
[00:49:38] Adam: Or maybe AI comes in and figures out a solution. Like I’ve read some stuff about synthetic users where AI can basically like almost pretend like it’s a user. And you know, maybe there’s that, that maybe there’s a breakthrough there that changes attribution, but I’ve always been a little suspect of attribution and I’ve always felt like the real way to see if marketing works is when you turn it off and basically say, because I have this, I have this time at Salesforce where we literally turned off all of our paid search for a week and we didn’t notice really any changes and we’re like, well, why are we spending all this money on paid search, but I do think that there is a long term impact, but I think it’s going to be interesting to see what happens.
[00:50:16] Adam: But I just know that if I were like, if I were coming out of college now, and I had a choice of being in the marketing analytics career or the product analytics career, if I wanted to be an analyst as a career, I would nine times out of 10 go into the product analytics, just because I think that’s where the industry is going. And that’s why I think you see Adobe trying to get into product analytics. I think that’s why you see, I actually think Firebase, you know, I don’t, we see a lot of cases where people use Google Analytics and Firebase. I mean, I don’t know what Google is going to do with that, but maybe Firebase gets sunset and basically anything that, you know, Firebase stuff gets wrapped into GA4. I’m not, again, you guys know more about the Google world than I do. But I think that might be their foray into product analytics.
[00:50:58] Adam: But I just don’t think that Google speaks to the marketing, to the product team, like they do to the marketing team and companies like Amplitude and Mixpanel, like we just know the product team and we know what they want. And so I think it’ll be a really tough road and I don’t know, how does Google make any money being in product analytics? The only reason they do analytics in the first place is to make more money in advertising. How do they make money? But if one day Google wakes up and says, people aren’t seeing the impact of their keywords, because the impact is a year down the road in a digital product, then I’m sure they will figure out a way to get into that space. Because that’s all they care about is their advertising business.
[00:51:34] Daniel: Well, that’s it. You hit the nail on the head there, Adam, which is like, can they monetize a product people, they can monetize marketers, right. Because they connect into the Google Ads and still, whether that changes one of the biggest investments for a marketing budget perspective for most businesses. The other thing I can think of is maybe the product quote unquote product analytics solution there comes in the Google Cloud. So through the BigQuery stuff, and maybe they have Google Cloud products that you can layer onto the data and use them as like data models, manipulations, funnels. Things like that so who knows? I don’t know if it’s a GA thing, but it might be like all their products now powered by GA because quote, unquote, they’ve solved a lot of these annoying issues like privacy and compliance and laws in the world of Google Analytics 4 right. And so, using that to power their technology, I think is where they’re going.
[00:52:18] Daniel: Like you said, there’s no need for a GA4 UI. It will just power BigQuery data sets, power your advertising in Google Ads and your advertising ecosystem. I know that they’ve now partnered with VWO, AB Tasty, and Optimizely to do their kind of measurement of those solutions. So it’s kind of plug and play for those and I think it’s becoming more of a software and more of a tech backend than it is a front end because you’re just consuming the data in these other products. So, you know, why log into GA4 anymore, especially with the rise of generative AI, when you can just plug that in. So anyway, interesting.
[00:52:49] Adam: I’m just glad is that finally there’s some competition happening in the industry and I think what’s kind of ironic is we’re a small 700 person company that’s really pushing, you know, Google’s the Adobe’s of the world to innovate. And I think that that’s always good for the industry and that was one of the reasons I came to Amplitude is just, I wanted to see our industry change and I wanted to see marketing and product and CDP, all this stuff kind of consolidate and I’m glad that it’s starting to happen, but it is a little bit scary at times.
[00:53:21] Dara: Okay Adam, just before we let you go, we’re going to hit you with our trademark rapid fire questions. So the first one is what’s the biggest challenge today that you think will be gone in five years time?
[00:53:33] Adam: We talked about this a little bit. I think it’s going to be reporting. I think generative AI is going to just solve that and it’s going to make analytics more accessible to many more people at the organisation. So I think it’ll cost some people jobs, but I think it’ll be better for the industry in the long run.
[00:53:52] Dara: Okay so what will be the biggest problem in five years once reporting is taking care of what’s going to take its place as the biggest issue to solve?
[00:54:01] Adam: I think the biggest issue is going to be, is going to burst the bubble of people who say they want to be data driven, but don’t actually want to be data driven. And I think right now there’s so many, there’s so many easy ways to say, well, I don’t know if I believe that data is correct. And I’m not sure if that’s really collecting the full customer experience. And so when they are faced with a report that actually does everything that they can’t poke holes in, I think it’s going to smash the facade that people say they want to be data driven, but they really just want to do what they want to do and they want to trust their gut.
[00:54:37] Adam: So I think we’re going to move into a change management dilemma and I think a lot of strategy, a lot of consultants who today do technology consulting are going to move into strategy consulting and change management consulting to convince people to actually use data.
[00:54:52] Dara: I like that, good answer. What’s one myth that you’d like to bust?
[00:54:59] Adam: I think the biggest myth that we talked about a little bit is that there’s only two vendors in the analytics space. And I think that there’s not saying you have to go with ours, but I think that you right now I think it’s like 95 percent of the world uses either Adobe or Google. And I just think that’s a myth that those are your only two options. And I think that it hurts the industry by having everyone use this duopoly because we all suffer in terms of innovation and I think that’s the one that bugs me the most.
[00:55:33] Dara: Okay, if you could wave a magic wand and make everybody know one thing, let’s probably keep it to analytics, but what would that, what would that one thing be?
[00:55:45] Adam: I would probably say that I would want everyone in our industry to magically have the skill of being able to communicate and present data in a way that in that stakeholders and executives can understand through better storytelling and better presentation skills just as a side note, when I was young, my mom forced me as a teenager to be in an acting group and I hated it, and I was embarrassed up on stage all the time singing and doing stuff. But now I feel really comfortable getting up in front of an audience and being able to talk about any topic or take data and be able to say, here’s why you should care about this.
[00:56:27] Adam: And I think there’s too many people in our industry that do like 95 percent of the work, but they, they fumbled the ball with five yards left using an American football analogy, because they just don’t know how to communicate it correctly, and if I could wave a magic wand, like I wish everyone in our industry could have that skill and my kids were in college, I’m basically begging them to say, take a speech communications class where you have to get up and learn how to give a presentation, no matter what, even though you’re a software engineer major, learn how to do that because that is the most important thing at a job because that’s like what people remember and that’s what people see, and you could be the best analyst in the world, but if you can’t communicate your ideas, then it all goes, you know, it’s all for nothing.
[00:57:17] Dara: Couldn’t agree more. That’s a really great answer and great advice as well. Last one’s an easy one or to some people, it’s the hardest one. What’s your favourite way to wind down?
[00:57:27] Adam: So mine is easy. I have a little side hobby. I’m into classic cars. So many years ago, I started getting into the first generation Corvettes from the 1950s and 60’s. And so I recently restored a 1962 convertible Corvette that now has all the modern, you know, stuff has Apple CarPlay, air conditioning, all the good stuff. And so when the weather is nice, and I just need to unwind, I just crank the radio and go drive on, you know, little side roads in my little convertible. So that’s my kind of like had a bad day, need to go do something, but unfortunately half the year living, I’m now back in Chicago, so that limits me to half a year, but one day I’d like to live someplace warm all the time, so I could do that anytime I have a bad day.
[00:58:16] Dara: Our colleague George is going to be really jealous listening to this. He’s also into restoring well, I think he’s done it with one car. I hope he won’t mind me saying he’s only done it once, but he’s restored a, I’m going to say an MG. Hopefully I’m not, yeah MG, MG Midget. So yeah, I’m going to say, I’m going to boldly say that a Corvette is, is a step up from that at least. So he’s going to be jealous. Adam, that’s it we thank you very much for your time, really enjoyed this chat. You’re off the hook now.
[00:58:44] Adam: Awesome, okay. Well, thank you guys so much.