Data Product Analytics & the Failings of "Data as a Product"

Last time on the Data Knowledge Newsletter, we dove into how an analytics hub can bring all of your team’s critical data assets into one place, and help your organization begin to tame the entropy of the modern analytics environment. Today, we zoom in closer on the related challenges teams face in understanding how their assets are being utilized.

Most often, analytics teams have little information about what analytics assets are used, and drive decisions within the organization. In contrast to traditional software products, where there are many ways to collect data about adoption and value, data teams have little insight into usage and adoption. 

In the context of the push to treat “data as a product,” how do data teams find themselves facing such a glaring blindspot?

‍

How have teams approached asset analytics in the past?

With gaps in the solutions historically available to understand analytics adoption, teams have taken various approaches to understanding how analytics are being used. These are inspired by the methodologies applied by product and customer experience teams, and largely hand rolled: 

Using NPS surveys to gauge satisfaction: 

• Net Promoter Score (NPS) is a tried-and-true method for determining if a given product or service is living up to user expectations. In the context of analytics, NPS can give you useful insights into which employees are well-served by your current suite of reports, dashboards and operational data.
• It‘s a good idea to send out NPS surveys periodically, especially if your analytics team is serving a broad range of functions in the organization. 
• The downside: surveys can be time-consuming to conduct and analyze at scale, especially if you include additional subjective questions beyond pure NPS metrics. They also have the inherent limitation of being subjective, as you are asking users about how they feel, rather than collecting objective usage metrics. 

Conduct user interviews

• User interviews are a crucial tool in the product manager playbook, and should be deployed to better understand whether your solutions address the wants and needs of your users.
• However, as with NPS surveys, user interviews are anecdotal, subjective, and probably the most time-consuming to conduct. 
• User interviews also run the risk of selection bias, meaning some perspectives – such as the lowest adopting team members – may be left out. 

Pulling native asset analytics: 

• The best source of quantitative data about asset usage comes from the very tools where assets are built and consumed. In an ideal world, your analytics platform of choice would give you the full picture about how your data is being used, guiding future development and letting you know which assets to invest in. But this ideal is rarely the case, for several reasons: 
• Tools vary widely when it comes to what data they actually track and expose. Many BI solutions require custom work to pull the data from an API, or in the worst case (but sadly not an uncommon one) the platform may not track this information at all. This great article from Sarah Krasnik provides a helpful summary of how to do this for some of the most popular BI platforms. 
• Finally, in a heterogenous analytics environment where multiple tools are deployed to consume data, what is available will vary wildly and home grown solutions become painful to support. This prevents most teams from unlocking the insights available to a product manager via a product analytics solution like Mixpanel or Amplitude. 

What does “data product analytics” mean?

“All of the effort that my team is doing, is it actually generating the output that the teams expect or that the team is actually able to use? Basically, limiting the amount of wasted work that our team is doing.” 

— Michelle Ballen-Griffin, Head of Data Analytics, Future

When measuring the success of your data products, there are a few good questions to start with: 

• What are the active users of any given data product (i.e. a dashboard)?
• Who are the people within the organization using any given data product?
• In what contexts are they using that data?

Answering the question of active users is both the simplest and the most important. If you do not know how many people are using your reports and dashboards, you have no guideposts for which assets are crucial to support, and which are not worth your time.

Discovering who is using your data assets provides additional insight beyond active use. A deeper level of understanding might include which reports your Chief Revenue Officer looks at every day, or which dashboards are most popular within your marketing organization.

Finally, contextual and relational data about how data is being consumed, and the decisions that it informs, directly connects your data to operations and business results. This might uncover a group of assets that are frequently used together, or help you realize that teammates have to use two or three reports to answer a single question. 

All of this granular, quantitative, and contextual data mirrors what traditional product teams would consider table stakes for doing their job effectively – why should analytics teams expect anything less?

The unfortunate truth is that – until now – there has been no solution that collected and exposed this data in a single place. There has been no equivalent of a CDP that allowed for either client or server side tracking across all of the data assets in your environment. 

But luckily, thanks to the new capabilities of data products analytics, this information is now available to any data team.

“I would say that the data team’s mandate is not just ‘produce the data,’ ‘produce a data warehouse,’ or ‘produce reports.’ I kind of think the data team is accountable for how data is used across the entire organization, and creating a culture where the entire organization is making data-driven decisions.”

— Scott Breitenother, Founder & CEO, Brooklyn Data Co.

How analytics hubs enable data product analytics

In our previous post, we spoke about the value of an analytics hub in consolidating disparate assets in a fractured data ecosystem. 

Providing teams with a single access plane for analytics is a necessary prerequisite to data asset analytics. Only once all of your analytics are in an analytics hub can the metadata we have discussed be automatically collected and normalized. 

What happens from there is up to the analytics team. Armed with adequate usage data, they can do what they do best: develop insights, iterate and evolve their offerings, and begin driving the greatest value for the most people in the organization. Having extensive and normalized usage data is the foundation for effective Data Knowledge Management.

‍