How to actually help product teams ship better features

We’ve all read the theoretical articles about using “insights” to “evaluate” and build better products. Now, there’s nothing wrong with these but they’re missing the most important part: how to actually do it in practice and in a repeatable manner that doesn’t take up your whole week.

This piece isn’t about hypotheticals. It’s a piece about how to turn insights into better products in the real world.

At Bucket, we’ve interviewed more than 250 product teams and leaders. The one consistent thing that we heard was this: data isn’t their problem. In fact, they have hoards of it and plenty of expensive analytics subscriptions to make use of it.

Despite this, and after 15 years of “product analytics”, they still have a problem getting their product teams to become data-driven and, consequently, customer-centric.

It’s not that the data isn’t there, there’s plenty of it. The challenge is that most teams can collect and analyze data, or ask a team of data analysts to do it, but neither has time. When they do manage to find a moment, there’s rarely a organization-wide evaluation framework in place. This leads to inconsistent reporting metrics across teams, reducing data trust.

While everyone from junior product managers or CPTOs wants to ship features customers will use instead of just shipping features for the sake of it, they rarely have the actionable insights on hand to know which features their customers are really enjoying.  

This contradicts the product team trend of late: To make product teams empowered and to only hire Product Engineers. 

In case you haven’t followed this trend, here’s an ultra-short recap:

Empowering product teams, hiring product engineers over software engineers, or optimizing for iteration velocity are all different things on the same spectrum. It’s about making sure that customer feedback flows to the product creators - the product teams - as fast as possible so they can iterate and deliver valuable features back to customers (as fast as possible).

The higher the iteration velocity, the higher the chances of product-market fit, and the higher the chances of business success.

The way to achieve this is to ensure that everyone on the product team - engineers, designers, product managers - understands the customer pain points so they can chime in with solutions. Engineers and designers are very skilled at what they do. They can oftentimes think of better solutions than product managers, but only if they’re empowered to do so. 

It’s very understandable that CPOs and CTOs want to optimize ROI on R&D. A 25-person R&D team can easily cost north of $250,000 a month. With resources scarce in the current economic climate, focusing resources on the right projects is more important than ever.

In the context of product and R&D teams, more than half of all features fail to have any customer impact. That can easily be $125,000 wasted every month. And that’s not even mentioning opportunity costs!

It’s no wonder companies want to "empower" product teams to make data-driven and customer-centric decisions as fast as possible. However, it's very time-consuming to analyze and combine engagement data with qualitative feedback. It’s not feasible for these product managers and engineers to become data analysts as well.

These challenges have led some organizations to hire ProductOps roles to facilitate “empowerment”. While the intentions are good, it just makes product and engineering teams even more bloated when they don’t need to be. That’s because the solution doesn’t lie in more specialists, it lies in better tooling. It’s not a data problem, it’s a workflow problem.

How to make feature evaluation feasible

Most product teams have a glut of data. The challenge is turning that data into something useful. Automation is the key to unlocking this goldmine of actionable insights for product teams. 

Leveraging automation to improve efficiency is everywhere in the business world. Marketing teams use it to monitor their campaign performance and conversion rates; sales teams leverage it to send and track sales emails at scale. But when it comes to product teams and the features they ship, there isn’t that same type of automation.

It’s a bit crazy when you consider that R&D is where we spend the most money. About 50% of the costs in pre-IPO tech companies are from R&D (and for good reason, of course).

To bring automation to product management and feature evaluation, a few things are needed:

A tracking plan

Both product and engineering teams need to be on the same page regarding how to implement feature tracking in the product. 

A consistent baseline 

There needs to be a commitment to a consistent organization-wide evaluation baseline. Without it, nothing can be benchmarked and data can’t be trusted. 

This is precisely why we’ve built an open-source framework, STARS. It's integrated into Bucket to make it easy for product teams to have this consistent baseline right out of the box. 

Quantitative and qualitative feedback

Quantitative metrics are important, but they don’t tell you the full story. You need to combine them with qualitative feedback give them context and truly understand the “why” behind your customers’ behavior.

Automated reporting

Last but not least, it needs to be automated. It’s not a good use of anyone’s time to have someone constantly creating and sharing dashboards. Instead, you need automated reporting workflows that share the key takeaways with the relevant team members in the tools they already use.

How to actually implement it in the real world

Now, we get to the most important part, how to make this happen. You can break it down into roughly 10 steps or watch how it’s done:

Use STARS as your evaluation framework

You need to implement a consistent reporting baseline to compare performance across features and make better feature roadmap decisions. This is exactly what we’ve built into Bucket with STARS (Segment, Tried, Adopted, Retained, Satisfied). It automatically gives you a common baseline for measuring features, letting you benchmark performance.

Implement your tracking code (manually or with Segment)

Next, you need to connect your product to Bucket to associate interactions with a feature. There are three ways to do this:

• Add Bucket as a destination in Segment.
• Implement our Javascript SDK directly as an external script or import it
• Leverage our HTTP API that can be used in browsers and backends.  

Set your STARS configuration

Some features are designed to be used daily, others monthly. Configure what success looks like for your feature. Since basing adoption criteria off events alone can be misleading, you can take a frequency-based approach. This means a customer needs to consistently interact with a feature over a specific period. You can define how many days in a given period a company needs to interact to be considered as adopting the feature. 

Create a Release

Create a new Release in Bucket. Give it a name, choose the release date, set your evaluation period, and select the Slack channel where you want to receive goal progression and qualitative feedback notifications.

Set your goals

Next, you need to decide what success looks like. You use the STARS metrics as goals, so it’s easy to get started establishing clear objectives and performance expectations for key releases. For example, you can look to have 50% of customers in the Adopted stage.

If you’re unsure what goal to set, you can go step by step with one of the templates. Start with smaller objectives and iterate as time goes on. This can look like collecting 10 pieces of initial qualitative feedback.

Collect customer feedback

Get qualitative feedback from the right customers at the right time to enrich and contextualize your quantitative data, helping you understand if they’re really satisfied.

Customer feedback flows right into Slack so you’ll have instant visibility of any suggestions or complaints. 

Monitor your goal progression in Slack

You’ll receive automatic reports on your goal progression in a Slack channel of your choice. You can see where different customers are in the STARS funnel to quickly track adoption and satisfaction metrics while also monitoring the effectiveness of any product marketing campaigns. 

Segment insights by customer type

Now, not all customers are equal. Feedback from companies on your free plan isn’t as valuable as that from your mid-market or enterprise customers. Segment the STARS funnel by pricing plan, company size, and more by simply selecting a subsegment from the dropdown. 

This gives you a more granular view of how your most important customers are using a feature and what they’re saying about it. 

Evaluate feature performance by combining quantitative metrics and qualitative feedback

At the end of your evaluation period, analyze the release performance (which comes to you automatically in Slack). See which goals you achieved and where you fell short.

The qualitative feedback will help you better contextualize your metrics and understand what customers liked and didn’t like.

Decide if the release was successful or needs another iteration

The goal progression reporting and qualitative feedback give you the insights you need to know on whether the release was successful or needs another iteration. If it wasn’t a success, you’ll have everything you need to understand if you’re dealing with a product marketing issue, feature deficiency, or segment targeting problem. 

Bucket makes feature evaluation feasible 

Data is abundant, but product and engineering teams are still struggling to become truly data-driven and customer-centric. Manual data analysis, inconsistency in reporting metrics, and the resulting lack of data trust remain.

To make feature evaluation feasible, you need automated reporting workflows, a consistent baseline, and a combination of quantitative and qualitative feedback.

That’s exactly what we’ve built in Bucket. Releases provides you with powerful automation, our open-source framework, STARS, gives you a consistent baseline, and Live Satisfaction collects qualitative feedback to contextualize your metrics. All this is in one user-friendly tool.

It’s free to get started. Track your next feature release in Bucket.

‍