Changelog

It’s common to track feature usage with attributes. For example, at Bucket, we’re tracking if customers use our weekly reporting feature. We do this by passing an attribute called "weekly_reporting" via Segment’s "group" call.

This is great for ad-hoc analysis. It’s straightforward to query all companies that have "weekly_reporting" set to "true" or "false".

Unfortunately, though, attribute values change over time and its history is lost, which means we can’t know whether customers that aren’t using the weekly reporting feature ("weekly_reporting=false") have never used it, or once were using it, but turned it off.

Additionally, we can’t know when companies started using the reporting feature, and for how long they kept it on.

These things matter a lot. Is there a feature awareness problem or a feature retention problem?

…🥁…

Great news - now you can know! Today, we’re releasing support for attribute-based features. We think this is a really powerful addition to Bucket.

If you’re already tracking attributes in your application, this will just work™ as is. Starting today, Bucket automatically keeps track of all company-level attributes and will automatically track any attribute value changes over time.

(If you’re not familiar with attributes, check out our post from last week: When to track features with events or attributes)

Getting started

Let’s say we work at Slack and that we want to track usage of the third-party Slack apps.

Whenever a user logs into Slack, we’re tracking which company they belong to and send the latest attribute values for the respective company. In code, this looks like this:

Followed by

Whenever a user logs into Slack, we check which Slack apps the user’s workspace is using and report that via the "apps" attribute. As you can see in the snippet above, “Acme Inc” is only using the Datadog app at the moment.

As a developer, simply grabbing the current state and passing it on as attributes is a really convenient workflow. Let Bucket handle the historic changes.

Feature reporting

With the attribute data flowing in, we can start tracking an attribute-based feature on Bucket.

Let’s say we want to track usage of the Stripe Slack app. To do so, we simply create a new feature and define active usage as: "apps" contains "stripe".

Here’s the feature report:

We now know that 6% of our customers use the Stripe app. We can also see that 2% have then stopped using the Stripe app. We can also see when customers installed and uninstalled the app, and who they are. Neat!

How it works

Let’s look at what’s going on under the hood. When the "group" call comes in, per company, we check if the "apps" values match our "apps" contains "stripe" filter. For "Acme Inc" there’s no match, so we place "Acme Inc" in the "Never used it"-state on the feature report.

Later, someone at "Acme Inc" installs the Stripe app. Bucket notices the difference in "apps" value, and runs the filter again. This time, there’s a match! "Acme Inc" is now actively using the Stripe app.

On the Bucket chart, we can even see which week the shift happened.

Later, "Acme Inc" decides to uninstall the Stripe app. Bucket again sees a difference in the "apps" value and now automatically places "Acme Inc" in the "Stopped using it"-state.

Notice that Bucket didn’t just put "Acme Inc" in "Never" when the "stripe" value disappeared. That’s because Bucket keeps track of the history and is aware that "Acme Inc" once used the Stripe app.

On the same chart, Bucket charts when the customer churned from this feature.

We love this chart and use it all the time. For most features, we eagerly check the amount of purple bars after shipping a feature. New users, yay! In the following weeks, though, we also keep track of the amount of red bars, which is just as "exciting" and crucially important.

More updates next week.

We’ve added typeahead search to the various dropdowns in the Bucket UI. It’s now much faster to find the event you’re looking for when tracking a new feature.

Similarly, it’s now much easier to pick the Slack channel in which you want the daily and/or weekly Bucket feature reports.

More updates next week!

This week, we’re adding to last week’s release, which focused on making the feature adoption criteria configurable. Today, we’ve added another configuration, which allows you to toggle the “Stopped”-metrics on or off in your report.

Feature awareness and retention

When you’ve just shipped a feature, you’re focused on feature awareness. Do users even know this feature exists, and how many have tried it?

Once users have found and tried the feature (hopefully!), you become more focused on feature retention. How many users keep using this feature after trying it out?

With our latest feature report configuration, you can now make a report based on your current focus. See below what it looks like in action.

Example: Evaluating a new chat message feature

You have a chat system where your users can send messages to each other. You have a hypothesis that when a user has sent more than 7 messages, they’re hooked and a lot more prone to keep using the product.

This feature is frequent in nature, but to begin with you don’t really care about the recurring use aspect - you just want as many users to 7+ chat messages as possible:

Here’s what that report looks like:

‍

87% of your users have passed the “tried it”-threshold, and are using the chat feature. That’s great!

Now you get curious about feature retention. Do the 87% keep using the chat system after their first 7 messages or not?

Flip a toggle to turn on the retention metrics and get the answer:

Here’s the updated report:

‍

Yikes! There’s red bars every week! 40% haven’t used the chat feature 2 weeks after becoming active users of the feature. This feature needs another iteration to make it stick…

More updates next week :)

One of the most powerful features on Bucket, is the pre-defined adoption states. Out of the box, Bucket buckets (see what we did there?) companies into the following states based on their historic feature usage pattern:

Until today, those state definitions were hard-coded. We believe great products have good defaults and rich configuration. So, from today, it’s now possible to configure the feature adoption KPIs  to suit the nature of the feature you’re tracking.

For example, for some features, companies should be marked as actively using it after just one interaction. For other features, 10 interactions is a better threshold.

On the other end of the spectrum, for some features, churn is when a company hasn’t interacted with the feature in 4 weeks. For other features, 1-2 weeks is more suitable

How it works

When you track a new feature on Bucket, you’ll instantly get the Bucket feature report. To then fine-tune the state definitions, simply click the pencil-icon next to the relevant state.

Here’s what it looks like in action:

More updates next week!

Our charts now show a partial data marker when looking at the current week’s incomplete data set. We indicate partial data by making a dotted line or dashed bar, like so:

We’ve also shipped a useful addition to our Slack integration. When you ship a new feature, it goes into the hands of your customers. But, when do customers start to use it, for real?

Our new Slack report gives you a daily digest of companies that just started using your feature. It’s especially useful in those early days and weeks after shipping a new feature where you want to track newly engaged customers closely to make sure they use the feature as intended.

‍

‍

You can toggle the daily digest on under feature settings, and toggle it off again once the feature looks healthy.

More updates next week!

We’ve just shipped a really useful addition to our feature report’s main chart. Until today, the Using it-chart showed the percentage of companies that are actively using a given feature, per week.

Now, the chart also includes the delta movements in and out of the Using it-state, in absolute numbers. In other words, the chart now shows how many companies, per week, that start using the feature and how many companies that stop using the feature.

Consider this chart. It shows percentage of companies that are actively using a feature:

‍

On the surface it looks like a slow but nice progression in usage, up from 25% to 50%. Nice, right? Not really! It turns out, this feature has a big churn issue, which is keeping it from growing much faster.

‍

Overlaying delta movements

Now consider this chart: It shows the same percentage, but now includes the weekly delta movements (right y-axis):

‍

‍

The light purple and light red bars indicate the amount of companies either starting or stopping use of this feature, per week. You ideally want many new users and only a few of them churning over time. In this updated chart, however, it's very clear that even though the percentage of active companies is trending upwards, there’s a large number of companies that churn every week. Yikes!

The good thing is that Bucket also shows you who the churned companies are, so you can investigate further and perhaps even reach out to some of them to learn why they churned - and how you can improve your feature in the next iteration.

To keep track of your features, get the Bucket feature report, including delta movements, on Slack every week, so you know, if your feature’s health is good or not — without breaking a sweat.

To learn more about how this works under the hood, check out our product handbook article.

Happy shipping!