Changelog

The Bucket feature report has four feature adoption states. All of your companies are placed in one of these states based on their usage of the respective feature:

Companies enter the “Tried it”-state as soon as they’ve interacted with the feature at least once.

For some features, though, interacting with that feature might require some prior actions to be completed first. We refer to this as the feature onboarding funnel.

For example, let’s say your product has a Slack integration and you want to track how many companies share reports to Slack from your product. The event to track this might be called “Send to Slack”.

However, before being able to send reports to Slack, your companies must first set up the Slack integration. They need to authenticate with Slack and then choose a channel. In this case, the feature’s onboarding funnel looks like this:

For companies to reach “Tried it”-state, they have to get through the funnel at least once.

From Never to Tried it

For features with onboarding steps, you specifically want to answer this question:

Are the companies in Never not interested in this feature or are they getting stuck trying to use it?

The feature onboarding funnel is designed to help you answer this question.

Let’s combine the adoption states with the onboarding funnel to get a good overview:

Looking at this, it’s quite clear that a lot of companies are trying to use the “Send to Slack” feature, but the vast majority fails to progress from the “Choose channel” step. Of the 455 companies that enter the funnel, only 135 manage to get through it!  

(Fun fact: We discovered this very problem ourselves at Bucket as we had forgotten to support pagination when fetching the Slack channels. Oops!)

Available today

The onboarding funnel is available today and is now a default part of the feature report. Here’s what it looks like in action:

For features with onboarding steps, simply click “+” to add those steps to the funnel and learn what the drop-off rates are between steps. Each step has to be a distinct event.

The onboarding funnel doesn’t affect how the Tried it or Using it is calculated – it’s simply a secondary insight into what happens between Never and Tried it. To learn more about how the funnel works exactly, check out this help article.

Happy shipping!

‍

When you track an event-based feature on Bucket, you want to know how often your users interact with the feature. The Bucket frequency score tells you exactly that.

Score, not count

Many tools let you see how many times users have interacted with a certain event (feature). This provides you with some insight - none, few or many times - but what does that actually mean?

Are they still using the feature? If so, how often? When did they stop?

To illustrate the above, users of these two companies have interacted with the same feature 100 times, but they have very different underlying usage patterns:

‍
Company A used the feature consecutively for a period and then churned. Company B on the other hand has recently started using the feature a lot.

Daily, Weekly, Monthly or Quarterly

To  transform count into something more useful and actionable, Bucket calculates a frequency score out of the box. We look at recent usage data, per company, and transform the result into an understandable and informative frequency score. The score results can be one of the following:

• (Never)
• (Tried it)
• Daily usage
• Weekly usage
• Monthly usage
• Quarterly usage
• (Stopped)

Here’s what that looks like when looking at feature usage on aggregate for all companies (average):

Each company has a frequency score, so you can sort them by frequency score:

You can also dive into the raw data, per company, to better understand the frequency score or to keep track of key feature usage over time:

How is the frequency score calculated?

The frequency score is primarily based on how many feature interactions a company has performed in a recent time period. Each score option has a dedicated time period. For “Daily” it’s past 2 weeks, for “Weekly” it’s past 4 weeks and for “Monthly” it’s past 12 weeks.

When calculating the score, we look each time period sequentially. The first period where the company has interacted with the feature more than 60% of the time, is the score the company gets.

For example, if a company has used the feature more than 60% of the time in the past 2 weeks - meaning equal or greater than 8 days out of 14 - the company gets scored as “Daily”.

Using individual time periods per score option means that companies will quite quickly jump from e.g. “Monthly” to “Daily” - or vice versa - when usage spikes or tanks, which is great for getting notified quickly about rapid usage changes in key accounts.

The frequency score is calculated for all companies in the “Using it” state, meaning all companies that are actively using the feature.

To learn more about how this all works in detail, with concrete examples, please read our help article.

More updates soon.

‍

We’ve given our UI a facelift and improved the layout. Most changes are subtle, but they add up and give a much more polished look and feel, and a more scalable interface.

Here’s three highlights:

App layout

Bucket is quickly getting increasingly powerful and configurable, which means we need to up the information density in the UI. We’ve therefore swapped the card-based layout for a flexible full-width layout that gives us more UI real estate. Furthermore, Bucket has a lot of tabular views with large datasets. When exploring multiple rows and columns, like in spreadsheets, you want as much space as possible, especially in the width. We liked the cards, but this change will enable us to grow and match our roadmap much better.

Adoption criteria

The sidebar holds a bunch of necessary feature information, but it’s all information that you only need to read once. For example, the new “Adoption criteria” section. The purpose of this section is to make it clear what data the feature report is based on. For both event- and attribute-based features, you’ll now see either event name or attribute expression in the top right, so you can instantly understand where the data is coming from and trust the feature report metrics.

Navigation polish

While the three main sections (Features, Companies, and Tracking) haven't changed, the nav links now have a better feel to it as the icon changes subtly on active state. The Bucket logo has also gotten a makeover (now rocking our purple brand color) and we've moved the app switcher down to the bottom of the screen and combined it with the user profile card.

More updates soon!

‍

It’s common to track feature usage with attributes. For example, at Bucket, we’re tracking if customers use our weekly reporting feature. We do this by passing an attribute called "weekly_reporting" via Segment’s "group" call.

This is great for ad-hoc analysis. It’s straightforward to query all companies that have "weekly_reporting" set to "true" or "false".

Unfortunately, though, attribute values change over time and its history is lost, which means we can’t know whether customers that aren’t using the weekly reporting feature ("weekly_reporting=false") have never used it, or once were using it, but turned it off.

Additionally, we can’t know when companies started using the reporting feature, and for how long they kept it on.

These things matter a lot. Is there a feature awareness problem or a feature retention problem?

…🥁…

Great news - now you can know! Today, we’re releasing support for attribute-based features. We think this is a really powerful addition to Bucket.

If you’re already tracking attributes in your application, this will just work™ as is. Starting today, Bucket automatically keeps track of all company-level attributes and will automatically track any attribute value changes over time.

(If you’re not familiar with attributes, check out our post from last week: When to track features with events or attributes)

Getting started

Let’s say we work at Slack and that we want to track usage of the third-party Slack apps.

Whenever a user logs into Slack, we’re tracking which company they belong to and send the latest attribute values for the respective company. In code, this looks like this:

Followed by

Whenever a user logs into Slack, we check which Slack apps the user’s workspace is using and report that via the "apps" attribute. As you can see in the snippet above, “Acme Inc” is only using the Datadog app at the moment.

As a developer, simply grabbing the current state and passing it on as attributes is a really convenient workflow. Let Bucket handle the historic changes.

Feature reporting

With the attribute data flowing in, we can start tracking an attribute-based feature on Bucket.

Let’s say we want to track usage of the Stripe Slack app. To do so, we simply create a new feature and define active usage as: "apps" contains "stripe".

Here’s the feature report:

We now know that 6% of our customers use the Stripe app. We can also see that 2% have then stopped using the Stripe app. We can also see when customers installed and uninstalled the app, and who they are. Neat!

How it works

Let’s look at what’s going on under the hood. When the "group" call comes in, per company, we check if the "apps" values match our "apps" contains "stripe" filter. For "Acme Inc" there’s no match, so we place "Acme Inc" in the "Never used it"-state on the feature report.

Later, someone at "Acme Inc" installs the Stripe app. Bucket notices the difference in "apps" value, and runs the filter again. This time, there’s a match! "Acme Inc" is now actively using the Stripe app.

On the Bucket chart, we can even see which week the shift happened.

Later, "Acme Inc" decides to uninstall the Stripe app. Bucket again sees a difference in the "apps" value and now automatically places "Acme Inc" in the "Stopped using it"-state.

Notice that Bucket didn’t just put "Acme Inc" in "Never" when the "stripe" value disappeared. That’s because Bucket keeps track of the history and is aware that "Acme Inc" once used the Stripe app.

On the same chart, Bucket charts when the customer churned from this feature.

We love this chart and use it all the time. For most features, we eagerly check the amount of purple bars after shipping a feature. New users, yay! In the following weeks, though, we also keep track of the amount of red bars, which is just as "exciting" and crucially important.

More updates next week.

We’ve added typeahead search to the various dropdowns in the Bucket UI. It’s now much faster to find the event you’re looking for when tracking a new feature.

Similarly, it’s now much easier to pick the Slack channel in which you want the daily and/or weekly Bucket feature reports.

More updates next week!

This week, we’re adding to last week’s release, which focused on making the feature adoption criteria configurable. Today, we’ve added another configuration, which allows you to toggle the “Stopped”-metrics on or off in your report.

Feature awareness and retention

When you’ve just shipped a feature, you’re focused on feature awareness. Do users even know this feature exists, and how many have tried it?

Once users have found and tried the feature (hopefully!), you become more focused on feature retention. How many users keep using this feature after trying it out?

With our latest feature report configuration, you can now make a report based on your current focus. See below what it looks like in action.

Example: Evaluating a new chat message feature

You have a chat system where your users can send messages to each other. You have a hypothesis that when a user has sent more than 7 messages, they’re hooked and a lot more prone to keep using the product.

This feature is frequent in nature, but to begin with you don’t really care about the recurring use aspect - you just want as many users to 7+ chat messages as possible:

Here’s what that report looks like:

‍

87% of your users have passed the “tried it”-threshold, and are using the chat feature. That’s great!

Now you get curious about feature retention. Do the 87% keep using the chat system after their first 7 messages or not?

Flip a toggle to turn on the retention metrics and get the answer:

Here’s the updated report:

‍

Yikes! There’s red bars every week! 40% haven’t used the chat feature 2 weeks after becoming active users of the feature. This feature needs another iteration to make it stick…

More updates next week :)

One of the most powerful features on Bucket, is the pre-defined adoption states. Out of the box, Bucket buckets (see what we did there?) companies into the following states based on their historic feature usage pattern:

Until today, those state definitions were hard-coded. We believe great products have good defaults and rich configuration. So, from today, it’s now possible to configure the feature adoption KPIs  to suit the nature of the feature you’re tracking.

For example, for some features, companies should be marked as actively using it after just one interaction. For other features, 10 interactions is a better threshold.

On the other end of the spectrum, for some features, churn is when a company hasn’t interacted with the feature in 4 weeks. For other features, 1-2 weeks is more suitable

How it works

When you track a new feature on Bucket, you’ll instantly get the Bucket feature report. To then fine-tune the state definitions, simply click the pencil-icon next to the relevant state.

Here’s what it looks like in action:

More updates next week!

Our charts now show a partial data marker when looking at the current week’s incomplete data set. We indicate partial data by making a dotted line or dashed bar, like so:

We’ve also shipped a useful addition to our Slack integration. When you ship a new feature, it goes into the hands of your customers. But, when do customers start to use it, for real?

Our new Slack report gives you a daily digest of companies that just started using your feature. It’s especially useful in those early days and weeks after shipping a new feature where you want to track newly engaged customers closely to make sure they use the feature as intended.

‍

‍

You can toggle the daily digest on under feature settings, and toggle it off again once the feature looks healthy.

More updates next week!

We’ve just shipped a really useful addition to our feature report’s main chart. Until today, the Using it-chart showed the percentage of companies that are actively using a given feature, per week.

Now, the chart also includes the delta movements in and out of the Using it-state, in absolute numbers. In other words, the chart now shows how many companies, per week, that start using the feature and how many companies that stop using the feature.

Consider this chart. It shows percentage of companies that are actively using a feature:

‍

On the surface it looks like a slow but nice progression in usage, up from 25% to 50%. Nice, right? Not really! It turns out, this feature has a big churn issue, which is keeping it from growing much faster.

‍

Overlaying delta movements

Now consider this chart: It shows the same percentage, but now includes the weekly delta movements (right y-axis):

‍

‍

The light purple and light red bars indicate the amount of companies either starting or stopping use of this feature, per week. You ideally want many new users and only a few of them churning over time. In this updated chart, however, it's very clear that even though the percentage of active companies is trending upwards, there’s a large number of companies that churn every week. Yikes!

The good thing is that Bucket also shows you who the churned companies are, so you can investigate further and perhaps even reach out to some of them to learn why they churned - and how you can improve your feature in the next iteration.

To keep track of your features, get the Bucket feature report, including delta movements, on Slack every week, so you know, if your feature’s health is good or not — without breaking a sweat.

To learn more about how this works under the hood, check out our product handbook article.

Happy shipping!