Configure a Metric - Confidence Documentation

The tutorial consists of the following steps:

Create an entity that represents the users that are part of your experiments.
Create an assignment table that tells Confidence how your entity is assigned to experiments.
Create a fact table that makes some measurable facts available for metrics.
Create a metric that aggregates a measurement from the fact table.

If you already have an entity and an assignment table you can skip directly to create a fact table.

This page targets the following audiences:

Data Engineers or Data Scientists who want to set up Confidence Metrics for their organization.

Before you begin:

You need to have a Confidence account.
(Step 2-3) You need to have connected Confidence Metrics to your data warehouse.
(Step 2-3) You need to have data for metrics in your data warehouse.

Use this guide to set up a metric from scratch. Use the metric in the rollout quickstart to measure the effect of the change you roll out.

Create an Entity

An entity is a representation of the users that are part of your experiments. Entities can be anything really, but typically they are users, customers, or visitors. In Confidence, the entity connects other concepts such as experiments, variants, facts, dimensions and metrics to each other. It’s via the entity that Confidence can understand how it should traverse the data in your data warehouse. For example, when you configure an experiment to use the User entity, Confidence only considers metrics that use facts that have a relationship to the User entity. Confidence comes with two default entities: User and Visitor. If you want to experiment on and create metrics for another entity, follow these steps.

Go to Confidence

Navigate to Entities

On the left sidebar, select Admin > Entities.

Click + Create

Enter User as the name of the entity

Select a data type

Select a data type for how your data warehouse represents the entity. For the sake of this tutorial, select String.

Click Create

🎉 That’s it! You have now created an entity that you can tie other concepts to in Confidence. In the next section, you use the entity when creating an assignment table so that Confidence knows how your new entity is assigned to experiments and variants.

Create an Assignment Table

If you have already set up an assignment table, you can skip this section. Jump straight to Create a metric.

The most fundamental part of Confidence Metrics is the experiment assignment data. Without it you can’t compute metrics for an A/B test or any type of experiment. Assignment data is a log of records that contain information about how users were assigned to experiments and variants. Confidence needs the following data:

A timestamp that indicates when the assignment happened.
An entity identifier, normally a user identifier, that uniquely identifies the user that was assigned a variant.
An experiment identifier that identifies the experiment that the user was assigned to.
A variant identifier that identifies the variant that the user was assigned to.

Any table in your data warehouse can store the assignment data. For Confidence to understand the data, you need to tell Confidence where to find it and how to interpret it. You define this in your Assignment Table, which is a query that projects the data outlined above. You can write the query in any SQL dialect that your data warehouse supports. For this tutorial, your assignment data should exist in a table called assignment_log in your data warehouse. Data should be continuously appended to the table as users are assigned to experiments. The table has the following schema:

CREATE TABLE assignment_log (
  timestamp TIMESTAMP,
  user_id STRING,
  experiment_id STRING,
  variant_id STRING
)

Follow these steps to set up an assignment table:

Go to Confidence

Navigate to Assignment tables

On the left sidebar, select Admin > Assignment tables.

Click + Create

Give the table a name

Give the table a name, such as assignment. If your data comes from a particular feature flagging system, you can name it after that, for example launchdarkly.

Enter the SQL query

Enter the SQL query that projects the assignment data from your data warehouse.

SELECT
  timestamp,
  user_id,
  experiment_id,
  variant_id
FROM assignment_log

SELECT
  timestamp,
  user_id,
  experiment_id,
  variant_id
FROM assignment_log

SELECT
  timestamp,
  user_id,
  experiment_id,
  variant_id
FROM assignment_log

Click Run Query

This executes the query and shows you a preview.

To make the query cheap, the query runs with a LIMIT clause to limit the number of rows.

Map the columns

To the right of the result table, you see a form where you can specify the columns in the result that correspond to the assignment data. It’s this mapping that tells Confidence how to interpret the data. Fill in the form as follows:

timestamp as the timestamp column.
For entity, select User and enter user_id as the entity column.
experiment_id as the experiment key column.
variant_id as the variant key column.

Configure data delivery cadence

For Data delivery cadence select Data is delivered continuously. Leave Commit delay at its default value. You can read more about these settings on the data delivery cadence page.

Click Create

You have now set up an assignment table and can move on to create fact tables and metrics for your experiments.

Create a Fact Table

Facts are measurable data that you want to use in your metrics. Facts can be anything that you can measure, such as revenue, number of clicks, or number of impressions. Facts are stored in tables in your data warehouse. For Confidence to understand the data, you need to tell Confidence where to find it and how to interpret it. This tutorial assumes you have your fact data in a table called sales in your data warehouse. Each time a sale occurs, the system adds a new row to the data. amount is the sales amount and product is the product sold. Assume that data is continuously appended to the table as sales happen. The table has the following schema:

CREATE TABLE sales (
  timestamp TIMESTAMP,
  user_id STRING,
  amount FLOAT,
  product STRING
)

Follow these steps to set up a fact table:

Go to Confidence

Navigate to Fact tables

On the left sidebar, select Admin > Fact tables.

Click + Create

Give the table a name

Give the table a name, such as sales.

Enter the SQL query

Enter the SQL query that projects the fact data from your data warehouse.

SELECT
  timestamp,
  user_id,
  amount,
  product
FROM sales

SELECT
  timestamp,
  user_id,
  amount,
  product
FROM sales

SELECT
  timestamp,
  user_id,
  amount,
  product
FROM sales

Click Run Query

This executes the query and shows you a preview.

Map the columns

To the right of the result table, you see a form where you can specify the columns in the result that correspond to the fact data. It’s this mapping that tells Confidence how to interpret the data. Fill in the form as follows:

Select timestamp as the timestamp column.
For entity, select User and select user_id as the entity column.
Add amount to the list of measurements.
Add product to the list of dimensions.

Configure data delivery cadence

For Data delivery cadence select Data is delivered continuously. Leave Commit delay at its default value. You can read more about these settings on the data delivery cadence page.

Click Create

Create a Metric

The final step is to create a metric that aggregates the fact data. A metric can be any aggregation of the fact data, such as sum, average, count, etc. All metrics aggregate measurements over some time window. Watch the following video for a 4-minute overview of the different ways to handle time in metrics.

For this tutorial, create a metric that computes the average weekly sales amount per user. To create a metric follow these steps:

Go to Confidence

Navigate to Metrics

On the left sidebar, select Metrics.

Click + Create

Enter metric details

Enter Average weekly sales per user as the name of the metric and assign yourself as the owner. Skip the description for now.

Select Consumption as the metric type

Select the consumption value

Select the User entity, the sales fact table, and the amount measurement as the Consumption value.

Add attribute criteria

Click Add Attribute criteria and select content_type as the attribute. Set the filter to content_type is podcast.

Configure when to include users

In the When to Include Users In Metrics Results step, select Cumulatively during a window and 1 week starting at exposure.

Click Next to skip the Suggested usage

Review and create

Review your metric setup, and then click Create.

That’s it! You have now created your first metric. You can now create an A/B test using the metric. Well done.

Launch an A/B Test

Use your metric to measure experiment results

Launch a Rollout

Use your metric to monitor a gradual release

Metrics Reference

Deep dive into metric types and configuration

Validate Metrics

Learn how to verify your metrics are working correctly

​Create an Entity

​Create an Assignment Table

​Create a Fact Table

​Create a Metric

​Related Resources

Launch an A/B Test

Launch a Rollout

Metrics Reference

Validate Metrics

Create an Entity

Create an Assignment Table

Create a Fact Table

Create a Metric

Related Resources