# Allocate Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/allocate-segment
/api/flags/api/segment.openapi.json post /v1/segments/{segment}:allocate
Allocate the segment so that it can be used. Note, if the segment is
coordinated, it can only be allocated if there is remaining space.
# Applys Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/applys-flag
/api/flags/api/resolve.openapi.json post /v1/flags:apply
Indicates that resolved values of a set of flags have been used. In many
situations there is a delay between the moment a flag is resolved and
when it is actually used in a client. This is often the case in mobile
clients where you typically batch resolve all flags at startup, but then
apply them later when the user interacts with a specific view. If the
`apply` flag is set to false in a resolve, the flag assignment event is
delayed until the flag is applied.
# Archive Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/archive-flag
/api/flags/api/flag.openapi.json post /v1/flags/{flag}:archive
Archive a flag. It will no longer be possible to resolve this flag, and it
will no be included in batch resolves.
# Archive Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/archive-segment
/api/flags/api/segment.openapi.json post /v1/segments/{segment}:archive
Archive the segment. To archive the segment it must be `ALLOCATED` and
there must be no rules currently referencing the segment.
# Availablespace Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/availablespace-segment
/api/flags/api/segment.openapi.json post /v1/segments/{segment}:segmentAvailableSpace
Check if there is space available for a specific segment to be allocated.
Used when a segment is coordinated before trying to allocate it.
# Cancel MetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/cancel-metriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/scheduledMetricCalculations/{scheduledMetricCalculation}/metricCalculations/{metricCalculation}:cancel
Cancels the metric calculation. Only applicable for a currently running metric calculation.
# Cancel ScheduledExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/cancel-scheduledexposurecalculation
/api/metrics/api/exposure-calculation.openapi.json post /v1/scheduledExposureCalculations/{scheduledExposureCalculation}:cancel
Cancel the scheduled exposure calculation. The calculation will still exist
but no new computations will be scheduled.
# Cancel ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/cancel-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/scheduledMetricCalculations/{scheduledMetricCalculation}:cancel
Cancel but do not delete a scheduled metric calculation. The schedule still
exists, but no metric computations are performed.
# Clone Clone
Source: https://confidence-auth-testing.mintlify.io/api-reference/clone-clone
/api/experiments/abtest.openapi.json post /v1/workflows/abtest/instances/{instance}:clone
Clones an existing abtest.
# Computespacecalendar Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/computespacecalendar-segment
/api/flags/api/segment.openapi.json post /v1/segments:computeSegmentSpaceCalendar
Compute a calendar view of space availability over time for a set of segments
with prospective start and end dates. This allows planning segment
allocation by showing how much space will be available on each date
in the requested range.
# Create Abtest
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-abtest
/api/experiments/abtest.openapi.json post /v1/workflows/abtest
Creates a new Abtest.
# Create AnalysisResult
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-analysisresult
/api/stats/api/analysis-result.openapi.json post /v2/workflows/{workflow}/instances/{instance}/analysisResults
# Create ApiClient
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-apiclient
/api/iam/api/api-client.openapi.json post /v1/apiClients
Create a new API client. An API client is used to programmatically interact
with the Confidence APIs.
# Create AssignmentTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-assignmenttable
/api/metrics/api/assignment-table.openapi.json post /v1/assignmentTables
Create an assignment table. An assignment table is a description of a
dataset of how flags have been assigned to variants. Specifically, it
should contain the id of the entity that was resolved, when it was resolved,
and which variant was assigned.
# Create Client
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-client
/api/iam/api/client.openapi.json post /v1/clients
Create a new client. A `Client` is used to resolve flags from all
applications of a particular type, like an iOS client. To resolve a flag
you need to pass credentials from a specific `Client` and have the flag
enabled for that client.
# Create ClientCredential
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-clientcredential
/api/iam/api/client.openapi.json post /v1/clients/{client}/credentials
Create a new client credential. A credential is a type of secret used to
authenticate a client when resolving flags.
# Create DataWarehouse
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-datawarehouse
/api/metrics/api/data-warehouse.openapi.json post /v1/dataWarehouses
Create a new data warehouse. This endpoint is used to configure access to
your data warehouse, were all assignment, exposure and metric data will be
stored and computed. The endpoint will verify that access to relevant
resources are set up correctly.
# Create DimensionTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-dimensiontable
/api/metrics/api/dimension-table.openapi.json post /v1/dimensionTables
Create a dimension table. A dimension table is a description of a dataset
that contains dimensions about an entity that are not included in the
fact table. For example, it could be general dimensions like the country
of a user, that can be used across many metrics. The dimension table maps
out the entity column and the dimension columns in the table.
# Create Entity
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-entity
/api/metrics/api/entity.openapi.json post /v1/entities
Create a new entity. An entity is a class of instances that can be
uniquely identified and measured. A typical example is a user, but can
also be something more abstract like a bank transaction.
# Create ExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-exposurecalculation
/api/metrics/api/exposure-calculation.openapi.json post /v1/exposureCalculations
Calculates exposure from an assignment table.
# Create ExposureTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-exposuretable
/api/metrics/api/exposure-table.openapi.json post /v1/exposureTables
Create an exposure table. An exposure table is a general description
of how to compute exposures for a specific entity. Unlike the assignment
table, the exposure table is more specific, and typically one is created
for each experiment.
# Create FactTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-facttable
/api/metrics/api/fact-table.openapi.json post /v1/factTables
Create a new fact table. A fact table is a description of a dataset,
specifically, it describes which columns correspond to entities,
dimensions and measurements. It is the basis for describing metrics.
# Create Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-flag
/api/flags/api/flag.openapi.json post /v1/flags
Creates a new flag. Flags are used to control part of an application by
defining a set of rules that deterministically or randomly assign variants
to users.
Only the `flag_id` is required when creating the flag, the remaining
attributes can be set by calling `UpdateFlag`. When creating a flag it will
start in the `ACTIVE` state, when you no longer need a flag you can call
`ArchiveFlag`.
# Create Group
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-group
/api/iam/api/group.openapi.json post /v1/groups
Creates a new group
# Create MaterializedSegment
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-materializedsegment
/api/flags/api/materialized-segments.openapi.json post /v1/materializedSegments
Create a new materialized segment. The materialization can then be attached to rules.
# Create Metric
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-metric
/api/metrics/api/metric.openapi.json post /v1/metrics
Create a metric. A metric is a general description of how to aggregate
a measurement from a fact table to an entity. For example, averaging the
number of seconds a user has been on a page, from a fact table of page
views.
# Create MetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-metriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/metricCalculations
Create a new metric calculation. The metric calculation represents the
calculation of a metric for a specific time window, for example, the date
2020-01-01, or the hour 2020-01-01T01. It is a long running operation and
the current status can be seen in the `state` field of the
`MetricCalculation`.
# Create Role
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-role
/api/iam/api/role.openapi.json post /v1/roles
Creates a new role
# Create Rollout
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-rollout
/api/experiments/rollout.openapi.json post /v1/workflows/rollout
Creates a new Rollout.
# Create Rule
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-rule
/api/flags/api/flag.openapi.json post /v1/flags/{flag}/rules
Create a new rule. A rule decides at which proportions user from a specific
segment should be assigned variants from the flag.
# Create ScheduledExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-scheduledexposurecalculation
/api/metrics/api/exposure-calculation.openapi.json post /v1/scheduledExposureCalculations
Create a scheduled exposure calculation. The schedule defines the start
time and frequency that the exposure table should be calculated. The
exposure table is calculated by finding the first assignment of each entity
in the referenced assignment table.
# Create ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/scheduledMetricCalculations
Create a scheduled metric calculation. The schedule defines the start
time and frequency that a metric should be calculated for a specific
exposure table.
# Create Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-segment
/api/flags/api/segment.openapi.json post /v1/segments
Create a new segment. A segment defines a part of the population of an
entity, for example, the population of users. It contains an overall
proportion of the population as well as optional targeting criteria, like
the country a user is from. A segment can also be coordinated with other
segments, by setting `exclusivity_tags` and `exclusive_to`, preventing one
user to be included in two segments, that are exclusive to each other,
simultaneously.
Note, the segment is not active until it is allocated.
# Create Surface
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-surface
/api/workflows/api/surface.openapi.json post /v1/surfaces
# Create UserInvitation
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-userinvitation
/api/iam/api/user.openapi.json post /v1/userInvitations
Creates a new invitation for a user and sends an email with a login link.
# Create Variant
Source: https://confidence-auth-testing.mintlify.io/api-reference/create-variant
/api/flags/api/flag.openapi.json post /v1/flags/{flag}/variants
Creates a new variant and add it to the given flag. A variant is a named
value that typically represents a group of settings that can be used to
control part of an application.
Before you can create a variant the flag must have a schema, and the
variant must satisfy the schema.
# Delete AnalysisResult
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-analysisresult
/api/stats/api/analysis-result.openapi.json delete /v2/workflows/{workflow}/instances/{instance}/analysisResults/{analysisResult}
# Delete ApiClient
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-apiclient
/api/iam/api/api-client.openapi.json delete /v1/apiClients/{apiClient}
Delete an API client.
# Delete AssignmentTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-assignmenttable
/api/metrics/api/assignment-table.openapi.json delete /v1/assignmentTables/{assignmentTable}
Delete an assignment table. The assignment table can only be deleted if
there are no active scheduled exposure calculations referencing the table.
# Delete Client
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-client
/api/iam/api/client.openapi.json delete /v1/clients/{client}
Delete an existing client.
# Delete ClientCredential
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-clientcredential
/api/iam/api/client.openapi.json delete /v1/clients/{client}/credentials/{credential}
Delete a client credential.
# Delete DimensionTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-dimensiontable
/api/metrics/api/dimension-table.openapi.json delete /v1/dimensionTables/{dimensionTable}
Delete a dimension table. A dimension table can only be deleted if no
metrics are using it.
# Delete Entity
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-entity
/api/metrics/api/entity.openapi.json delete /v1/entities/{entity}
Delete an entity. The entity cannot be used by a fact table or metric.
# Delete ExposureTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-exposuretable
/api/metrics/api/exposure-table.openapi.json delete /v1/exposureTables/{exposureTable}
Deletes an entity relation table.
# Delete FactTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-facttable
/api/metrics/api/fact-table.openapi.json delete /v1/factTables/{factTable}
Delete a fact table. The fact table can only be deleted if there are no
metrics referencing it.
# Delete Group
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-group
/api/iam/api/group.openapi.json delete /v1/groups/{group}
Deletes a group.
# Delete MaterializedSegment
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-materializedsegment
/api/flags/api/materialized-segments.openapi.json delete /v1/materializedSegments/{materializedSegment}
Delete a materialized segment.
# Delete Metric
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-metric
/api/metrics/api/metric.openapi.json delete /v1/metrics/{metric}
Delete a metric. A metric can only be deleted if there are no scheduled
metric calculations referencing it.
# Delete Role
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-role
/api/iam/api/role.openapi.json delete /v1/roles/{role}
Deletes a role
# Delete Rule
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-rule
/api/flags/api/flag.openapi.json delete /v1/flags/{flag}/rules/{rule}
Delete a rule. Note, entities that previously matched this rule, will
now be passed on to the next rule.
# Delete ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json delete /v1/scheduledMetricCalculations/{scheduledMetricCalculation}
Delete a scheduled metric calculation. A scheduled metric calculation can
always be deleted.
# Delete Surface
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-surface
/api/workflows/api/surface.openapi.json delete /v1/surfaces/{surface}
# Delete User
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-user
/api/iam/api/user.openapi.json delete /v1/users/{user}
Delete user by name.
# Delete UserInvitation
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-userinvitation
/api/iam/api/user.openapi.json delete /v1/userInvitations/{userInvitation}
Revokes an existing invitation.
# Delete Variant
Source: https://confidence-auth-testing.mintlify.io/api-reference/delete-variant
/api/flags/api/flag.openapi.json delete /v1/flags/{flag}/variants/{variant}
Deletes a variant. Variants that are used by one or more rules cannot be
deleted.
# Deriveevaluationcontextschema Client
Source: https://confidence-auth-testing.mintlify.io/api-reference/deriveevaluationcontextschema-client
/api/flags/api/flag.openapi.json post /v1/clientEvaluationContextSchema:derive
Infer the schema of the evaluation context for a specific set of clients,
that has previously been used to resolve one or more flags.
# Describe AssignmentTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/describe-assignmenttable
/api/metrics/api/assignment-table.openapi.json post /v1/assignmentTable:describe
Describe the assignments table
# Execute AddExploratoryAnalysis
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-addexploratoryanalysis
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:addexploratoryanalysis
Executes the addExploratoryAnalysis function.
# Execute ArchiveAction
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-archiveaction
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:archive
Executes the archive action
# Execute ConvertToRolloutAction
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-converttorolloutaction
/api/experiments/abtest.openapi.json post /v1/workflows/abtest/instances/{instance}:converttorollout
Executes the convertToRollout action
# Execute DeleteExploratoryAnalysis
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-deleteexploratoryanalysis
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:deleteexploratoryanalysis
Executes the deleteExploratoryAnalysis function.
# Execute DeleteReviewResponse
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-deletereviewresponse
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:deletereviewresponse
Executes the deleteReviewResponse function.
# Execute EditExploratoryAnalysis
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-editexploratoryanalysis
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:editexploratoryanalysis
Executes the editExploratoryAnalysis function.
# Execute EndAction
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-endaction
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:end
Executes the end action
# Execute ExecuteScheduledIncrease
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-executescheduledincrease
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:executescheduledincrease
Executes the executeScheduledIncrease function.
# Execute GetExploratoryAnalysis
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-getexploratoryanalysis
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:getexploratoryanalysis
Executes the getExploratoryAnalysis function.
# Execute GetMandatoryMetric
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-getmandatorymetric
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:getmandatorymetrics
Executes the getMandatoryMetrics function.
# Execute GetRolloutInfo
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-getrolloutinfo
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:getrolloutinfo
Executes the getRolloutInfo function.
# Execute LaunchAction
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-launchaction
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:launch
Executes the launch action
# Execute PauseIntake
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-pauseintake
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:pauseintake
Executes the pauseIntake function.
# Execute RefreshExploratoryAnalysisCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-refreshexploratoryanalysiscalculation
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:refreshexploratoryanalysiscalculations
Executes the refreshExploratoryAnalysisCalculations function.
# Execute RequestReview
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-requestreview
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:requestreviews
Executes the requestReviews function.
# Execute ResumeIntake
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-resumeintake
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:resumeintake
Executes the resumeIntake function.
# Execute RetriggerExposure
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-retriggerexposure
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:retriggerexposure
Retrigger the exposure calculation for this experiment. This will also retrigger all dependent metric calculations if 'retriggerDependentMetrics' is set to true.
# Execute RetriggerMetric
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-retriggermetric
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:retriggermetric
Retrigger a specific metric calculation for this experiment.
# Execute SubmitReviewResponse
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-submitreviewresponse
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:submitreviewresponse
Executes the submitReviewResponse function.
# Execute UpdateReview
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-updatereview
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:updatereviews
Executes the updateReviews function.
# Execute UpdateSegment
Source: https://confidence-auth-testing.mintlify.io/api-reference/execute-updatesegment
/api/experiments/rollout.openapi.json post /v1/workflows/rollout/instances/{instance}:updatesegment
Executes the updateSegment function.
# Exists DataWarehouse
Source: https://confidence-auth-testing.mintlify.io/api-reference/exists-datawarehouse
/api/metrics/api/data-warehouse.openapi.json get /v1/dataWarehouses:exists
Checks if a data warehouse is currently configured
# Get Abtest
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-abtest
/api/experiments/abtest.openapi.json get /v1/workflows/abtest/instances/{instance}
Retrieves a abtest by name.
# Get AnalysisResult
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-analysisresult
/api/stats/api/analysis-result.openapi.json get /v2/workflows/{workflow}/instances/{instance}/analysisResults/{analysisResult}
# Get ApiClient
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-apiclient
/api/iam/api/api-client.openapi.json get /v1/apiClients/{apiClient}
Fetch an API client.
# Get AssignmentTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-assignmenttable
/api/metrics/api/assignment-table.openapi.json get /v1/assignmentTables/{assignmentTable}
Get an assignment table.
# Get Client
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-client
/api/iam/api/client.openapi.json get /v1/clients/{client}
Get an existing client.
# Get ClientCredential
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-clientcredential
/api/iam/api/client.openapi.json get /v1/clients/{client}/credentials/{credential}
Get an existing client credential.
# Get DataWarehouse
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-datawarehouse
/api/metrics/api/data-warehouse.openapi.json get /v1/dataWarehouses/{dataWarehouse}
Get the data warehouse.
# Get DimensionTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-dimensiontable
/api/metrics/api/dimension-table.openapi.json get /v1/dimensionTables/{dimensionTable}
Get a dimension table.
# Get Entity
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-entity
/api/metrics/api/entity.openapi.json get /v1/entities/{entity}
Get the given entity.
# Get ExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-exposurecalculation
/api/metrics/api/exposure-calculation.openapi.json get /v1/scheduledExposureCalculations/{scheduledExposureCalculation}/exposureCalculations/{exposureCalculation}
Get an existing exposure calculation.
# Get ExposureTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-exposuretable
/api/metrics/api/exposure-table.openapi.json get /v1/exposureTables/{exposureTable}
Get the exposure table.
# Get FactTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-facttable
/api/metrics/api/fact-table.openapi.json get /v1/factTables/{factTable}
Get a fact table.
# Get Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-flag
/api/flags/api/flag.openapi.json get /v1/flags/{flag}
Get a flag.
# Get Group
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-group
/api/iam/api/group.openapi.json get /v1/groups/{group}
Gets a group by name.
# Get MaterializedSegment
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-materializedsegment
/api/flags/api/materialized-segments.openapi.json get /v1/materializedSegments/{materializedSegment}
Get a materialized segment.
# Get Metric
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-metric
/api/metrics/api/metric.openapi.json get /v1/metrics/{metric}
Get a metric.
# Get MetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-metriccalculation
/api/metrics/api/metric-calculation.openapi.json get /v1/scheduledMetricCalculations/{scheduledMetricCalculation}/metricCalculations/{metricCalculation}
Get an existing metric calculation.
# Get Role
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-role
/api/iam/api/role.openapi.json get /v1/roles/{role}
Fetch a single role.
# Get Rollout
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-rollout
/api/experiments/rollout.openapi.json get /v1/workflows/rollout/instances/{instance}
Retrieves a rollout by name.
# Get Rule
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-rule
/api/flags/api/flag.openapi.json get /v1/flags/{flag}/rules/{rule}
Get a rule of a flag.
# Get ScheduledExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-scheduledexposurecalculation
/api/metrics/api/exposure-calculation.openapi.json get /v1/scheduledExposureCalculations/{scheduledExposureCalculation}
Get the scheduled exposure calculation.
# Get ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json get /v1/scheduledMetricCalculations/{scheduledMetricCalculation}
Get the scheduled metric calculation.
# Get Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-segment
/api/flags/api/segment.openapi.json get /v1/segments/{segment}
Get a segment.
# Get Surface
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-surface
/api/workflows/api/surface.openapi.json get /v1/surfaces/{surface}
# Get User
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-user
/api/iam/api/user.openapi.json get /v1/users/{user}
Fetch a single user.
# Get UserInvitation
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-userinvitation
/api/iam/api/user.openapi.json get /v1/userInvitations/{userInvitation}
Get an invitation.
# Get Variant
Source: https://confidence-auth-testing.mintlify.io/api-reference/get-variant
/api/flags/api/flag.openapi.json get /v1/flags/{flag}/variants/{variant}
Get a specific variant of the given flag.
# List Abtest
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-abtest
/api/experiments/abtest.openapi.json get /v1/workflows/abtest/instances
Lists abtests with optional filtering and pagination.
# List AnalysisResult
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-analysisresult
/api/stats/api/analysis-result.openapi.json get /v2/workflows/{workflow}/instances/{instance}/analysisResults
# List ApiClient
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-apiclient
/api/iam/api/api-client.openapi.json get /v1/apiClients
List API clients.
# List AssignmentTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-assignmenttable
/api/metrics/api/assignment-table.openapi.json get /v1/assignmentTables
List assignment tables. The assignment tables are returned in no particular
order.
# List Client
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-client
/api/iam/api/client.openapi.json get /v1/clients
List clients. Clients are listed in no particular order.
# List ClientCredential
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-clientcredential
/api/iam/api/client.openapi.json get /v1/clients/{client}/credentials
List client credentials. Credentials are listed in no particular order.
# List DataWarehouse
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-datawarehouse
/api/metrics/api/data-warehouse.openapi.json get /v1/dataWarehouses
List data warehouses. The data warehouses are returned in no particular
order.
# List DimensionTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-dimensiontable
/api/metrics/api/dimension-table.openapi.json get /v1/dimensionTables
List dimension tables. The dimension tables are returned in no particular
order.
# List Entity
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-entity
/api/metrics/api/entity.openapi.json get /v1/entities
List entities. Entities are returned in no particular order.
# List ExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-exposurecalculation
/api/metrics/api/exposure-calculation.openapi.json get /v1/scheduledExposureCalculations/{scheduledExposureCalculation}/exposureCalculations
List exposure calculations from a schedule. The results are returned from latest to oldest.
# List ExposureTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-exposuretable
/api/metrics/api/exposure-table.openapi.json get /v1/exposureTables
List exposure tables. The exposure tables are returned in no particular
order.
# List FactTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-facttable
/api/metrics/api/fact-table.openapi.json get /v1/factTables
List fact tables. The fact tables are returned in no particular order.
# List Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-flag
/api/flags/api/flag.openapi.json get /v1/flags
List all flags optionally filtered by state. The flags are ordered by
`updateTime`.
# List Group
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-group
/api/iam/api/group.openapi.json get /v1/groups
List the groups.
# List MaterializedSegment
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-materializedsegment
/api/flags/api/materialized-segments.openapi.json get /v1/materializedSegments
List all materialized segments.
# List Metric
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-metric
/api/metrics/api/metric.openapi.json get /v1/metrics
List metrics. The metrics are in no particular order.
# List MetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-metriccalculation
/api/metrics/api/metric-calculation.openapi.json get /v1/scheduledMetricCalculations/{scheduledMetricCalculation}/metricCalculations
List metric calculations from a schedule. The results are returned in no
particular order.
# List Role
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-role
/api/iam/api/role.openapi.json get /v1/roles
List roles.
# List Rollout
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-rollout
/api/experiments/rollout.openapi.json get /v1/workflows/rollout/instances
Lists rollouts with optional filtering and pagination.
# List Rule
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-rule
/api/flags/api/flag.openapi.json get /v1/flags/{flag}/rules
List all rules of the flag. Rules are returned in the same order as they
are evaluated.
# List ScheduledExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-scheduledexposurecalculation
/api/metrics/api/exposure-calculation.openapi.json get /v1/scheduledExposureCalculations
List scheduled exposure calculations. The results are returned in no
particular order.
# List ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json get /v1/scheduledMetricCalculations
List scheduled metric calculations. The results are returned in no
particular order.
# List Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-segment
/api/flags/api/segment.openapi.json get /v1/segments
List segments. Segments are listed in no particular order.
# List Surface
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-surface
/api/workflows/api/surface.openapi.json get /v1/surfaces
# List User
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-user
/api/iam/api/user.openapi.json get /v1/users
List users within the organization.
# List UserGroup
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-usergroup
/api/iam/api/user.openapi.json get /v1/users/{user}/groups
List groups for a user.
# List UserInvitation
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-userinvitation
/api/iam/api/user.openapi.json get /v1/userInvitations
List the invitations currently existing for this account.
# List Variant
Source: https://confidence-auth-testing.mintlify.io/api-reference/list-variant
/api/flags/api/flag.openapi.json get /v1/flags/{flag}/variants
List all the variants of this flag. The variants are returned in the order
they were added to the flag.
# Query MetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/query-metriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/scheduledMetricCalculations/{scheduledMetricCalculation}/metricCalculations/{metricCalculation}:query
Query results for a single metric calculation.
# Query ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/query-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/scheduledMetricCalculations/{scheduledMetricCalculation}:query
Query results from a given metric calculation schedule. This is a general endpoint
for querying and aggregating metric data. The aggregation can be performed
over both time and specific dimensions.
# Queryreferences Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/queryreferences-segment
/api/flags/api/segment.openapi.json get /v1/segments/{segment}:queryReferences
Query all segments that reference the specified segment in their targeting.
# Queryusage Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/queryusage-flag
/api/flags/api/flag.openapi.json post /v1/flags/{flag}/usage:query
Query time series data about how much this flag was used recently. You
cannot query more than 7 days back in time.
# Request AccessToken
Source: https://confidence-auth-testing.mintlify.io/api-reference/request-accesstoken
/api/iam/api/auth.openapi.json post /v1/oauth/token
# Resolves Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/resolves-flag
/api/flags/api/resolve.openapi.json post /v1/flags:resolve
Resolve multiple flags into variants and values. This method resolves
all flags that are enabled for the given client, or a subset of them
specified in the request.
A flag is resolved by evaluating its rules in order, a rule matches if:
1) it is enabled, 2) the referred segment is active, and 3) the
randomization unit is in the population indicated by the segment's
targeting criteria and population allocation. The first rule that matches
will assign a variant and value to the unit. Archived flags are not included.
# Resume ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/resume-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/scheduledMetricCalculations/{scheduledMetricCalculation}:resume
Resumes the scheduled metric calculation.
# Retrigger MetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/retrigger-metriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/scheduledMetricCalculations/{scheduledMetricCalculation}/metricCalculations/{metricCalculation}:retrigger
Retriggers a completed metric calculation. This will invalidate the results of the existing
calculation and replace with the new results. Can only be done if the current calculation is completed
and not already retriggered. Returns the new metric calculation.
# Retrigger ScheduledExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/retrigger-scheduledexposurecalculation
/api/metrics/api/exposure-calculation.openapi.json post /v1/scheduledExposureCalculations/{scheduledExposureCalculation}:retrigger
Retriggers the scheduled exposure calculation. This will truncate the exposure table and reset the schedule
# Retrigger ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/retrigger-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json post /v1/scheduledMetricCalculations/{scheduledMetricCalculation}:retrigger
Retriggers the scheduled metric calculation. This will directly retrigger all calculations that have
been run so far
# Run Analysis
Source: https://confidence-auth-testing.mintlify.io/api-reference/run-analysis
/api/stats/api/analysis.openapi.json post /v2/stats:runAnalysis
Runs an analysis that tests multiple hypotheses and evaluates their results
by an optional decision rule.
# Run PowerAnalysis
Source: https://confidence-auth-testing.mintlify.io/api-reference/run-poweranalysis
/api/stats/api/power.openapi.json post /v2/stats:runPowerAnalysis
Runs a power analysis.
# Abtest
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest
Represents an Abtest.
# Abtest.AbTestData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-abtestdata
Module data for abtest.
# Abtest.DecisionRecordData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-decisionrecorddata
Module data for decision.
# Abtest.ExploreData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-exploredata
Module data for explore.
# Abtest.ExposureData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-exposuredata
Module data for exposure.
# Abtest.FlagsData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-flagsdata
Module data for flags.
# Abtest.HypothesisData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-hypothesisdata
Module data for hypothesis.
# Abtest.MetricsData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-metricsdata
Module data for metrics.
# Abtest.PlanningData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-planningdata
Module data for planning.
# Abtest.ReportData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-reportdata
Module data for report.
# Abtest.ReviewsData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-reviewsdata
Module data for reviews.
# Abtest.SampleSizeData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-samplesizedata
Module data for samplesize.
# Abtest.StatsData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/abtest-statsdata
Module data for stats.
# AccessToken
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/accesstoken
Schema reference for the AccessToken object in the Confidence API.
# AnalysisData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/analysisdata
Data used for the analysis.
# AnalysisPlan
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/analysisplan
The description of which hypotheses are going to be tested, which group comparisons are included, and how the result should be evaluated.
# AnalysisResult
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/analysisresult
The result of a statistical analysis for an experiment instance.
# ApiClient
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/apiclient
An API client that can be used to programmatically access the Confidence APIs.
# AssignmentTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/assignmenttable
An assignment table is a description of a dataset of how entities have been assigned to variants.
# Client
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/client
A Client allows an app to access certain Confidence services like resolving flags and sending events.
# ClientCredential
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/clientcredential
The credentials required for a client to access Confidence.
# DimensionTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/dimensiontable
A description of a dataset that contains one or more dimensions of an entity.
# Entity
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/entity
Description of a uniquely identifiable entity.
# ExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/exposurecalculation
Represents the calculation of an exposure table for a specific time window from an assignment table.
# ExposureTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/exposuretable
An exposure table describes how to compute exposures from assignments.
# FactTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/facttable
A set of events from some business process, for example, a sale occurred.
# Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/flag
A flag controlling how entities are assigned variants.
# Flag.Rule
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/flag-rule
A rule that decides how a subset of the flag's variants are assigned.
# Flag.Variant
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/flag-variant
A possible named value the flag can assign.
# Group
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/group
A group of identities.
# MaterializedSegment
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/materializedsegment
A slice of the entity population that is materialized as individual entity identifiers stored in a database.
# Metric
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/metric
A general description of how to aggregate a measurement from a fact table across entities.
# MetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/metriccalculation
Represents the calculation of a metric for a specific time window.
# PowerAnalysisData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/poweranalysisdata
Data used for the power analysis.
# Role
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/role
A role is a named set of permissions, like `Admin` or `Flag Editor`.
# Rollout
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout
Represents a Rollout.
# Rollout.ExploreData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-exploredata
Module data for explore.
# Rollout.ExposureData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-exposuredata
Module data for exposure.
# Rollout.FlagsData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-flagsdata
Module data for flags.
# Rollout.MetricsData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-metricsdata
Module data for metrics.
# Rollout.PlanningData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-planningdata
Module data for planning.
# Rollout.ReviewsData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-reviewsdata
Module data for reviews.
# Rollout.RolloutData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-rolloutdata
Module data for rollout.
# Rollout.SampleSizeData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-samplesizedata
Module data for samplesize.
# Rollout.StatsData
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/rollout-statsdata
Module data for stats.
# ScheduledExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/scheduledexposurecalculation
Represents the schedule of an exposure calculation.
# ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/scheduledmetriccalculation
Represents a schedule of metric calculations.
# Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/segment
A reusable slice of an entity population.
# Surface
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/surface
A surface that represents some area of an application that can be experimented on.
# Targeting
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/targeting
Schema reference for the Targeting object in the Confidence API.
# Targeting.Criterion
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/targeting-criterion
Schema reference for the Targeting.Criterion object in the Confidence API.
# User
Source: https://confidence-auth-testing.mintlify.io/api-reference/schemas/user
A Confidence user.
# Suggest Metric
Source: https://confidence-auth-testing.mintlify.io/api-reference/suggest-metric
/api/metrics/api/metric.openapi.json post /v1/metrics:suggestMetric
Suggests a metric based on a title and / or description
# Unarchive Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/unarchive-flag
/api/flags/api/flag.openapi.json post /v1/flags/{flag}:unarchive
unarchive a flag that has been archived.
# Undelete FactTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/undelete-facttable
/api/metrics/api/fact-table.openapi.json post /v1/factTables/{factTable}:undelete
Undeletes a fact table.
# Update Abtest
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-abtest
/api/experiments/abtest.openapi.json patch /v1/workflows/abtest/instances/{instance}
Updates an existing abtest.
# Update AnalysisResult
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-analysisresult
/api/stats/api/analysis-result.openapi.json patch /v2/workflows/{workflow}/instances/{instance}/analysisResults/{analysisResult}
# Update ApiClient
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-apiclient
/api/iam/api/api-client.openapi.json patch /v1/apiClients/{apiClient}
Update an existing API client.
# Update AssignmentTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-assignmenttable
/api/metrics/api/assignment-table.openapi.json patch /v1/assignmentTables/{assignmentTable}
Update an assignment table.
# Update Client
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-client
/api/iam/api/client.openapi.json patch /v1/clients/{client}
Update an existing client.
# Update ClientCredential
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-clientcredential
/api/iam/api/client.openapi.json patch /v1/clients/{client}/credentials/{credential}
Update an existing client client credential.
# Update DataWarehouse
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-datawarehouse
/api/metrics/api/data-warehouse.openapi.json patch /v1/dataWarehouses/{dataWarehouse}
Update the data warehouse. It is not possible to switch between data
warehouses, e.g., going from BigQuery to Databricks. The endpoint will
verify that access to relevant resources are set up correctly.
# Update DimensionTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-dimensiontable
/api/metrics/api/dimension-table.openapi.json patch /v1/dimensionTables/{dimensionTable}
Updates the given dimension table.
# Update Entity
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-entity
/api/metrics/api/entity.openapi.json patch /v1/entities/{entity}
Updates the given entity.
# Update ExposureTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-exposuretable
/api/metrics/api/exposure-table.openapi.json patch /v1/exposureTables/{exposureTable}
Update the exposure table.
# Update FactTable
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-facttable
/api/metrics/api/fact-table.openapi.json patch /v1/factTables/{factTable}
Updates the given fact table.
# Update Flag
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-flag
/api/flags/api/flag.openapi.json patch /v1/flags/{flag}
Updates the flag. When updating a flag, all variants must match the schema,
segments and variants referenced by rules must exist.
# Update Group
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-group
/api/iam/api/group.openapi.json patch /v1/groups/{group}
Updates a group
# Update MaterializedSegment
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-materializedsegment
/api/flags/api/materialized-segments.openapi.json patch /v1/materializedSegments/{materializedSegment}
Update a rule. The segment and variants referred to by the rule must exist.
# Update Metric
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-metric
/api/metrics/api/metric.openapi.json patch /v1/metrics/{metric}
Update a metric. This will affect future calculations using this metric
# Update Role
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-role
/api/iam/api/role.openapi.json patch /v1/roles/{role}
Updates a role
# Update Rollout
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-rollout
/api/experiments/rollout.openapi.json patch /v1/workflows/rollout/instances/{instance}
Updates an existing rollout.
# Update Rule
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-rule
/api/flags/api/flag.openapi.json patch /v1/flags/{flag}/rules/{rule}
Update a rule. The segment and variants referred to by the rule must exist.
# Update ScheduledExposureCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-scheduledexposurecalculation
/api/metrics/api/exposure-calculation.openapi.json patch /v1/scheduledExposureCalculations/{scheduledExposureCalculation}
Updates a scheduled exposure calculation.
# Update ScheduledMetricCalculation
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-scheduledmetriccalculation
/api/metrics/api/metric-calculation.openapi.json patch /v1/scheduledMetricCalculations/{scheduledMetricCalculation}
Updates a scheduled metric calculation.
# Update Segment
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-segment
/api/flags/api/segment.openapi.json patch /v1/segments/{segment}
Update the segment. Only segments in state UNALLOCATED can be updated.
# Update Surface
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-surface
/api/workflows/api/surface.openapi.json patch /v1/surfaces/{surface}
# Update User
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-user
/api/iam/api/user.openapi.json patch /v1/users/{user}
Update a user.
# Update Variant
Source: https://confidence-auth-testing.mintlify.io/api-reference/update-variant
/api/flags/api/flag.openapi.json patch /v1/flags/{flag}/variants/{variant}
Updates a variant of a flag. The updated variant must satisfy the schema
of the flag.
# Validate Metric
Source: https://confidence-auth-testing.mintlify.io/api-reference/validate-metric
/api/metrics/api/metric.openapi.json post /v1/metrics:validateMetric
Validates a metric configuration
# Changelog
Source: https://confidence-auth-testing.mintlify.io/changelog
New features, improvements, and fixes in Confidence.
Follow the latest improvements to Confidence. Subscribe through the RSS feed to receive new entries.
## Target string values with contains
String targeting conditions now support **contains**, making it easier to match values when an exact, prefix, or suffix comparison is not appropriate.
## Reorder metrics in exploratory analysis
Metrics in the metrics explorer can now be reordered, giving you more control over how an analysis is organized and presented.
## Schedule materialized segment loads
Materialized segments support scheduled load jobs. Owners can configure when segment data should be refreshed and follow the status of those jobs.
## Additional improvements
* Use two data warehouses during a warehouse migration.
* Search for metric display names containing percent signs.
* Edit credential display names.
* Configure verified login domains through account administration.
* GitHub Enterprise Cloud repositories are supported by GitHub integrations.
## Manage the metric lifecycle
Metrics can now be created as drafts and published when they are ready for use. Lifecycle states are visible in metric lists and pickers, and metrics can be archived or restored.
Verified and code-managed indicators make it clearer which metric definitions have been reviewed or originate from a repository.
## Configure flag rules more easily
The flag rule creation flow has been redesigned with clearer targeting controls and a more focused configuration experience.
## Explore metric definitions
Open a metric directly in the metrics explorer with its relevant configuration already selected. Metric diagnostics also expose more of the data behind calculations.
## Additional improvements
* Filter the metrics catalog by lifecycle state.
* Select numerator and denominator filters in the metric explorer.
* Choose an owner when creating a materialized segment.
* Access documentation, the blog, and support through the header help menu.
## Create metrics from the metrics explorer
The metrics explorer can now create a metric or measurement from the current analysis. Metric filters support **contains** and **does not contain**, and metric pickers show the preferred direction of each metric.
## Inspect materialized segments
Materialized segment details now include Bloom filter information, making storage and distribution characteristics easier to understand.
## Additional improvements
* Rollout schedule editing and validation are clearer.
* Group identities appear in identity selection.
* Theme names can be edited inline.
* Flag rule cards display readable country names instead of country codes.
## Archive flags with active rules
Flags can now be archived without first removing all active rules. This simplifies cleanup while preserving the flag's existing configuration and history.
## Filter experiments by surface properties
Experiment lists on surface pages can be filtered by property, making it easier to find experiments that affect a particular part of a surface.
## Additional improvements
* Randomization and allocation can be optional where the workflow supports it.
* Metric SQL previews correctly support `LIKE` filters.
* Exposure filters are preserved when metric data is updated.
* The updated flag page is now the default experience.
## Explore fact table dimensions
Explorations can now use dimensions from the fact table that backs a metric, in addition to dimensions from related dimension tables.
## Automate flag cleanup
[Automatic flag cleanup](/docs/flags/flag-cleanup) runs scheduled checks for unused flags. Agents can remove unused flags from your codebase and archive them in Confidence. The first release supports GitHub and Cursor, with more integrations and cleanup capabilities planned.
To get started, go to **Flags** and select **Cleanup**.
## Filter ratio metric numerators
Ratio metrics now support filters that apply only to the numerator. This gives you more control over which events contribute to the numerator without changing the denominator.
## Configure multi-variant experiments
Experiments with more than two variants now have simpler weight selection. The transition from an A/B test to a rollout also preserves multi-variant configurations more smoothly.
## Additional improvements
* Confidence has an updated look and feel, which is rolling out gradually.
* Cloudflare telemetry is more reliable.
## Receive activity notifications through webhooks
[Webhooks](/docs/notifications/webhook-configuration) can send Confidence activity notifications to your HTTPS endpoint. Use them with external systems, monitoring tools, or custom notification workflows.
## Use HyperLogLog metrics
Confidence now supports HyperLogLog metrics for approximate distinct counts over large datasets.
## Follow resource activity
The home page now includes an [activity feed](/docs/notifications/activity-feeds) for important events from resources you follow. You automatically follow resources that you create.
## Additional improvements
* Result pages load faster.
* Selecting environments in flag rules is easier.
## Create richer reports
The [report editor](/docs/experiments/reports) has been redesigned. It now includes templates, an AI writing assistant, more formatting options, image uploads, and metric tables. Open it from the **Reports** tab of any live or completed experiment.
## Target string prefixes and suffixes
New **starts with** and **ends with** criteria support rules such as targeting users whose email address ends with a specific domain. This capability is available as a beta.
## Coordinate experiments with exclusivity guidance
Exclusivity groups can now be marked as required or suggested for a surface. These settings make the intended coordination between experiments clearer.
## Estimate experiment duration
The updated [sample size calculator](/docs/experiments/sample-size-calculator) is now available to everyone. It estimates how many days an experiment design requires and provides a visual breakdown of its sample size requirements.
## Additional improvements
* Ratio metrics support numerator-only filters.
* Configure the initial reach of a rollout from the sidebar.
* Launch and end notifications in Slack include the hypothesis, variants, targeting, results, surfaces, and owner.
* Entity pages support editing more properties, including the display name.
## Review experiments with AI
The [AI review agent](/docs/how-to-guides/create-review-agent) is now available to everyone. It helps teams review their experiments autonomously against your organization's guidelines.
## Inspect flags through MCP
The Flag MCP server has three new capabilities:
* Pass `summary=false` to `getFlag` to retrieve detailed rule configurations and timestamps.
* Use `getFlagHistory` to see who changed a flag, which fields changed, and when. You can filter the history by date range.
* Use `getIdentityInfo` to resolve identity references to readable names when reviewing flag history.
## Forecast experiment duration
The sample size calculator beta now estimates expected traffic, shows how expected and required sample sizes change over time, and predicts how many days an experiment needs to run.
## Access experiments through MCP
The new Experiments MCP server is available as a beta. AI agents can list experiments, inspect individual experiments, read results, and retrieve related metric and fact table definitions.
## Organize resources with labels
[Labels](/docs/labels/introduction) are now available for flags, metrics, A/B tests, and rollouts. Create your own labels to categorize resources and make them easier to find.
## Resolve flags locally
Production-ready [local OpenFeature providers](/docs/flags/local-resolver) are now available for Go, JavaScript, Java, and Rust. They provide near-zero-latency flag resolution and greater resilience.
## Additional improvements
* Select all metrics returned by a search in the metric picker.
* Boolean flags automatically populate treatments when you create an A/B test.
* Large exploratory analyses load faster.
* The Go OpenFeature provider supports native Go structs for stronger type safety and less parsing.
* The JavaScript OpenFeature provider includes React and Next.js utilities for server-side and client-side feature flags.
## Choosing proxy metrics
Our new article, [When proxy metrics shape product development](https://confidence.spotify.com/blog/proxy-metrics), explores the risks of optimizing for a metric that only approximates the outcome you care about.
## Early previews
* An agentic review assistant can check whether rollouts and A/B tests follow your organization's guidelines.
* Metric labels let you annotate and organize metrics with your own labels.
These capabilities are available as opt-in previews.
## Additional improvements
* Minimum detectable effect (MDE) is now optional for success metrics.
* The hypothesis is now in the main experiment design area instead of the sidebar.
## Configure environments
Environments separate flag behavior across deployment stages such as development, staging, and production. Use the same client in several environments while assigning different credentials to each one.
Flag rules can now apply only in selected environments. To get started, configure your environments, assign client credentials, and add environments to your flag rules.
## Get targeting value suggestions
Confidence can suggest recently used values when you configure targeting. Mark non-personally identifiable information (non-PII), such as an app version, in your context schema override to enable suggestions for that field.
## Additional improvements
* React to comments and review responses with emoji.
* View rollout progress in the overview table.
* Set and format hypotheses with Markdown more easily.
* Flag filter conditions have a clearer visualization.
# API Fundamentals
Source: https://confidence-auth-testing.mintlify.io/docs/api/api-basics
Learn the fundamentals of working with Confidence APIs including authentication, pagination, and resource names.
The Confidence API uses the REST structure. Confidence supports authentication via
access tokens and OAuth 2.0.
You make requests via HTTP endpoints with clear functions and appropriate response codes.
The API follows the [Google API guidelines](https://google.aip.dev/general) as much as possible.
## Authentication
Before you can access an endpoint within Confidence, you need to have
valid authentication. Two kinds of APIs exist in Confidence: the management API that you use to configure
the different entities in the platform (flags, experiments, fact tables etc), and the resolve/events API that you
use in the Client SDKs to resolve flag values and emit events.
Because these two kinds have different characteristics, they use different methods for authentication. For the resolve
API, you create a "Client" which has an `API_TOKEN` that you use to authenticate. This API token is long-lived so you
can, for example, bundle it inside your mobile app to resolve flags. A client can have multiple API tokens attached to it, to allow
for rotating tokens as needed. You create clients in the **Admin** panel in [Confidence](https://app.confidence.spotify.com).
The management API uses a [Client Credentials OAuth flow](https://auth0.com/docs/get-started/authentication-and-authorization-flow/client-credentials-flow),
where you supply a client ID and a client secret and in exchange get an access token that is valid for 24 hours. The token is then included in
all requests to the API in an HTTP header. You create API clients in the **Admin** panel in [Confidence](https://app.confidence.spotify.com).
### Request a Token for an API Client
To request an access token, make a POST request to the `/v1/oauth/token` endpoint with your client credentials.
Try the interactive API playground to request an access token.
## Pagination
List resources allow for [pagination](https://google.aip.dev/158) by allowing a `page_token` and a `page_size` in the requests.
If there are more results, a list response has a `next_page_token` field.
Use this token in the next request to ask for the next page of results.
## Resource Names
The API uses [resource names](https://google.aip.dev/122) to uniquely identify each entity that you can use to reference that resource. A resource
name consists of a type and an identifier, for example `flags/my-flag`. A resource name can be hierarchical if the parent owns a child resource.
The resource name can then have multiple levels, for example `flags/my-flag/variants/control`.
## Response Status Codes
The Confidence API uses standard HTTP status codes to signal the status of a request to a consumer, for example `200 (OK)`, `401 (Unauthorized)`, `404 (Not found)`.
## Errors
The Confidence API uses standard HTTP status codes to signal error conditions. Is some cases the response also includes a JSON payload with a more detailed
error message:
```json theme={null}
{
"code": 7,
"message": "Permission denied, missing permission list:metrics",
"details": []
}
```
## Update Masks
When updating a resource, the update request can take an optional field mask, which specifies which fields on the resource the request should update.
If the field mask is not included, the request updates the whole resource.
More details about field masks and how to specify them are available in the [Google API guidelines](https://google.aip.dev/161).
# Connectors
Source: https://confidence-auth-testing.mintlify.io/docs/api/connectors
Confidence Event connectors is a managed service for exporting different types of event data from Confidence.
Read more about how to get started with [exporting data](/docs/api/how-to-guides/connectors/export-data) from Confidence.
As for everything else in the Confidence platform you pay as you go and only pay for what you use of Confidence Events.
## Use Confidence Events
Learn how to export event from Confidence.
Learn how to export assignment events from Confidence.
Learn how to export different kinds of internal events from Confidence.
# Experiments
Source: https://confidence-auth-testing.mintlify.io/docs/api/experiments
Confidence Experiments is a fully managed service for A/B testing and experimentation.
Confidence Experiments is a fully managed service for A/B testing and experimentation. With Confidence Experiments you can run controlled experiments to measure the impact of changes in your application, make data-driven decisions, and continuously optimize your product.
Experiments allow you to test hypotheses, analyze results with statistical rigor, and make informed decisions about which features and changes drive the most value for your users.
## Use Confidence Experiments
View the complete API reference for managing experiments.
The Experiments API provides endpoints for creating and managing A/B tests and rollouts. See the API Reference section below for detailed endpoint documentation.
# Flag Concepts
Source: https://confidence-auth-testing.mintlify.io/docs/api/flags/concepts
Understand the key concepts in Confidence Flags.
This section explains the core concepts in Confidence Flags. These primitives work together to enable Confidence to flexibly support a broad range of applications where you need to remotely change an experience or behavior of your app.
## Flags
A `Flag` is a mechanism to remotely configure different parts of an application. The configuration can either be deterministic such that, for example, all employees see a blue button and everyone else sees a red button. It can also be random such as if users are randomly assigned a blue or red button.
## Variants
A `Variant` is a named configuration. For example, you could have a variant called `big red button` that configures a button to be big and red. You can have as many variants as you want, but a user (or service) can only resolve one variant at a time for a given flag.
```js Example variant value theme={null}
{
"color": "red",
"size": 12
}
```
The value of a variant is a JSON object that conforms to a `Schema`. The schema describes the structure of the value. It ensures that the value is valid and that it conforms to the expectations of the application.
```js Example schema theme={null}
{
"schema": {
"color": {
"stringSchema": {}
},
"size": {
"intSchema": {}
}
}
}
```
## Segments
A `Segment` is a mechanism to define a subpopulation of users. For example, you could have a segment that represents 10% of users from Sweden. A set of `Criteria` determines what users are in a segment. A `Criteria` is a mechanism to define a property of a user. For example, you could have a criteria that says that a user is an employee if they have the `employee` attribute set to `true`.
Segments have two key parts:
* **Targeting**: A set of criteria that filter users based on attributes
* **Allocation**: What percentage (0% to 100%) of the targeted users should be in the segment
Segments can be mutually exclusive or overlapping. For example, you could have a segment that represents users from Sweden, and another segment that represents users from the United States. These two segments are mutually exclusive, meaning that no user can be in both segments. You could also have a segment that represents 10% of users from Sweden and another segment that represents 10% of users from Sweden that are also employees. These two segments are overlapping, meaning that some users can be in both segments. With coordination and exclusivity tags it's possible to create exclusive segments that would naturally be overlapping.
## Targeting Criteria
Targeting criteria allow you to filter users based on attributes in the evaluation context. You define named criteria and compose them into expressions using logical operators.
Targeting has two parts:
* **Criteria**: Named conditions that define individual filters
* **Expression**: Logical composition of criteria using operators
### Attribute Criteria
Attribute criteria match a value in the evaluation context against a specified value using an operator. You can use different value types (Boolean, Number, String, Timestamp, Version) and matching operators (equality, set, range).
You can reference nested fields using dot notation (for example, `device.model` to access `{device: {model: "iPhone14"}}`).
### Segment Criteria
Segment criteria check if a user is part of another segment, allowing for composing sophisticated targeting logic by combining multiple segments.
## Randomization and Allocation
Confidence randomizes users based on a field in the evaluation context (defaults to `targeting_key`). The randomization is:
* **Consistent**: The same user always gets the same allocation
* **Distributed**: On average, Confidence allocates the specified percentage of users to the segment
If the randomization field is missing or `null` in the evaluation context, the segment doesn't match.
## Coordination
Coordination makes segments mutually exclusive, ensuring users can only be in one experiment at a time. This uses two sets of tags on each segment:
* **Exclusivity tags**: Incoming tags that identify what this segment is (for example, `ranking-experiment`, `checkout-flow`)
* **Exclusive to tags**: Outgoing tags that specify which other segments to exclude from
Two segments are mutually exclusive if there's an overlap between one segment's exclusivity tags and another segment's exclusive-to tags.
For example:
* Segment A has `exclusivityTags: ["ranking"]` and `exclusiveTo: ["ranking"]`
* Segment B has `exclusivityTags: ["ranking"]` and `exclusiveTo: ["ranking"]`
* These segments are exclusive because A's exclusivity tags overlap with B's exclusive-to tags (and vice versa)
When allocating a segment with coordination tags, Confidence verifies there's enough available space. The allocation fails if:
* Too many coordinating segments already exist
* The combined allocations exceed 100%
* There's insufficient space for the requested proportion
If allocation fails, you must reduce the proportion, archive other segments, or adjust coordination tags.
## Rules
A `Rule` combines a segment (who is eligible) with variant assignments (what they receive). Rules are the mechanism that determines which variant a user gets.
Rules evaluate in priority order, with lower priority numbers evaluated first. The first rule that matches determines the variant. Newly created rules start disabled—you must explicitly enable them.
## Evaluation Context
The evaluation context is how clients give contextual data for rule evaluation. It's a schema-less key-value map (JSON object) containing any data needed for targeting, such as:
* User IDs or identifiers
* User attributes (country, device, browser, etc.)
* Session or environment information
Example:
```json theme={null}
{
"user_id": "rosling",
"country": "SE",
"device": {
"vendor": "apple",
"os": "ios"
}
}
```
## Variant Assignment
When a rule matches, Confidence assigns a variant using bucket-based randomization. This ensures consistent, stable assignments while distributing users across variants.
Here's how it works:
1. **Hash computation**: Computes a hash of the field value specified by `targetingKeySelector` (defaults to `targeting_key`) from the evaluation context
2. **Bucket calculation**: Normalizes the hash to a bucket number: `hash % bucketCount`
3. **Variant selection**: Assigns the variant whose bucket range includes the calculated bucket number
The `targetingKeySelector` specifies which field from the evaluation context to use for randomization. Common patterns include using `targeting_key` for general user identification, `user_id` for user-level randomization, `device_id` for device-level randomization, or `session_id` for session-level randomization.
If the specified field is missing or `null`, the rule doesn't match. The empty string `""` is a valid value.
## Resolve a Flag
Resolving a flag is the process of determining which variant a user should see for a given flag. The application passes in an evaluation context that has information about the user and other information that you can use to decide if a user should be eligible. If no rule matches the user, the application normally falls back to a default variant.
The rules evaluate in order. The rule used is the first rule that matches the user. This means that you can have a rule that says that all employees should see the `big red button` variant, and a rule that says that 50% of users should see the `big red button` variant and 50% should see the `small blue button` variant. In this case, all employees would see the `big red button` variant, and the other 50% would be randomly assigned a variant.
A rule can "fall through." This means that the rule matches, but instead of assigning a variant to the user, the user is assigned a variant from one of the following rules.
## Apply a Flag
When you resolve a flag and use the value in your application, the flag is **applied**. You must report this back to Confidence using the apply operation. Confidence then writes "flag applied" events to your data warehouse via a configured connector.
Apply events matter because they:
* **Track exposure**: Compute who was exposed to which variants and how many times
* **Enable analysis**: Enable accurate A/B test and experiment analysis
* **Measure adoption**: Track how often features are actually used
* **Aid debugging**: Help debug flag behavior and targeting issues
Apply a flag when:
* The user sees a UI element controlled by the flag
* The application executes code controlled by the flag
* A backend service makes a decision based on the flag value
Do NOT apply a flag when:
* You resolved it but didn't use the value
* You cached the value but haven't displayed it yet
* A conditional check prevented you from using the flag
## Flag Clients
A flag client represents a single application that uses flags. The flag client authenticates with Confidence using a shared secret. You have to authenticate flags with a flag client before they are available for use by that client.
## Archive
When flags and segments are no longer needed, archive them instead of deleting. Archiving preserves resources for historical reference and analysis while removing them from active use.
Confidence uses archiving instead of deletion to:
* Keep historical data for analysis
* Keep references in experiment results
* Prevent breaking changes to existing integrations
* Keep audit trails intact
* Allow potential restoration (contact support)
When you archive a flag, resolve requests return the user-specified default value and you can't use the flag in new experiments. When you archive a segment, it enters the `ARCHIVED` state and you can't use it in new rules, but existing rules using the segment continue to work.
# Flags Reference
Source: https://confidence-auth-testing.mintlify.io/docs/api/flags/flags-reference
Technical reference for Confidence Flags.
This section provides technical specifications and reference information for Confidence Flags.
For conceptual explanations of how flags work, see [Flag Concepts](./concepts).
## Rules
### Key Characteristics
* **Priority-based evaluation**: Rules evaluate in priority order, with lower priority numbers evaluated first
* **First match wins**: The first rule that matches determines the variant
* **Required state**: Newly created rules start disabled—you must explicitly enable them
* **Segment dependency**: Rules require an allocated segment to function
### Rule Composition
* **Segment**: Defines who is eligible for the rule
* **Assignment specification**: Defines variant assignments using bucket-based randomization
* **Targeting key selector**: Specifies which field from evaluation context to use for randomization (defaults to `targeting_key`)
* **Priority**: Determines evaluation order
* **Enabled state**: Controls whether the rule is active
### Best Practices
* Create rules in a disabled state, test thoroughly, then enable
* Use meaningful segment names that clearly describe the target audience
* Put more specific rules before general rules (using priority)
* Use bucket count 100 for percentage-based splits (or 1000 for per-mille precision)
* If the targeting key field is missing or `null` in the evaluation context, the rule doesn't match (empty string `""` is valid)
## Variant Assignment
### Bucket Ranges
* `bucketCount` defines how many buckets to divide users into
* Each assignment specifies one or more bucket ranges (for example, `{lower: 0, upper: 50}`)
* Ranges are inclusive at lower bound, exclusive at upper bound
* Common pattern: Use `bucketCount: 100` for percentage-based splits (50/50 = buckets 0-50, 50-100)
### Assignment Types
1. **Variant assignment**: Assigns a bucket range to a specific variant (most common use case)
* Use for A/B tests, multivariate tests, rollouts
2. **Fall-through assignment**: Passes assignment to the next matching rule, but logs an assignment event.
Suitable for logging which users matched a segment without changing their experience
* Creates complex rule chains where different segments handle different aspects
3. **Client default assignment**: Returns the default values specified by each client.
Allows clients to define their own fallback behavior.
Suitable for gradual rollouts or feature toggles
### Targeting Key Selector
The `targetingKeySelector` specifies which field from the evaluation context to use for randomization. Common patterns:
* `targeting_key` (default): General user identifier
* `user_id`: User-level randomization
* `device_id`: Device-level randomization (same user, different devices get different variants)
* `session_id`: Session-level randomization (new variant each session)
## Archive Flags
### Archive Behavior
**When you archive a flag:**
* Flag still exists and you can reference it
* Resolve requests return the user-specified default value
* Flag appears as archived in the UI
* Historical data remains available
* You can't use it in new experiments
* SDKs return default values for all users
**When you archive a segment:**
* Segment enters the `ARCHIVED` state
* You can't use it in new rules
* Existing rules using the segment continue to work
* Confidence preserves historical allocation data
* Frees up coordination space for other segments
### When to Archive
**Archive flags when:**
* Feature has been fully rolled out to all users
* Experiment has concluded and you chose a winner
* Feature is being permanently removed
* Flag is deprecated and no longer needed
**Archive segments when:**
* Experiment using the segment has ended
* Targeting criteria is no longer relevant
* Consolidating segments
* You created the segment for testing only
### Before You Archive
1. Check that no active experiments depend on the resource
2. Consider removing archived segments from rules (optional)
3. Document why you're archiving the resource
4. Notify team members about the archival
### After You Archive
1. Verify the resource shows as archived
2. Check that the change doesn't affect active experiments
3. Update documentation to reflect the change
4. Remove feature flag code from application (if applicable)
5. Archive related resources (segments for a flag, flags using a segment)
### Find Archival Candidates
List flags and segments, then review for:
* Flags/segments with no recent activity
* Resources from completed experiments
* Flags with all rules disabled
* Segments in ALLOCATED state with no active rules
* Test resources no longer in use
### Clean Up Application Code
After archiving a flag, remove the flag code from your application and use the winning variant's behavior directly.
### Best Practices
* Archive promptly—don't let unused resources accumulate
* Document reasons and timing of archival
* Clean up in stages: archive first, then remove code later
* Review regularly for archival candidates
* Coordinate with team when archiving shared resources
# Resolution Reference
Source: https://confidence-auth-testing.mintlify.io/docs/api/flags/resolution-reference
Technical reference for flag resolution and application.
This section provides technical specifications and reference information for flag resolution and application.
For conceptual explanations of flag resolution and application, see [Resolve a Flag](./concepts#resolve-a-flag) and [Apply a Flag](./concepts#apply-a-flag) in the Flag Concepts page.
## Apply a Flag
### Timestamps
Each apply requires two timestamps using the client's local clock:
* `appliedTime`: When the flag value was actually used in the application
* `sentTime`: When the client reports the apply to Confidence
These two timestamps allow accurate tracking without requiring synchronized clocks. The difference helps account for:
* Network latency
* Batching delays
* Offline operation
### Batch Requests
It's recommended to batch apply operations:
* Reduces network overhead
* Saves battery life on mobile devices
* More efficient for the service
Wait for opportune moments (like when doing other network requests) to report applies.
### Mobile and Offline Scenarios
For mobile apps and offline scenarios:
1. Queue applies locally while offline
2. Send all queued applies when network is available
3. Piggyback applies on other API calls
4. Preserve the original `appliedTime` even if sent later
### Handle Errors
Apply operations should be fire-and-forget:
* Log errors for debugging but don't block application flow
* Don't retry immediately to avoid overwhelming the service
* Consider queuing for later retry
* Applying is not critical to UX—the flag has already been resolved
### Monitoring
Track apply metrics to ensure data quality:
* **Apply rate**: How often flags are being applied
* **Apply delay**: Time between `appliedTime` and `sentTime`
* **Apply failures**: How many applies are failing
* **Missing applies**: Flags that were resolved but never applied
### Best Practices
* Apply only when the flag value is actually used
* Batch applies together when possible
* Use accurate timestamps (set `appliedTime` when flag is actually used)
* Handle failures gracefully without breaking application flow
* Queue applies when offline and send when connected
* Monitor apply rates to ensure experiment data is complete
# Segments Reference
Source: https://confidence-auth-testing.mintlify.io/docs/api/flags/segments-reference
Technical reference for Confidence Segments.
This section provides technical specifications and reference information for Confidence Segments.
For conceptual explanations of segments, see [Segments](./concepts#segments) in the Flag Concepts page.
## Segment States
Segments have three lifecycle states:
| State | Description |
| :-------------- | :------------------------------------------------------ |
| **UNALLOCATED** | Initial state after creation |
| **ALLOCATED** | Active and ready to use in flag rules |
| **ARCHIVED** | No longer in use but preserved for historical reference |
You must allocate a segment before using it in a flag rule.
## Targeting Criteria
### Expression Operators
Supported expression operators:
| Operator | Description |
| :------- | :--------------------------------- |
| `ref` | Reference a named criterion |
| `not` | Logical NOT of a nested expression |
| `and` | Logical AND between expressions |
| `or` | Logical OR between expressions |
### Attribute Value Types
Available value types:
| Type | Description | Example |
| :-------- | :------------------------ | :----------------------- |
| Boolean | Boolean value | `true`, `false` |
| Number | Integer or floating point | `42`, `3.14` |
| String | String value | `"HELLO"` |
| Timestamp | ISO 8601 timestamp | `"2023-01-01T00:23:54Z"` |
| Version | Semantic version | `"2.1.3"` |
### Match Operators
* **Equality (`eqRule`)**: Match exact value
* **Set (`setRule`)**: Match any value from a set
* **Range (`rangeRule`)**: Match values within a range (supports inclusive/exclusive bounds and open-ended ranges)
## Coordination
### Common Coordination Patterns
1. **Mutual exclusion**: All experiments in a feature area exclude each other using the same tag
Use `exclusivityTags: ["homepage"]` and `exclusiveTo: ["homepage"]`
2. **Hierarchical coordination**: Specific experiments exclude from broader categories
* Specific: `exclusivityTags: ["ranking-v2"]` and `exclusiveTo: ["ranking-v2", "all-experiments"]`
* Broad: `exclusivityTags: ["all-experiments"]` and `exclusiveTo: ["all-experiments"]`
3. **Cross-feature coordination**: Related features that shouldn't run simultaneously
* Search: `exclusivityTags: ["search"]` and `exclusiveTo: ["search", "ui-changes"]`
* UI: `exclusivityTags: ["ui-changes"]` and `exclusiveTo: ["search", "ui-changes"]`
### Best Practices
* Use descriptive tag names that clearly indicate the feature or experiment type
* Plan coordination strategy before creating segments
* Monitor allocation space within each coordination group
* Archive completed experiments to free up space
* Document which tags represent which feature areas
# Event Connectors
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/connectors/events
Export event data ingested through the Confidence event sender SDKs.
Event connectors export event data ingested through the Confidence event sender SDKs.
Export events to BigQuery. The connector writes each event type to a separate table in the configured dataset with an optional prefix.
### Required GCP Roles
* BigQuery data owner for the destination dataset.
### Configuration
* **Project** - The GCP project the destination table exists in
* **Service account** - A GCP service account that has write access to the destination table. Configure the service account so that the Confidence service account can impersonate it.
* **Dataset** - The dataset in which to create the destination tables
* **Table prefix** - An optional prefix to use for the tables created by this connector
Export events to tables in Redshift, by first writing the data as Parquet files to an S3 bucket, and then importing these files into Redshift tables, one per event type.
### Required AWS Permissions/Policies for Role
* `s3:GetObject`
* `s3:GetObjectAcl`
* `s3:PutObject`
* `s3:PutObjectAcl`
* `AmazonRedshiftDataFullAccess` policy
* `AmazonRedshiftAllCommandsFullAccess` policy
The role also needs to have permissions to create tables and insert rows into those tables in the configured database.
### Configuration
* **Table Prefix** - A prefix to use for all tables created by the connector. The connector loads each event type into its own table.
* **Cluster** - The name of the Redshift cluster to use.
* **Database** - The name of the Redshift database to use.
* **Schema** - The name of the Redshift schema in which the connector creates the table.
* **Redshift region** - The AWS region of the cluster. Because of AWS limitations, the region of the cluster needs to match the region of the S3 bucket.
* **Redshift Role ARN** - The role the Redshift `COPY` jobs use. This role needs to have permission to create tables and copy data into tables in the configured schema, and load files from the S3 bucket.
* **Bucket** - The S3 bucket to write the Parquet files to.
* **Bucket Role ARN** - The role Confidence has when writing files to the S3 bucket.
* **Bucket Region** - The AWS region of the bucket. The bucket needs to be in the same region as the Redshift cluster.
* **Batch settings** - These settings control the size and max age of the Parquet files written to S3.
Export events to Databricks tables, by first writing the data as Parquet files to an S3 bucket, and then importing these files into Databricks tables. Events are written to separate tables per event type in the configured schema/catalog.
### Required AWS Permissions for Role
* `s3:GetObject`
* `s3:GetObjectAcl`
* `s3:PutObject`
* `s3:PutObjectAcl`
### Configuration
* **Databricks schema** - The name of the Databricks schema/catalog in which to create tables.
* **Databricks host** - The hostname for the Databricks instance, for example `xx.x.gcp.databricks.com`.
* **Databricks HTTP path** - The HTTP path to use for the Databricks JDBC connection, available in the [connection details](https://docs.gcp.databricks.com/integrations/jdbc-odbc-bi.html#get-connection-details-for-a-cluster) for the cluster.
* **Databricks Access Token** - An access token that has write access to the configured table.
* **Role ARN** - The ARN for the AWS Role that has read and write access to the S3 bucket. The role needs to have a trust relationship configured so that the Confidence service account can assume this role.
* **Bucket** - The S3 bucket to write the Parquet files to.
* **Batch settings** - These settings control the size of the Parquet files written to S3. If not set or 0, Confidence uses the default settings.
# Export Data From Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/connectors/export-data
Learn how to export data from Confidence using connectors.
To export data Confidence has the concept of *"Connectors."* A connector takes events generated inside the Confidence platform (for example events specifying who got assigned to what experience) and forwards them to a destination.
Configure connectors in the **Admin** panel in [Confidence](https://app.confidence.spotify.com).
Three types of connectors are available. Each connector forwards different kinds of data:
* **Flag Applied Connectors** - These connectors export the assignment data.
* **Event Connectors** - These connectors export event data ingested through the Confidence event sender SDKs.
* **Platform Connectors** - These connectors forwards internal events to the customer's system. The events could for example be `MetricCalculationCompleted` or `WorkflowInstanceCreated`.
The connector destination can be either a warehouse such as BigQuery or Databricks, or a Pub/Sub topic or a Kinesis Stream that exists in your environment. When the data is in your warehouse, Confidence can query it by defining assignment and fact tables on top of the data.
## Authentication Setup
Confidence tries to avoid storing credentials for the connector destinations whenever possible. Instead, every Confidence account gets a unique GCP service account that's used to authenticate when writing to a destination. The procedure to configure the authentication differs between GCP and AWS. The next sections describe the authentication steps.
* Create a service account with the required permissions
* On the Permissions tab for the service account you just created in the GCP console, grant access to the principal `account-@spotify-confidence.iam.gserviceaccount.com` to impersonate the account by adding the "Workload Identify User" role to it.
For AWS Confidence uses [AssumeRoleWithWebIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html) to securely authenticate without having to store any credentials.
* Create a role with the required permissions in the AWS IAM Console. Specify the required permissions for each connector destination type.
* Edit the trust relationship for the role, and allow the GCP Service Account to assume the role by using a definition that looks like this:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "accounts.google.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"accounts.google.com:sub": ""
}
}
}
]
}
```
# Flag Applied Connectors
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/connectors/flag-applied
Export assignment data from Confidence.
Flag applied connectors export assignment data.
Export assignment data to a table in BigQuery.
### Required GCP Roles
* BigQuery data owner for the destination table.
### Configuration
* **Project** - The GCP project the destination table exists in
* **Service account** - A GCP service account that has write access to the destination table, and that you've configured so that the Confidence service account can impersonate this account.
* **Dataset** - The dataset in which to create the destination table
* **Table** - The table to write the data to. The connector automatically creates the table. If the table already exists, the connector verifies that the schema looks as expected, otherwise it attempts to create the required columns.
Export assignment data to a table in Redshift, by first writing the data as Parquet files to a S3 bucket, and then import these files into the configured table.
### Required AWS Permissions/Policies for Role
* `s3:GetObject`
* `s3:GetObjectAcl`
* `s3:PutObject`
* `s3:PutObjectAcl`
* `AmazonRedshiftDataFullAccess` policy
* `AmazonRedshiftAllCommandsFullAccess` policy
The role also needs to have permissions to create tables and insert rows into those tables in the configured database.
### Configuration
* **Table** - The name of the Redshift table to use for writing assignment data. The connector creates the table automatically when the first import is done.
* **Cluster** - The name of the Redshift cluster to use.
* **Database** - The name of the Redshift database to use.
* **Schema** - The name of the Redshift schema where the connector creates the preceding table.
* **Redshift region** - The AWS region of the cluster. Because of AWS limitations, the region of the cluster needs to match the region of the S3 bucket.
* **Redshift Role ARN** - The role to use for the Redshift `COPY` jobs. This role needs to have permission to create tables and copy data into tables in the configured schema, and load files from the S3 bucket.
* **Bucket** - The S3 bucket to write the Parquet files to.
* **Bucket Role ARN** - The role Confidence assumes when writing files to the S3 bucket.
* **Bucket Region** - The AWS region of the bucket. The bucket needs to be in the same region as the Redshift cluster.
* **Batch settings** - These settings control the size and max age of the Parquet files written to S3.
Export assignment data to a table in Databricks, by first writing the data as Parquet files to a S3 bucket, and then import these files into the specified table.
### Required AWS Permissions for Role
* `s3:GetObject`
* `s3:GetObjectAcl`
* `s3:PutObject`
* `s3:PutObjectAcl`
### Configuration
* **Table** - The name of the Databricks table to use for writing assignment data. The connector creates the table automatically when the first import is done.
* **Schema** - The name of the Databricks Schema/Catalogue where the connector creates the preceding table.
* **Databricks HTTP path** - The HTTP path to use for the Databricks JDBC connection, available in the [connection details](https://docs.gcp.databricks.com/integrations/jdbc-odbc-bi.html#get-connection-details-for-a-cluster) for the cluster.
* **Databricks Access Token** - An access token that has write access to the configured table.
* **Role ARN** - The ARN for the AWS Role that has read/write access to the S3 bucket. You need to configure the role to have a trust relationship so that the Confidence service account can to assume this role.
* **Bucket** - The S3 bucket to write the parquet files to.
* **Databricks host** - The hostname for the Databricks instance, for example `xx.x.gcp.databricks.com`.
* **Batch settings** - These settings control the size of the Parquet files written to S3.
Forward assignment data to a Pub/Sub topic.
### Required GCP Roles
* Pub/Sub Editor
### Configuration
* **Event Type** - The Event type to export to the stream
* **Project** - The GCP project that the topic exists in.
* **Service account** - A GCP service account that has publish permissions to the configured topic and that you've configured so that the Confidence service account can impersonate this account.
* **Topic** - The name of the topic
* **Output format** - if the connector should write events in JSON or Protobuf binary format. Regardless of the format chosen, it wraps the event in a [CloudEvents](https://cloudevents.io/) envelope.
Forward assignment data to a Kinesis stream.
### Required AWS Permissions
* `kinesis:PutRecord`
* `kinesis:PutRecords`
* `kinesis:PutRecordBatch`
* `kinesis:DescribeStream`
### Configuration
* **Event Type** - The Event type to export to the stream
* **Role ARN** - The ARN for the AWS role that has read and write access to the Kinesis stream.
* **Region** - The AWS region the Kinesis stream exists in
* **Stream** - The Kinesis stream name.
* **Output format** - if the connector should write events in JSON or Protobuf binary format. Regardless of the format chosen, the connector wraps the event in a [CloudEvents](https://cloudevents.io/) envelope.
# Platform Connectors
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/connectors/platform
Forward internal events from Confidence to your system.
Platform connectors forward internal events to your system. The events could, for example, be `MetricCalculationCompleted` or `WorkflowInstanceCreated`.
Export platform events of a configurable type to a Google Pub/Sub topic.
### Required GCP Roles
* Pub/Sub Editor
### Configuration
* **Event Type** - The Event type to export to the stream
* **Project** - The GCP project that the topic exists in.
* **Service account** - A GCP service account that has publish permissions to the configured topic. Configure the account so that the Confidence service account can impersonate it.
* **Topic** - The name of the topic
* **Output format** - If the connector should write the events in JSON or Protobuf binary format. Regardless of the format chosen, the connector wraps the event in a [CloudEvents](https://cloudevents.io/) envelope.
Export platform events of a configurable type to a Kinesis stream.
### Required AWS Permissions
* `kinesis:PutRecord`
* `kinesis:PutRecords`
* `kinesis:PutRecordBatch`
* `kinesis:DescribeStream`
### Configuration
* **Event Type** - The Event type to export to the stream
* **Role ARN** - The ARN for the AWS Role that has read and write access to the Kinesis stream.
* **Region** - The AWS region the Kinesis stream exists in
* **Stream** - The Kinesis stream name.
* **Output format** - If the connector should write the events in JSON or Protobuf binary format. Regardless of the format chosen, the connector wraps the event in a [CloudEvents](https://cloudevents.io/) envelope.
# Manage A/B Tests
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/experiments/manage-abtests
Learn how to create and manage A/B tests using the Confidence API.
Use this API to create and manage A/B tests. For an introduction to A/B testing concepts, see the [A/B test workflows documentation](/docs/experiments/workflows/abtests).
All requests require a bearer token in the `Authorization` header. See [Get Started with Confidence APIs](/docs/api/quickstart) for how to obtain one using your client ID and secret.
## Get an A/B Test
Retrieve a specific A/B test by its name to view its configuration and current state.
```bash theme={null}
curl -X GET "https://experiments.confidence.dev/v1/workflows/abtest/instances/dwimxjvyjsfno42agkl8" \
-H "Authorization: Bearer $TOKEN"
```
Response:
```json theme={null}
{
"name": "workflows/abtest/instances/dwimxjvyjsfno42agkl8",
"displayName": "Checkout Flow Experiment",
"owner": "identities/cc9yglwi8kpgh0glddnkz",
"state": "live"
}
```
## List A/B Tests
List all A/B tests in your account. You can filter by state or any other criteria in the response. Use `nextPageToken` for getting the next page of results.
```bash theme={null}
curl -X GET "https://experiments.confidence.dev/v1/workflows/abtest/instances?pageSize=50&filter=state:live" \
-H "Authorization: Bearer $TOKEN"
```
Response:
```json theme={null}
{
"abtests": [
{
"name": "workflows/abtest/instances/dwimxjvyjsfno42agkl8",
"displayName": "Checkout Flow Experiment",
"state": "live"
},
{
"name": "workflows/abtest/instances/eioavdylbf9idlembel5",
"displayName": "Pricing Page Test",
"state": "live"
}
],
"nextPageToken": ""
}
```
## Create an A/B Test
Create a new A/B test with treatments and metrics configuration.
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/abtest" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Checkout Flow Experiment",
"flags": {
"targetingKeySelector": "targeting_key"
},
"stats": {
"testHorizonStrategy": "SEQUENTIAL"
},
"abtest": {
"treatments": [
{
"variant": "flags/checkout-flag/variants/control",
"weight": "5000"
},
{
"variant": "flags/checkout-flag/variants/new-checkout",
"weight": "5000"
}
]
},
"metrics": {
"assignmentTable": "assignmentTables/my-assignment-table",
"entity": "entities/user",
"bucket": "DAYS",
"metrics": [
{
"metric": "metrics/conversion-rate",
"metricRole": {
"metricKind": "SUCCESS",
"minimumDetectableEffect": 0.01
},
"preferredDirection": "INCREASE"
}
]
}
}'
```
Response:
```json theme={null}
{
"name": "workflows/abtest/instances/checkout-experiment",
"displayName": "Checkout Flow Experiment",
"state": "draft",
"createTime": "2025-11-01T10:00:00Z",
"updateTime": "2025-11-01T10:00:00Z"
}
```
## Create an A/B Test with Exposure Filters
You can create an A/B test with exposure filters to segment your analysis by different user behaviors or contexts.
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/abtest" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "A/B test with exposure filter",
"flags": {
"targetingKeySelector": "targeting_key"
},
"stats": {
"testHorizonStrategy": "SEQUENTIAL"
},
"abtest": {
"treatments": [
{
"variant": "flags/page-feature-flag/variants/control",
"weight": "5000"
},
{
"variant": "flags/page-feature-flag/variants/variant",
"weight": "5000"
}
]
},
"metrics": {
"assignmentTable": "assignmentTables/my-assignment-table",
"entity": "entities/user",
"bucket": "HOURS",
"filters": [
{
"displayName": "home",
"filter": {
"criteria": {
"page-1": {
"attribute": {
"attribute": "page",
"eqRule": {
"value": {
"stringValue": "home"
}
}
}
}
},
"expression": {
"and": {
"operands": [{"ref": "page-1"}]
}
}
},
"factTable": "factTables/my-fact-table"
}
],
"metrics": [
{
"metric": "metrics/conversion-rate",
"metricRole": {
"metricKind": "SUCCESS",
"minimumDetectableEffect": 0.01
},
"preferredDirection": "INCREASE"
}
]
}
}'
```
Response:
```json theme={null}
{
"name": "workflows/abtest/instances/eioavdylbf9idlembel5",
"displayName": "A/B test with exposure filter",
"state": "draft",
"createTime": "2025-11-13T10:00:00Z",
"updateTime": "2025-11-13T10:00:00Z"
}
```
## Set Targeting on an A/B Test
After creating an A/B test, call the `UpdateSegment` action with the desired targeting configuration. Include the `updateMask` field relative to `segment`.
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/abtest/instances/pinxh4k3ekqmr2fltek0:updateSegment" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"parameters": {
"updateMask": "targeting",
"segment": {
"name": "segments/pztjlarivt3tyfie5gsb",
"targeting": {
"criteria": {
"country": {
"attribute": {
"attributeName": "country",
"eqRule": {
"value": {
"stringValue": "SE"
}
}
}
}
},
"expression": {
"and": {
"operands": [{"ref": "country"}]
}
}
}
}
}
}'
```
## Update Allocation on an A/B Test
Call the `UpdateSegment` action with the allocation proportion to update the traffic allocation for an A/B test.
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/abtest/instances/pinxh4k3ekqmr2fltek0:updateSegment" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"parameters": {
"updateMask": "allocation",
"segment": {
"name": "segments/pztjlarivt3tyfie5gsb",
"allocation": {
"proportion": {
"value": "0.5"
}
}
}
}
}'
```
## Action Methods
Action methods return immediately with an empty result and start the action in the background. Use the `GET` endpoint for the A/B test to monitor the progress by inspecting the `pendingTransition` field.
### Launch an A/B Test
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/abtest/instances/dwimxjvyjsfno42agkl8:launch" \
-H "Authorization: Bearer $TOKEN"
```
### End an A/B Test
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/abtest/instances/dwimxjvyjsfno42agkl8:end" \
-H "Authorization: Bearer $TOKEN"
```
### Archive an A/B Test
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/abtest/instances/dwimxjvyjsfno42agkl8:archive" \
-H "Authorization: Bearer $TOKEN"
```
## View Results for an A/B Test
To view statistical results for your A/B test, use a two-step process.
### Step 1: Get the A/B Test
First, get the A/B test to retrieve the `analysisResult` resource name from the `stats` field.
```bash theme={null}
curl -X GET "https://experiments.confidence.dev/v1/workflows/abtest/instances/dwimxjvyjsfno42agkl8" \
-H "Authorization: Bearer $TOKEN"
```
Response:
```json theme={null}
{
"name": "workflows/abtest/instances/dwimxjvyjsfno42agkl8",
"displayName": "Checkout Flow Experiment",
"stats": {
"analysisResult": "workflows/abtest/instances/dwimxjvyjsfno42agkl8/analysisResults/abc123"
}
}
```
If your A/B test uses multiple exposure filters, check the `stats.analysisResults[]` array instead. Each entry has an `analysisResult` resource name and the corresponding `exposureFilterName`.
### Step 2: Get the Analysis Result
Use the `analysisResult` name from Step 1 to fetch the full statistical analysis
from the [Analysis Results API](/api-reference/get-analysisresult).
```bash theme={null}
curl -X GET "https://stats.confidence.dev/v2/workflows/abtest/instances/dwimxjvyjsfno42agkl8/analysisResults/abc123" \
-H "Authorization: Bearer $TOKEN"
```
Response:
```json theme={null}
{
"name": "workflows/abtest/instances/dwimxjvyjsfno42agkl8/analysisResults/abc123",
"exposureFilter": "",
"annotations": [
{
"context": "OVERALL",
"info": ["Analysis completed successfully"]
}
],
"results": [
{
"id": "metrics/conversion-rate",
"statsSettings": {
"method": "METHOD_Z_TEST",
"adjustedAlpha": 0.05
},
"result": {
"status": {
"status": "METRIC_RESULT_STATUS_SIGNIFICANT_POSITIVE",
"metricType": "METRIC_TYPE_SUCCESS"
}
}
}
]
}
```
# Manage Rollouts
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/experiments/manage-rollouts
Learn how to create and manage rollouts using the Confidence API.
Use this API to create and manage Rollouts. For an introduction to rollout concepts, see the [rollout workflows documentation](/docs/experiments/workflows/rollouts).
## Get a Rollout
Retrieve a specific rollout by its name to view its configuration and current state.
```bash theme={null}
curl -X GET "https://experiments.confidence.dev/v1/workflows/rollout/instances/brcdajvw7dfuod7cj9iq" \
-H "Authorization: Bearer $TOKEN"
```
Response:
```json theme={null}
{
"name": "workflows/rollout/instances/brcdajvw7dfuod7cj9iq",
"displayName": "New Search Feature Rollout",
"createTime": "2024-01-15T10:00:00Z",
"updateTime": "2024-01-20T14:30:00Z"
}
```
## List Rollouts
List all rollouts in your account. You can filter by state or any other criteria in the response. Use `nextPageToken` for getting the next page of results.
```bash theme={null}
curl -X GET "https://experiments.confidence.dev/v1/workflows/rollout/instances?pageSize=50&filter=state:live" \
-H "Authorization: Bearer $TOKEN"
```
Response:
```json theme={null}
{
"rollouts": [
{
"name": "workflows/rollout/instances/brcdajvw7dfuod7cj9iq",
"displayName": "New Search Feature Rollout",
"state": "live"
},
{
"name": "workflows/rollout/instances/bnkv2onacdpvxk8cmqbm",
"displayName": "Mobile UI Update",
"state": "live"
}
],
"nextPageToken": ""
}
```
## Create a Rollout
Create a new rollout with initial exposure settings.
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/rollout/instances" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "New Search Feature Rollout",
"flags": {
"targetingKeySelector": "targeting_key"
},
"rollout": {
"flag": "flags/search-flag",
"variant": "flags/search-flag/variants/enabled"
},
"metrics": {
"assignmentTable": "assignmentTables/my-assignment-table",
"entity": "entities/user",
"bucket": "DAYS",
"metrics": [
{
"metric": "metrics/conversion-rate",
"metricRole": {
"metricKind": "SUCCESS",
"minimumDetectableEffect": 0.01
},
"preferredDirection": "INCREASE"
}
]
}
}'
```
Response:
```json theme={null}
{
"name": "rollouts/new-search-feature",
"displayName": "New Search Feature Rollout",
"state": "draft",
"createTime": "2024-01-15T10:00:00Z",
"updateTime": "2024-01-15T10:00:00Z"
}
```
## Set Targeting on a Rollout
After creating a rollout, call the `UpdateSegment` action with the desired targeting configuration. Include the `updateMask` field relative to `segment`.
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/rollout/instances/brcdajvw7dfuod7cj9iq:updateSegment" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"updateMask": "targeting",
"segment": {
"name": "segments/pztjlarivt3tyfie5gsb",
"targeting": {
"criteria": {
"country": {
"attribute": {
"attributeName": "country",
"eqRule": {
"value": {
"stringValue": "SE"
}
}
}
}
},
"expression": {
"and": {
"operands": [{"ref": "country"}]
}
}
}
}
}'
```
## Action Methods
Action methods return immediately with an empty result and start the action in the background. Use the `GET` endpoint for the rollout to monitor the progress by inspecting the `pendingTransition` field.
### Launch a Rollout
```bash theme={null}
curl -X POST "https://experiments.confidence.dev/v1/workflows/rollout/instances/brcdajvw7dfuod7cj9iq:launch" \
-H "Authorization: Bearer $TOKEN"
```
### Increase Reach
Update the rollout to increase the reach (percentage of users exposed to the feature).
```bash theme={null}
curl -X PATCH "https://experiments.confidence.dev/v1/workflows/rollout/instances/brcdajvw7dfuod7cj9iq?updateMask=rollout" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"rollout": {
"flag": "flags/search-flag",
"reach": "0.5",
"variant": "flags/search-flag/variants/enabled"
}
}'
```
Response:
```json theme={null}
{
"name": "rollouts/new-search-feature",
"displayName": "New Search Feature Rollout"
}
```
## View Results for a Rollout
To view statistical results for your rollout, use a two-step process.
### Step 1: Get the Rollout
First, get the rollout to retrieve the `analysisResult` resource name from the `stats` field.
```bash theme={null}
curl -X GET "https://experiments.confidence.dev/v1/workflows/rollout/instances/brcdajvw7dfuod7cj9iq" \
-H "Authorization: Bearer $TOKEN"
```
Response:
```json theme={null}
{
"name": "workflows/rollout/instances/brcdajvw7dfuod7cj9iq",
"displayName": "New Search Feature Rollout",
"stats": {
"analysisResult": "workflows/rollout/instances/brcdajvw7dfuod7cj9iq/analysisResults/abc123"
}
}
```
If your rollout uses multiple exposure filters, check the `stats.analysisResults[]` array instead. Each entry has an `analysisResult` resource name and the corresponding `exposureFilterName`.
### Step 2: Get the Analysis Result
Use the `analysisResult` name from Step 1 to fetch the full statistical analysis
from the [Analysis Results API](/api-reference/get-analysisresult).
```bash theme={null}
curl -X GET "https://stats.confidence.dev/v2/workflows/rollout/instances/brcdajvw7dfuod7cj9iq/analysisResults/abc123" \
-H "Authorization: Bearer $TOKEN"
```
Response:
```json theme={null}
{
"name": "workflows/rollout/instances/brcdajvw7dfuod7cj9iq/analysisResults/abc123",
"exposureFilter": "",
"annotations": [
{
"context": "OVERALL",
"info": ["Analysis completed successfully"]
}
],
"results": [
{
"id": "metrics/conversion-rate",
"statsSettings": {
"method": "METHOD_Z_TEST",
"adjustedAlpha": 0.05
},
"result": {
"status": {
"status": "METRIC_RESULT_STATUS_SIGNIFICANT_POSITIVE",
"metricType": "METRIC_TYPE_SUCCESS"
}
}
}
]
}
```
# Apply Flags
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/apply-flags
Learn how to track flag usage by applying flags.
Report when your application uses flags by applying them.
See [Apply a Flag](../../flags/resolution-reference#apply-a-flag) in the reference for details on why applies matter, when to apply, batching strategies, and best practices.
## Before You Begin
Before applying flags, you must:
1. [Resolve flags](./resolve-flags) to get variant values and a resolve token
2. Save the `resolveToken` from the resolve response
3. Use the flag value in your application
## Apply a Single Flag
Report that a flag was applied:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags:apply" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"resolveToken": "",
"flags": [
{
"flag": "flags/image-size",
"appliedTime": "2023-03-06T19:45:07.436Z",
"sentTime": "2023-03-06T19:45:08.002Z"
}
]
}'
```
## Apply Multiple Flags
Batch multiple applies together (recommended):
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags:apply" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"resolveToken": "",
"flags": [
{
"flag": "flags/image-size",
"appliedTime": "2023-03-06T19:45:07.436Z",
"sentTime": "2023-03-06T19:45:08.002Z"
},
{
"flag": "flags/button-colors",
"appliedTime": "2023-03-06T19:45:09.123Z",
"sentTime": "2023-03-06T19:45:10.002Z"
}
]
}'
```
## Complete Workflow
Here's the full workflow from resolve to apply:
```bash theme={null}
# Step 1: Resolve flags
RESPONSE=$(curl -X POST "https://flags.confidence.dev/v1/flags:resolve" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"evaluationContext": {
"user_id": "user123"
}
}')
# Step 2: Extract resolve token
RESOLVE_TOKEN=$(echo $RESPONSE | jq -r '.resolveToken')
# Step 3: Use flag values in your application
# (Your application code here)
# Step 4: Report that flags were applied
curl -X POST "https://flags.confidence.dev/v1/flags:apply" \
-H "Content-Type: application/json" \
-d "{
\"clientSecret\": \"YOUR_CLIENT_SECRET\",
\"resolveToken\": \"$RESOLVE_TOKEN\",
\"flags\": [
{
\"flag\": \"flags/my-flag\",
\"appliedTime\": \"$(date -u +"%Y-%m-%dT%H:%M:%S.%3NZ")\",
\"sentTime\": \"$(date -u +"%Y-%m-%dT%H:%M:%S.%3NZ")\"
}
]
}"
```
## Next Steps
After applying flags:
1. Monitor flag applied data in your data warehouse
2. Use the data to analyze experiments
3. Review apply patterns to optimize flag usage
# Archive Flags and Segments
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/archive-flag
Learn how to archive flags and segments when they're no longer needed.
Archive flags and segments when they're no longer needed instead of deleting them.
See [Archiving Flags](/docs/flags/create-flags#archiving-flags) in the reference for details on what happens when you archive, when to archive, and best practices.
## Archive a Flag
Archive a flag that's no longer needed:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag:archive" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Archive a Segment
Archive a segment that's no longer needed:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments/my-segment:archive" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Verify Archive Status
Check if a flag is archived:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/flags/example-flag" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
The response includes an `archived` field set to `true`.
## Archive Multiple Resources
Archive multiple flags or segments in sequence:
```bash theme={null}
#!/bin/bash
# Archive multiple flags
for flag in "old-feature" "test-flag" "deprecated-config"; do
curl -X POST "https://flags.confidence.dev/v1/flags/$flag:archive" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
echo "Archived flag: $flag"
done
# Archive multiple segments
for segment in "experiment-1" "experiment-2" "test-segment"; do
curl -X POST "https://flags.confidence.dev/v1/segments/$segment:archive" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
echo "Archived segment: $segment"
done
```
## Find Archival Candidates
List all flags to find archival candidates:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/flags" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
List all segments to find archival candidates:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/segments" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Next Steps
After archiving resources:
1. Remove feature flag code from your application
2. Update documentation to reflect production behavior
3. Review analytics from the archived experiment
4. Plan next experiments based on learnings
# Coordinate Segments
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/coordinate-segments
Learn how to make segments mutually exclusive using coordination tags.
Use coordination to make segments mutually exclusive, ensuring users can only be in one experiment at a time.
See [Coordination](../../flags/segments-reference#coordination) in the reference for details on how coordination works, common patterns, and best practices.
## Set Up Basic Coordination
Create segments that exclude each other using matching tags:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=ranking-exp-1" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Ranking Experiment 1",
"targeting": {},
"allocation": {
"proportion": {"value": "0.1"},
"exclusivityTags": ["ranking"],
"exclusiveTo": ["ranking"]
}
}'
```
Create a second segment with the same coordination:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=ranking-exp-2" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Ranking Experiment 2",
"targeting": {},
"allocation": {
"proportion": {"value": "0.1"},
"exclusivityTags": ["ranking"],
"exclusiveTo": ["ranking"]
}
}'
```
These segments are now mutually exclusive—no user can be in both.
## Coordinate Across Multiple Groups
Exclude a segment from multiple coordination groups:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=mixed-experiment" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Mixed Experiment",
"targeting": {},
"allocation": {
"proportion": {"value": "0.05"},
"exclusivityTags": ["search"],
"exclusiveTo": ["search", "ranking", "mixer"]
}
}'
```
This segment doesn't overlap with any segment that has `search`, `ranking`, or `mixer` as exclusivity tags.
## Update Coordination Tags
Change coordination tags on an existing segment:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/segments/ranking-exp-1?updateMask=allocation.exclusivityTags,allocation.exclusiveTo" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"allocation": {
"exclusivityTags": ["ranking", "search"],
"exclusiveTo": ["ranking", "search", "mixer"]
}
}'
```
Changing coordination tags on an allocated segment may require re-allocating it, which can affect which users are in the segment.
## Allocate with Coordination
Allocate a segment with coordination tags:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments/ranking-exp-1:allocate" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
If successful, the segment is allocated and guaranteed to be mutually exclusive with coordinating segments.
If allocation fails due to insufficient space:
* Reduce the allocation proportion
* Archive some existing segments
* Change coordination tags
## Check Available Space
List all segments to check available space in a coordination group:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/segments" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
Filter the results for segments with overlapping coordination tags and sum their allocation proportions to see how much space the segments use.
## Next Steps
After setting up coordination:
1. [Create flag rules](./create-flag-rule) to use your coordinated segments
2. [Allocate segments](./create-segment#allocate-a-segment) to activate them
3. [Resolve flags](./resolve-flags) to test your coordination logic
# Create a Flag
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/create-flag
Learn how to create a flag using the Confidence API.
You identify flags by a string called the flag key. This key is unique within the account. All you have to do to create a new flag is specify the flag key using the `flagId` query parameter.
## Create a Basic Flag
To create a flag with just a flag key:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags?flagId=example-flag" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json"
```
Use a name that is understandable and memorable. For example, `new-navbar`. Avoid too long names, and don't include the configuration in the name itself. For example, don't use `new-navbar-mobile-experience` or `new-navbar-enabled` as the flag name.
## Create a Flag with Clients
You can associate the flag with clients when creating it to control which applications can resolve it:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags?flagId=example-flag" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"clients": ["clients/my-client"]
}'
```
Not all clients should have access to all flags. Associate a flag with as few clients as possible to:
* Limit the number of resolved flags in batch operations
* Reduce network payload and costs
* Prevent sensitive flags from being available in untrusted environments (like mobile apps)
## Retrieve a Flag
To retrieve an existing flag and see its configuration:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/flags/example-flag" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Associate with Additional Clients
You can add more clients to a flag after creation using the `addFlagClient` operation:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag:addFlagClient" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"client": "clients/another-client"
}'
```
Or update the entire list of clients using the PATCH operation:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag?updateMask=clients" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"clients": ["clients/my-client", "clients/another-client"]
}'
```
## Next Steps
After creating a flag:
1. [Define the flag schema](./define-flag-schema)
2. [Create variants](./create-flag-variants)
3. [Set up flag clients](./setup-flag-clients) if you haven't already
4. [Create segments and rules](./create-segment) to control who sees what
5. [Resolve the flag](./resolve-flags) in your application
# Create Flag Rules
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/create-flag-rule
Learn how to create rules that assign variants to users based on segments.
Create rules to assign variants to users in a segment.
See [Rules](/docs/flags/define-rules) and [Variant Assignment](/docs/flags/define-rules#general-rules) in the reference for details on how rules work, assignment types, and best practices.
## Before You Begin
Before creating a rule, you need:
1. A flag with a defined [schema](./define-flag-schema) and [variants](./create-flag-variants)
2. An [allocated segment](./create-segment) that defines your target audience
## Create a Basic Rule
Assign all users in a segment to a single variant:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/rules" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"segment": "segments/all-users",
"assignmentSpec": {
"bucketCount": 100,
"assignments": [
{
"variant": {
"variant": "flags/example-flag/variants/enabled"
},
"bucketRanges": [{"lower": 0, "upper": 100}]
}
]
},
"targetingKeySelector": "user_id"
}'
```
## Create an A/B Test Rule
Randomly assign users to control or treatment (50/50 split):
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/rules" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"segment": "segments/experiment-segment",
"assignmentSpec": {
"bucketCount": 100,
"assignments": [
{
"variant": {
"variant": "flags/example-flag/variants/control"
},
"bucketRanges": [{"lower": 0, "upper": 50}]
},
{
"variant": {
"variant": "flags/example-flag/variants/treatment"
},
"bucketRanges": [{"lower": 50, "upper": 100}]
}
]
},
"targetingKeySelector": "user_id"
}'
```
## Create a Multi-Variant Rule
Assign users across three variants (40/30/30 split):
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/rules" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"segment": "segments/multivariate-test",
"assignmentSpec": {
"bucketCount": 100,
"assignments": [
{
"variant": {"variant": "flags/example-flag/variants/option-a"},
"bucketRanges": [{"lower": 0, "upper": 40}]
},
{
"variant": {"variant": "flags/example-flag/variants/option-b"},
"bucketRanges": [{"lower": 40, "upper": 70}]
},
{
"variant": {"variant": "flags/example-flag/variants/option-c"},
"bucketRanges": [{"lower": 70, "upper": 100}]
}
]
},
"targetingKeySelector": "user_id"
}'
```
## Create a Fall-Through Rule
Create a rule that matches but passes to the next rule:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/rules" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"segment": "segments/logging-segment",
"assignmentSpec": {
"bucketCount": 100,
"assignments": [
{
"fallthrough": {},
"bucketRanges": [{"lower": 0, "upper": 100}]
}
]
},
"targetingKeySelector": "user_id"
}'
```
## Create a Client Default Rule
Assign users to their client-specified default values:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/rules" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"segment": "segments/default-segment",
"assignmentSpec": {
"bucketCount": 100,
"assignments": [
{
"clientDefault": {},
"bucketRanges": [{"lower": 0, "upper": 100}]
}
]
},
"targetingKeySelector": "user_id"
}'
```
## Enable a Rule
Newly created rules start disabled. Enable a rule to activate it:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag/rules/RULE_ID?updateMask=enabled" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"enabled": true
}'
```
## Update a Rule
Change variant assignments or the segment:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag/rules/RULE_ID?updateMask=assignmentSpec" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"assignmentSpec": {
"bucketCount": 100,
"assignments": [
{
"variant": {"variant": "flags/example-flag/variants/control"},
"bucketRanges": [{"lower": 0, "upper": 30}]
},
{
"variant": {"variant": "flags/example-flag/variants/treatment"},
"bucketRanges": [{"lower": 30, "upper": 100}]
}
]
}
}'
```
## Reorder Rules
Change rule evaluation order by updating priority (lower numbers evaluate first):
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag/rules/RULE_ID?updateMask=priority" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"priority": 1
}'
```
## Delete a Rule
Remove a rule from a flag:
```bash theme={null}
curl -X DELETE "https://flags.confidence.dev/v1/flags/example-flag/rules/RULE_ID" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Use a Custom Targeting Key
Specify a different field for randomization (for example, device instead of user):
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/rules" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"segment": "segments/all-users",
"assignmentSpec": {
"bucketCount": 100,
"assignments": [
{
"variant": {"variant": "flags/example-flag/variants/enabled"},
"bucketRanges": [{"lower": 0, "upper": 100}]
}
]
},
"targetingKeySelector": "device_id"
}'
```
## Next Steps
After creating rules:
1. [Resolve flags](./resolve-flags) to test your rules with different evaluation contexts
2. [Apply flags](./apply-flags) to track which users see which variants
3. Monitor your experiments and adjust rules as needed
# Create Flag Variants
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/create-flag-variants
Learn how to create and manage flag variants.
Create variants to define the different values a flag can return.
## Create a Control Variant
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/variants" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "flags/example-flag/variants/control",
"value": {
"enabled": false,
"color": "blue",
"size": 10
},
"description": "Control variant"
}'
```
## Create a Treatment Variant
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/variants" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "flags/example-flag/variants/treatment",
"value": {
"enabled": true,
"color": "red",
"size": 20
},
"description": "Treatment variant"
}'
```
Variant values don't need to include all schema fields - only set the fields you need.
## Create More Variants
For multi-variant tests, create as many variants as needed:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags/example-flag/variants" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "flags/example-flag/variants/option-c",
"value": {
"enabled": true,
"color": "green",
"size": 15
},
"description": "Third variant"
}'
```
## Update a Variant
Change an existing variant's value or description:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag/variants/control?updateMask=value,description" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"value": {
"enabled": false,
"color": "navy",
"size": 12
},
"description": "Updated control variant"
}'
```
## List All Variants
Get all variants for a flag:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/flags/example-flag/variants" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Get a Specific Variant
Retrieve a single variant:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/flags/example-flag/variants/control" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Next Steps
After creating variants:
1. [Create segments](./create-segment) to target audiences
2. [Create rules](./create-flag-rule) to assign variants to segments
3. [Resolve flags](./resolve-flags) to get variant values
# Create Materialized Segments
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/create-materialized-segment
Learn how to create materialized segments that load units from BigQuery.
A materialized segment is a segment that loads its list of units from your BigQuery instance using a SQL query.
Materialized segments are only available for BigQuery.
## Create a Materialized Segment
To create a materialized segment:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/materializedSegments?materializationId=my-mat-segment" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"materializedSegment": {
"displayName": "My Materialized Segment"
}
}'
```
## Create a Load Job
After creating the segment, create a load job to populate it with data from your SQL query:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/materializedSegments/my-mat-segment/loadJobs" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"materializedSegmentJob": {
"entityIdColumn": "user_id",
"sql": "SELECT user_id FROM your_dataset.your_table WHERE your_criteria"
}
}'
```
The `entityIdColumn` specifies which column in your query results contains the entity IDs to include in the segment.
## Get a Materialized Segment
To retrieve a materialized segment's configuration:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/materializedSegments/my-mat-segment" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## List Load Jobs
To see the load jobs for a materialized segment:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/materializedSegments/my-mat-segment/loadJobs" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Delete a Materialized Segment
To delete a materialized segment:
```bash theme={null}
curl -X DELETE "https://flags.confidence.dev/v1/materializedSegments/my-mat-segment" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Next Steps
After creating materialized segments:
1. Use the segment in [flag rules](./create-flag-rule) to target specific users
2. Set up [coordination](./coordinate-segments) to make segments mutually exclusive
# Create Segments
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/create-segment
Learn how to create segments to define target audiences.
A segment is a cohort of users. Targeting and allocation define the cohort:
* **Targeting**: A set of criteria that filter users based on attributes
* **Allocation**: What percentage (0% to 100%) of the targeted users should be in the segment
Confidence makes no assumptions about the entity that you target or randomize on. Usually, the entity is a type of user (represented by an identifier). Because of this, the examples on this page involve users. Your unit could be something else.
## Create a Basic Segment
To create a segment that targets 100% of all users:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=all-users" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "All Users",
"targeting": {},
"allocation": {
"proportion": {
"value": "1.0"
}
}
}'
```
## Create a Segment with Targeting
To create a segment that targets users from Sweden with 10% allocation:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=sweden-10pct" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "10% of users from Sweden",
"targeting": {
"criteria": {
"sweden": {
"attribute": {
"attributeName": "country",
"eqRule": {
"value": {
"stringValue": "SE"
}
}
}
}
},
"expression": {
"ref": "sweden"
}
},
"allocation": {
"proportion": {
"value": "0.1"
}
}
}'
```
## Set the Randomization Unit
By default, Confidence randomizes based on the `targeting_key` field in the evaluation context. You can specify a different field using `targetingKeySelector`:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=device-based" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Device-based segment",
"targeting": {},
"allocation": {
"proportion": {
"value": "0.5"
},
"targetingKeySelector": "device_id"
}
}'
```
If the randomization field is missing from the evaluation context or is `null`, the segment doesn't match. The empty string (`""`) is a valid value for randomization.
## Allocate a Segment
Segments start in an `UNALLOCATED` state. To use a segment in a flag rule, you must first allocate it:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments/sweden-10pct:allocate" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
When coordinating with other segments, the allocation operation may fail if there's not enough space to make the segment mutually exclusive with overlapping segments. See [Coordinate Segments](./coordinate-segments) for more details.
## Get a Segment
To retrieve a segment's configuration:
```bash theme={null}
curl -X GET "https://flags.confidence.dev/v1/segments/sweden-10pct" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Update a Segment
You can update a segment's targeting or allocation using the PATCH endpoint:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/segments/sweden-10pct?updateMask=allocation.proportion" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"allocation": {
"proportion": {
"value": "0.2"
}
}
}'
```
Updating an allocated segment may require re-allocating it. Make changes carefully to avoid disrupting active experiments.
## Archive a Segment
When a segment is no longer needed, archive it:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments/sweden-10pct:archive" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
You can no longer use archived segments in new rules, but existing rules continue to work.
## Next Steps
After creating segments:
1. [Add targeting criteria](./target-with-criteria) for more sophisticated audience targeting
2. [Set up coordination](./coordinate-segments) to make segments mutually exclusive
3. [Create flag rules](./create-flag-rule) to assign variants to segments
# Define a Flag Schema
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/define-flag-schema
Learn how to define the schema for a flag.
After creating a flag, define its schema to specify what fields the flag value can contain.
## Define the Schema
Set the schema on your flag:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag?updateMask=schema" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"schema": {
"schema": {
"enabled": {
"boolSchema": {}
},
"color": {
"stringSchema": {}
},
"size": {
"intSchema": {}
}
}
}
}'
```
Available schema types: `boolSchema`, `stringSchema`, `intSchema`, `doubleSchema`, `listSchema`, `structSchema`. See [Variants](../../flags/concepts#variants) for details on how schemas work.
## Add Fields to an Existing Schema
To add new fields, include them in the schema:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag?updateMask=schema" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"schema": {
"schema": {
"enabled": {
"boolSchema": {}
},
"color": {
"stringSchema": {}
},
"size": {
"intSchema": {}
},
"opacity": {
"doubleSchema": {}
}
}
}
}'
```
## Define a Nested Schema
Use `structSchema` for nested fields:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag?updateMask=schema" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"schema": {
"schema": {
"button": {
"structSchema": {
"schema": {
"color": {
"stringSchema": {}
},
"size": {
"intSchema": {}
}
}
}
}
}
}
}'
```
## Define a List Schema
Use `listSchema` for array fields:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag?updateMask=schema" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"schema": {
"schema": {
"colors": {
"listSchema": {
"elementSchema": {
"stringSchema": {}
}
}
}
}
}
}'
```
## Remove a Field
To remove a field, first update all variants to remove that field, then update the schema:
```bash theme={null}
curl -X PATCH "https://flags.confidence.dev/v1/flags/example-flag?updateMask=schema" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"schema": {
"schema": {
"enabled": {
"boolSchema": {}
},
"color": {
"stringSchema": {}
}
}
}
}'
```
The request will fail if any variant still uses the field you're trying to remove.
## Next Steps
After defining your schema:
1. [Create variants](./create-flag-variants) with values matching the schema
2. [Create segments](./create-segment) to target audiences
3. [Create rules](./create-flag-rule) to assign variants
# Resolve Flags
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/resolve-flags
Learn how to resolve flags to get variant values for users.
Resolving flags means getting the value of a flag for a given evaluation context. The evaluation context contains information about the user, device, or other contextual data that Confidence uses to determine which variant to return.
## Before You Begin
Before resolving flags, you need:
1. A [flag client with credentials](./setup-flag-clients)
2. Flags [associated with the client](./create-flag#associate-with-additional-clients)
3. [Rules defined](./create-flag-rule) on your flags
While you can resolve flags directly using the API as shown here, the most efficient way is using one of the Confidence SDKs. The SDKs handle caching, batching, and other optimizations automatically.
## Authentication
Resolve operations use client secrets for authentication, not Bearer tokens:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags:resolve" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"evaluationContext": { ... }
}'
```
## Evaluation Context
The evaluation context specifies contextual data that Confidence uses to evaluate rules. It's a schema-less key-value map (JSON object) containing any data needed for rule evaluation, such as:
* User identifiers
* User attributes (country, age, plan level)
* Device information (OS, model, browser)
* Session data (URL, referrer, timestamp)
Example evaluation context:
```json theme={null}
{
"user_id": "rosling",
"country": "SE",
"device": {
"vendor": "apple",
"os": "ios"
}
}
```
Do not pass sensitive data in the keys of the evaluation context. Values are never stored, but keys are temporarily stored to help derive the schema for targeting.
## Batch Resolve (Recommended)
The recommended approach is to batch resolve all flags for a client. This returns values for all flags associated with the client:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags:resolve" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"evaluationContext": {
"user_id": "rosling",
"country": "SE",
"device": {
"vendor": "apple",
"os": "ios"
}
}
}'
```
Response:
```json theme={null}
{
"resolvedFlags": [
{
"flag": "flags/button-colors",
"variant": "",
"value": {},
"flagSchema": {
"schema": {}
},
"reason": "RESOLVE_REASON_NO_SEGMENT_MATCH"
},
{
"flag": "flags/image-size",
"variant": "flags/image-size/variants/control",
"value": {
"name": "costello",
"img-size": 42
},
"flagSchema": {
"schema": {
"name": {
"stringSchema": {}
},
"img-size": {
"intSchema": {}
}
}
},
"reason": "RESOLVE_REASON_MATCH"
}
],
"resolveToken": ""
}
```
Keep the `resolveToken` from the response. You'll need it when [applying flags](./apply-flags) to track usage.
## Resolve Specific Flags
You can limit the resolve operation to specific flags using the `flags` parameter:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags:resolve" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"flags": ["flags/image-size", "flags/button-colors"],
"evaluationContext": {
"user_id": "rosling",
"country": "SE"
}
}'
```
This returns only the specified flags, which can reduce response size and processing time.
## Understand Resolve Reasons
Each resolved flag includes a `reason` field explaining why it resolved the way it did:
| Reason | Description |
| :-------------------------------- | :------------------------------------- |
| `RESOLVE_REASON_MATCH` | A rule matched and assigned a variant |
| `RESOLVE_REASON_NO_SEGMENT_MATCH` | No rule matched the evaluation context |
| `RESOLVE_REASON_FLAG_ARCHIVED` | The flag is archived |
| `RESOLVE_REASON_ERROR` | An error occurred during resolution |
## Test Resolve Logic
To test your resolve logic with different evaluation contexts:
1. **Vary the context**: Try different user IDs, countries, device types
2. **Check reasons**: Look at the `reason` field to understand why each flag resolved as it did
3. **Verify variants**: Ensure the returned variants match your expectations
4. **Test edge cases**: Try missing fields, null values, unexpected data types
Example testing different contexts:
```bash theme={null}
# Test user from Sweden
curl -X POST "https://flags.confidence.dev/v1/flags:resolve" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"evaluationContext": {
"user_id": "user123",
"country": "SE"
}
}'
# Test user from US
curl -X POST "https://flags.confidence.dev/v1/flags:resolve" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"evaluationContext": {
"user_id": "user456",
"country": "US"
}
}'
```
## Handle No Match
When no rule matches, the flag returns with `RESOLVE_REASON_NO_SEGMENT_MATCH` and empty values:
```json theme={null}
{
"flag": "flags/my-flag",
"variant": "",
"value": {},
"reason": "RESOLVE_REASON_NO_SEGMENT_MATCH"
}
```
Your application should handle this case by:
* Using a default value specified in your code
* Falling back to the original behavior
* Logging the no-match event for debugging
## Best Practices
1. **Batch resolve**: Always resolve all flags in one call to minimize network overhead
2. **Cache locally**: Cache resolved values to avoid repeated API calls
3. **Provide complete context**: Include all relevant fields that your targeting rules might use
4. **Handle failures gracefully**: Have fallback values if the resolve call fails
5. **Use consistent keys**: Ensure the `targeting_key` (or custom field) is stable for each user
## Next Steps
After resolving flags:
1. Use the variant values in your application
2. [Apply the flags](./apply-flags) to track usage and enable experiment analysis
3. Monitor resolve patterns and optimize your rules as needed
# Set Up Flag Clients
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/setup-flag-clients
Learn how to create flag clients and manage their credentials.
Clients resolve flags into values. A client could be a mobile app, website, or backend service. These clients often run in an untrusted environment and authenticate with a different mechanism than service APIs of Confidence.
The calling client authenticates using a shared secret between the caller and Confidence. This secret, called "client secret," belongs to a client resource in Confidence.
## Create a Flag Client
To create a client, provide a display name that identifies the application:
```bash theme={null}
curl -X POST "https://iam.confidence.dev/v1/clients" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "My client"
}'
```
Response:
```json theme={null}
{
"name": "clients/1bhq4c2zqigdzqg6ufni",
"displayName": "My client",
"createTime": "2023-08-29T09:36:57.163017Z",
"updateTime": "2023-08-29T09:36:57.163017Z"
}
```
## Create Client Credentials
After creating a client, generate credentials (client secret) for authentication:
```bash theme={null}
curl -X POST "https://iam.confidence.dev/v1/clients/1bhq4c2zqigdzqg6ufni/credentials" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json"
```
Response:
```json theme={null}
{
"name": "clients/1bhq4c2zqigdzqg6ufni/credentials/abc123",
"secret": "ZXhhbXBsZV9zZWNyZXRfa2V5X3RoYXRfeW91X3Nob3VsZF9rZWVwX3NhZmU",
"createTime": "2023-08-29T09:40:12.456789Z"
}
```
Make note of the client secret. It's only returned once, from the create operation. If you lose it, you have to create a new one.
Store the client secret securely. Anyone with this secret can resolve flags on behalf of this client.
## Use Client Secrets
The client secret authenticates when resolving and applying flags. Unlike other API operations that use Bearer tokens, resolve and apply operations use the client secret directly:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/flags:resolve" \
-H "Content-Type: application/json" \
-d '{
"clientSecret": "YOUR_CLIENT_SECRET",
"evaluationContext": {
"user_id": "example-user"
}
}'
```
## List Clients
To view all clients in your account:
```bash theme={null}
curl -X GET "https://iam.confidence.dev/v1/clients" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Get a Specific Client
To retrieve details about a specific client:
```bash theme={null}
curl -X GET "https://iam.confidence.dev/v1/clients/1bhq4c2zqigdzqg6ufni" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Rotate Client Secrets
For security best practices, periodically rotate client secrets:
1. Create a new credential for the client
2. Update your application to use the new secret
3. Delete the old credential after the migration is complete
To delete a credential:
```bash theme={null}
curl -X DELETE "https://iam.confidence.dev/v1/clients/1bhq4c2zqigdzqg6ufni/credentials/abc123" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
## Next Steps
After setting up flag clients:
1. [Create flags](./create-flag) and associate them with clients
2. [Resolve flags](./resolve-flags) using the client secret
3. [Apply flags](./apply-flags) to track usage
# Target with Criteria
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/flags/target-with-criteria
Learn how to use targeting criteria to filter users in segments.
Add targeting criteria to your segments to filter users based on attributes from the evaluation context.
See [Targeting Criteria](../../flags/segments-reference#targeting-criteria) in the reference for details on operators, value types, and how criteria work.
## Use Equality Matching
Match users where a field equals a specific value:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=sweden-users" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Users from Sweden",
"targeting": {
"criteria": {
"sweden": {
"attribute": {
"attributeName": "country",
"eqRule": {
"value": {"stringValue": "SE"}
}
}
}
},
"expression": {"ref": "sweden"}
},
"allocation": {"proportion": {"value": "1.0"}}
}'
```
## Use Set Matching
Match users where a field equals any value from a set:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=nordic-users" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Nordic users",
"targeting": {
"criteria": {
"nordics": {
"attribute": {
"attributeName": "country",
"setRule": {
"values": [
{"stringValue": "SE"},
{"stringValue": "DK"},
{"stringValue": "NO"},
{"stringValue": "FI"},
{"stringValue": "IS"}
]
}
}
}
},
"expression": {"ref": "nordics"}
},
"allocation": {"proportion": {"value": "1.0"}}
}'
```
## Use Range Matching
Match users where a field falls within a range:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=age-40-49" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Users aged 40-49",
"targeting": {
"criteria": {
"age-range": {
"attribute": {
"attributeName": "age",
"rangeRule": {
"startInclusive": {"numberValue": 40},
"endExclusive": {"numberValue": 50}
}
}
}
},
"expression": {"ref": "age-range"}
},
"allocation": {"proportion": {"value": "1.0"}}
}'
```
## Match Against Another Segment
Check if a user is in another segment:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=premium-sweden" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Premium users from Sweden",
"targeting": {
"criteria": {
"sweden": {
"attribute": {
"attributeName": "country",
"eqRule": {"value": {"stringValue": "SE"}}
}
},
"premium": {
"segment": {"segment": "segments/premium-users"}
}
},
"expression": {
"and": {
"operands": [
{"ref": "sweden"},
{"ref": "premium"}
]
}
}
},
"allocation": {"proportion": {"value": "0.1"}}
}'
```
## Combine Multiple Criteria
Use AND, OR, and NOT to create complex logic:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=nordic-android" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Nordic Android users",
"targeting": {
"criteria": {
"nordics": {
"attribute": {
"attributeName": "country",
"setRule": {
"values": [
{"stringValue": "SE"},
{"stringValue": "DK"},
{"stringValue": "NO"},
{"stringValue": "FI"}
]
}
}
},
"ios": {
"attribute": {
"attributeName": "device.os",
"eqRule": {"value": {"stringValue": "ios"}}
}
}
},
"expression": {
"and": {
"operands": [
{"ref": "nordics"},
{"not": {"ref": "ios"}}
]
}
}
},
"allocation": {"proportion": {"value": "1.0"}}
}'
```
## Use Nested Fields
Reference nested fields with dot notation:
```bash theme={null}
curl -X POST "https://flags.confidence.dev/v1/segments?segmentId=iphone-users" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "iPhone users",
"targeting": {
"criteria": {
"iphone": {
"attribute": {
"attributeName": "device.model",
"eqRule": {"value": {"stringValue": "iPhone14"}}
}
}
},
"expression": {"ref": "iphone"}
},
"allocation": {"proportion": {"value": "1.0"}}
}'
```
This matches against evaluation context like:
```json theme={null}
{
"user_id": "123",
"device": {
"model": "iPhone14",
"os": "ios"
}
}
```
## Next Steps
After setting up targeting criteria:
1. [Coordinate segments](./coordinate-segments) to make them mutually exclusive
2. [Create flag rules](./create-flag-rule) to assign variants
3. [Resolve flags](./resolve-flags) with the appropriate evaluation context
# OAuth Apps
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/iam/create-oauth-apps
Learn about Confidence and third-party apps.
OAuth Apps is a way for third-party applications to access Confidence on behalf of a user. With an OAuth App, you can, for example, develop your own application that accesses Confidence.
## Create an OAuth App
To create an OAuth App, pass a name, description, and callback URLs.
```bash theme={null}
curl -X POST "https://iam.confidence.dev/v1/oauthApps" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "My OAuth App",
"description": "Test app",
"logoUri": "https://confidence.spotify.com/logo.png",
"allowedCallbackUrls": [
"https://confidence.spotify.com"
],
"allowedLogoutUrls": [
"https://confidence.spotify.com"
],
"allowedWebOrigins": [
"https://confidence.spotify.com",
"http://localhost:3000"
]
}'
```
Response:
```json theme={null}
{
"name": "oauthApps/vkq6qw3qp6nbefxgbkr0",
"displayName": "My OAuth App",
"clientId": "vrBQfwAz8qesNWE84BAImPzCnkAhql7u",
"clientSecret": "PWySgGDYi5gnxCRTNEI29UFalrlF_m2fhZOiiEw_YFVeCxoPHeQlE5m4kid-WtNB",
"description": "Test app",
"logoUri": "https://confidence.spotify.com/logo.png",
"allowedCallbackUrls": [
"https://confidence.spotify.com"
],
"allowedLogoutUrls": [
"https://confidence.spotify.com"
],
"allowedWebOrigins": [
"https://confidence.spotify.com",
"http://localhost:3000"
],
"createdBy": "users/xa8cecs2cc9xsz8jvgmc",
"updatedBy": "users/xa8cecs2cc9xsz8jvgmc",
"labels": {},
"createTime": "2023-08-31T11:25:11.536748Z",
"updateTime": "2023-08-31T11:25:11.536748Z"
}
```
After you've created the OAuth App, you receive a `clientId` and `clientSecret` that you can use to get an access token from the OAuth API at `https://auth.confidence.dev/oauth/token`. You may only use one of the following grant types: `implicit`, `authorization_code`, or `refresh_token`. For more information on grant types, see the [Auth0 grant types guide](https://auth0.com/docs/get-started/applications/application-grant-types). For details on obtaining access tokens, see the [Auth0 access token guide](https://auth0.com/docs/secure/tokens/access-tokens/get-access-tokens).
## Get an OAuth App
You can get an OAuth App by passing the name of the app to the following endpoint.
```bash theme={null}
curl -X GET "https://iam.confidence.dev/v1/oauthApps/vkq6qw3qp6nbefxgbkr0" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
Response:
```json theme={null}
{
"name": "oauthApps/vkq6qw3qp6nbefxgbkr0",
"displayName": "My OAuth App",
"clientId": "vrBQfwAz8qesNWE84BAImPzCnkAhql7u",
"clientSecret": "PWySgGDYi5gnxCRTNEI29UFalrlF_m2fhZOiiEw_YFVeCxoPHeQlE5m4kid-WtNB",
"description": "Test app",
"logoUri": "https://confidence.spotify.com/logo.png",
"allowedCallbackUrls": [
"https://confidence.spotify.com"
],
"allowedLogoutUrls": [
"https://confidence.spotify.com"
],
"allowedWebOrigins": [
"https://confidence.spotify.com",
"http://localhost:3000"
],
"createdBy": "users/xa8cecs2cc9xsz8jvgmc",
"updatedBy": "users/xa8cecs2cc9xsz8jvgmc",
"labels": {},
"createTime": "2023-08-31T11:25:11.536748Z",
"updateTime": "2023-08-31T11:25:11.536748Z"
}
```
## Delete an OAuth App
You can delete an OAuth App by calling the delete endpoint. This does not revoke any already-issued access tokens.
```bash theme={null}
curl -X DELETE "https://iam.confidence.dev/v1/oauthApps/vkq6qw3qp6nbefxgbkr0" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
# Manage API Clients
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/iam/manage-api-clients
Learn how to create API clients.
API Clients are machine users that can interact with the Confidence APIs. Confidence has APIs for managing API clients programmatically.
## Create an API Client
To create an API client you need to specify the name and desired permissions. You can either specify each permission separately or specify a role. In this example, you create an API client for managing flags. Give it a `Flag editor` role.
```bash theme={null}
curl -X POST "https://iam.confidence.dev/v1/apiClients" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "My API Client",
"description": "Client for setting up flags in GHE repos",
"permissions": [],
"roles": [
"roles/726f6c5f68386b6c46596c4a335350384e6a7377"
]
}'
```
Response:
```json theme={null}
{
"name": "apiClients/donlangsbyz6iiksdt4c",
"displayName": "My API Client",
"description": "Client for setting up flags in GHE repos",
"clientId": "k9KkmxZfQtG6ZPqk9wyQfe7G5EzZsjza",
"clientSecret": "4wvbCSdF7wlTdn8HJEqSkcuSqyndjwcHzQ3KEwxRcqvkfLeEG4eFapHQUIrRuDxr",
"permissions": [],
"roles": [
"roles/726f6c5f68386b6c46596c4a335350384e6a7377"
],
"createdBy": "users/xa8cecs2cc9xsz8jvgmc",
"updatedBy": "users/xa8cecs2cc9xsz8jvgmc",
"labels": {},
"createTime": "2023-08-29T09:36:57.163017Z",
"updateTime": "2023-08-29T09:36:57.163017Z"
}
```
## Get an API Client
You can get an API client by passing the name of the client to the following endpoint.
```bash theme={null}
curl -X GET "https://iam.confidence.dev/v1/apiClients/donlangsbyz6iiksdt4c" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
Response:
```json theme={null}
{
"name": "apiClients/donlangsbyz6iiksdt4c",
"displayName": "My API Client",
"description": "Client for setting up flags in GHE repos",
"clientId": "k9KkmxZfQtG6ZPqk9wyQfe7G5EzZsjza",
"clientSecret": "4wvbCSdF7wlTdn8HJEqSkcuSqyndjwcHzQ3KEwxRcqvkfLeEG4eFapHQUIrRuDxr",
"permissions": [],
"roles": [
"roles/726f6c5f68386b6c46596c4a335350384e6a7377"
],
"createdBy": "users/xa8cecs2cc9xsz8jvgmc",
"updatedBy": "users/xa8cecs2cc9xsz8jvgmc",
"labels": {},
"createTime": "2023-08-29T09:36:57.163017Z",
"updateTime": "2023-08-29T09:36:57.163017Z"
}
```
## List API Clients
You can list all API clients in the organization using the following endpoint.
```bash theme={null}
curl -X GET "https://iam.confidence.dev/v1/apiClients" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
Response:
```json theme={null}
{
"apiClients": [
{
"name": "apiClients/donlangsbyz6iiksdt4c",
"displayName": "My API Client",
"description": "Client for setting up flags in GHE repos",
"clientId": "k9KkmxZfQtG6ZPqk9wyQfe7G5EzZsjza",
"clientSecret": "4wvbCSdF7wlTdn8HJEqSkcuSqyndjwcHzQ3KEwxRcqvkfLeEG4eFapHQUIrRuDxr",
"permissions": [],
"roles": [
"roles/726f6c5f68386b6c46596c4a335350384e6a7377"
],
"createdBy": "users/xa8cecs2cc9xsz8jvgmc",
"updatedBy": "users/xa8cecs2cc9xsz8jvgmc",
"labels": {},
"createTime": "2023-08-29T09:36:57.163017Z",
"updateTime": "2023-08-29T09:36:57.163017Z"
}
],
"nextPageToken": ""
}
```
# Manage Users
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/iam/manage-users
Learn how to fetch and list users.
Confidence has endpoints for fetching and listing users that are part of your Confidence organization.
## Get a User
You can get a user by passing the name of the user to the following endpoint.
```bash theme={null}
curl -X GET "https://iam.confidence.dev/v1/users/xa8cecs2cc9xsz8jvgmc" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
Response:
```json theme={null}
{
"name": "users/xa8cecs2cc9xsz8jvgmc",
"fullName": "John Doe",
"email": "johndoe@confidence.dev",
"pictureUri": "https://lh3.googleusercontent.com/a/AAcHTtxy1zIWynFKa3DWtaQAy4VAXOMnIj0SBwwzviO05YTvpD=s96-c",
"blocked": false,
"labels": {},
"lastLoginTime": "2023-08-29T08:48:16.266Z",
"createTime": "2022-12-08T12:12:58.332Z",
"updateTime": "2023-08-29T08:48:16.266Z"
}
```
## List Users
You can list all users in the organization using the following endpoint.
```bash theme={null}
curl -X GET "https://iam.confidence.dev/v1/users" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
Response:
```json theme={null}
{
"users": [
{
"name": "users/xa8cecs2cc9xsz8jvgmc",
"fullName": "John Doe",
"email": "johndoe@confidence.dev",
"pictureUri": "https://lh3.googleusercontent.com/a/AAcHTtxy1zIWynFKa3DWtaQAy4VAXOMnIj0SBwwzviO05YTvpD=s96-c",
"blocked": false,
"labels": {},
"lastLoginTime": "2023-08-29T08:48:16.266Z",
"createTime": "2022-12-08T12:12:58.332Z",
"updateTime": "2023-08-29T08:48:16.266Z"
}
],
"nextPageToken": "MjAyMy0wOC0yNCAxMjo1OToyMy4wMjcwMDBVVEMsZ29vZ2xlLW9hdXRoMnwxMDQzMTQwODE5OTM2Mjk5NjgyNDE"
}
```
# Create Assignment Tables
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/metrics/create-assignment-table
Learn how to create assignment tables using the API.
Create assignment tables that store records of what entities have been assigned what configuration.
## Before You Begin
Before creating an assignment table, ensure you have:
* An API access token with appropriate permissions
* Created the entity you want to track assignments for
* Prepared a SQL query that selects assignment rows from your data warehouse
* Identified the columns for timestamp, entity ID, exposure key, and variant key
## Create an Assignment Table
Create an assignment table with automatic data delivery:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/assignmentTables" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "My Assignment Table",
"dataDeliveredUntilUpdateStrategyConfig": {
"strategy": "AUTOMATIC",
"automaticUpdateConfig": {
"incrementDuration": "P13H",
"commitDelay": "P13H"
}
},
"sql": "SELECT * FROM my_table",
"entityColumnMapping": {
"column": {
"name": "user_id"
},
"entity": "entities/user"
},
"timestampColumn": {
"name": "timestamp"
},
"exposureKeyColumn": {
"name": "experiment_id"
},
"variantKeyColumn": {
"name": "group"
}
}'
```
## Column Mapping Fields
* **entityColumnMapping**: Maps the column containing entity IDs to the entity resource
* `column.name`: Name of the column in your query results
* `entity`: Resource name of the entity (for example, `entities/user`)
* `timestampColumn`: Column containing the assignment timestamp
* `exposureKeyColumn`: Column identifying which experiment the assignment belongs to
* `variantKeyColumn`: Column identifying which variant was assigned
## Data Delivery Strategy
Configure how Confidence updates data from your warehouse:
```json theme={null}
"dataDeliveredUntilUpdateStrategyConfig": {
"strategy": "AUTOMATIC",
"automaticUpdateConfig": {
"incrementDuration": "P13H",
"commitDelay": "P13H"
}
}
```
* **incrementDuration**: How frequently to check for new data
* **commitDelay**: Buffer time before considering data complete
```json theme={null}
"dataDeliveredUntilUpdateStrategyConfig": {
"strategy": "DAILY",
"dailyUpdateConfig": {}
}
```
Updates once per day at a scheduled time.
## Next Steps
After creating an assignment table:
* [Create fact tables](./create-fact-table) to measure outcomes
* [Create metrics](./create-metric) using your assignment and fact tables
* Configure experiments to use this assignment table
# Create Dimension Tables
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/metrics/create-dimension-table
Learn how to create dimension tables using the API.
Create dimension tables that let you segment your entities for analysis.
## Before You Begin
Before creating a dimension table, ensure you have:
* An API access token with appropriate permissions
* Created the entities you want to segment
* Prepared a SQL query that selects dimension data from your data warehouse
* Identified entity and dimension columns
## Create a Dimension Table
Create a dimension table with a country dimension:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/dimensionTables" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "User Location",
"sql": "SELECT * FROM confidence.user_data.location",
"timestampColumn": {
"name": "date",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entityColumnMapping": [
{
"column": {
"name": "user_id",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entity": "entities/user"
}
],
"dimensions": [
{
"name": "country",
"type": "COLUMN_TYPE_STRING",
"repeated": false
}
],
"dataDeliveredUntilUpdateStrategyConfig": {
"dailyUpdateConfig": {}
}
}'
```
## Create a Dimension Table with Multiple Dimensions
Track multiple attributes for segmentation:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/dimensionTables" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "User Demographics",
"sql": "SELECT * FROM user_demographics",
"timestampColumn": {
"name": "updated_at",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entityColumnMapping": [
{
"column": {
"name": "user_id",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entity": "entities/user"
}
],
"dimensions": [
{
"name": "country",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
{
"name": "age_group",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
{
"name": "is_premium",
"type": "COLUMN_TYPE_BOOLEAN",
"repeated": false
}
],
"dataDeliveredUntilUpdateStrategyConfig": {
"dailyUpdateConfig": {}
}
}'
```
## Dimension Types
Supported dimension types:
* `COLUMN_TYPE_STRING`: Categorical values (country, platform, etc.)
* `COLUMN_TYPE_BOOLEAN`: Binary attributes (`is_premium`, `is_active`, etc.)
* `COLUMN_TYPE_INTEGER`: Numeric categories (age group codes, tier levels, etc.)
## Data Delivery
After creation, the dimension table enters the `CREATING` state. Confidence runs a sample query to verify the SQL produces the expected columns, then transitions to either `ACTIVE` or `FAILED`.
## Next Steps
After creating dimension tables:
* Use dimensions to segment metrics in experiment analysis
* [Create metrics](./create-metric) that you can break down by these dimensions
* Configure experiments to analyze results by dimension
# Create Entities
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/metrics/create-entity
Learn how to create entities using the API.
Create entities that can be uniquely identified and randomized, like users or sessions.
## Before You Begin
Before creating entities, ensure you have:
* An API access token with appropriate permissions
* Determined the data type of your entity identifier (string, integer, etc.)
## Create an Entity
To create an entity of type string called `User`:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/entities" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "User",
"primaryKeyType": "COLUMN_TYPE_STRING"
}'
```
The response includes the entity resource name you can use to reference it:
```json theme={null}
{
"name": "entities/user",
"displayName": "User",
"primaryKeyType": "COLUMN_TYPE_STRING"
}
```
## Create an Integer Entity
For entities with integer identifiers:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/entities" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Session",
"primaryKeyType": "COLUMN_TYPE_INTEGER"
}'
```
## Supported Primary Key Types
* `COLUMN_TYPE_STRING`: String identifiers (UUIDs, usernames, etc.)
* `COLUMN_TYPE_INTEGER`: Integer identifiers
* `COLUMN_TYPE_BOOLEAN`: Boolean values
* `COLUMN_TYPE_DOUBLE`: Floating point numbers
## Next Steps
After creating entities:
* [Create an assignment table](./create-assignment-table) to track assignments
* [Create fact tables](./create-fact-table) to measure entity behaviors
* [Create dimension tables](./create-dimension-table) to segment entities
# Create Fact Tables
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/metrics/create-fact-table
Learn how to create fact tables using the API.
Create fact tables that contain measurements describing your entities.
## Before You Begin
Before creating a fact table, ensure you have:
* An API access token with appropriate permissions
* Created the entities you want to measure
* Prepared a SQL query that selects measurement rows from your data warehouse
* Identified timestamp, entity, and measurement columns
## Create a Fact Table
Create a fact table with a boolean measurement:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/factTables" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "checkouts",
"displayName": "Checkouts",
"sql": "SELECT * FROM confidence.analysis.checkouts",
"timestampColumn": {
"name": "date",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entities": [
{
"column": {
"name": "user_id",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entity": "entities/user"
}
],
"measurements": [
{
"name": "completed_checkout",
"type": "COLUMN_TYPE_BOOLEAN",
"repeated": false
}
],
"dataDeliveredUntilUpdateStrategyConfig": {
"dailyUpdateConfig": {}
}
}'
```
## Create a Fact Table with Numeric Measurements
For tables with numeric measurements like revenue:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/factTables" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "purchases",
"displayName": "Purchases",
"sql": "SELECT * FROM purchases",
"timestampColumn": {
"name": "purchase_date",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entities": [
{
"column": {
"name": "user_id",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entity": "entities/user"
}
],
"measurements": [
{
"name": "purchase_amount",
"type": "COLUMN_TYPE_DOUBLE",
"repeated": false
},
{
"name": "item_count",
"type": "COLUMN_TYPE_INTEGER",
"repeated": false
}
],
"dataDeliveredUntilUpdateStrategyConfig": {
"dailyUpdateConfig": {}
}
}'
```
## Create a Fact Table with Multiple Entities
Track measurements across multiple entities:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/factTables" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "sessions",
"displayName": "User Sessions",
"sql": "SELECT * FROM sessions",
"timestampColumn": {
"name": "session_start",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entities": [
{
"column": {
"name": "user_id",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entity": "entities/user"
},
{
"column": {
"name": "session_id",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entity": "entities/session"
}
],
"measurements": [
{
"name": "duration_seconds",
"type": "COLUMN_TYPE_INTEGER",
"repeated": false
}
],
"dataDeliveredUntilUpdateStrategyConfig": {
"dailyUpdateConfig": {}
}
}'
```
## Column Types
Supported measurement types:
* `COLUMN_TYPE_BOOLEAN`: True/false values
* `COLUMN_TYPE_INTEGER`: Whole numbers
* `COLUMN_TYPE_DOUBLE`: Decimal numbers
* `COLUMN_TYPE_STRING`: Text values
## Data Delivery
After creation, the fact table enters the `CREATING` state. Confidence runs a sample query to verify the SQL produces the expected columns, then transitions to either `ACTIVE` or `FAILED`.
## Next Steps
After creating fact tables:
* [Create metrics](./create-metric) to aggregate measurements
* [Create dimension tables](./create-dimension-table) to segment analysis
* Configure experiments to track these measurements
# Create Metrics
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/metrics/create-metric
Learn how to create metrics using the API.
Create metrics that aggregate measurements across instances of an entity.
## Before You Begin
Before creating a metric, ensure you have:
* An API access token with appropriate permissions
* Created the entity you want to measure
* Created a fact table with the measurements to aggregate
* Determined the aggregation type and time windows
## Create an Average Metric
Create an average metric that measures the average sales amount per user:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/metrics" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Average sales amount per user during the first week after exposure",
"entity": "entities/user",
"factTable": "factTables/my-fact-table",
"aggregationWindow": "604800s",
"exposureOffset": "0s",
"typeSpec": {
"averageMetricSpec": {
"measurement": {
"name": "sales_amount",
"repeated": false
},
"aggregation": {
"type": "AGGREGATION_TYPE_SUM"
}
}
},
"varianceReductionConfig": {
"disabled": false
}
}'
```
## Create a Ratio Metric
Create a ratio metric like conversion rate:
```bash theme={null}
curl -X POST "https://metrics.confidence.dev/v1/metrics" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"displayName": "Conversion rate in first week",
"entity": "entities/user",
"factTable": "factTables/conversions",
"aggregationWindow": "604800s",
"exposureOffset": "0s",
"typeSpec": {
"ratioMetricSpec": {
"numerator": {
"measurement": {
"name": "converted",
"repeated": false
},
"aggregation": {
"type": "AGGREGATION_TYPE_SUM"
}
},
"denominator": {
"measurement": {
"name": "eligible",
"repeated": false
},
"aggregation": {
"type": "AGGREGATION_TYPE_SUM"
}
}
}
},
"varianceReductionConfig": {
"disabled": false
}
}'
```
## Aggregation Types
For average metrics, choose how to aggregate data within units:
* `AGGREGATION_TYPE_SUM`: Sum all measurements
* `AGGREGATION_TYPE_COUNT`: Count occurrences
* `AGGREGATION_TYPE_COUNT_DISTINCT`: Count unique values
* `AGGREGATION_TYPE_MAX`: Maximum value
* `AGGREGATION_TYPE_MIN`: Minimum value
* `AGGREGATION_TYPE_UNIQUE`: Single unique value (fails if multiple values exist)
## Time Windows
### Aggregation Window
The `aggregationWindow` defines how long after exposure to aggregate measurements:
```json theme={null}
"aggregationWindow": "604800s" // 7 days
```
Common windows:
* `86400s`: 1 day
* `604800s`: 7 days
* `2592000s`: 30 days
### Exposure Offset
The `exposureOffset` defines how long to wait after exposure before starting measurement:
```json theme={null}
"exposureOffset": "0s" // Start immediately
```
Example with delay:
```json theme={null}
"exposureOffset": "86400s" // Wait 1 day before measuring
```
## Variance Reduction (CUPED)
Enable or disable variance reduction to improve statistical power:
### Enabled (Default)
```json theme={null}
"varianceReductionConfig": {
"disabled": false
}
```
### Disabled
```json theme={null}
"varianceReductionConfig": {
"disabled": true
}
```
## Next Steps
After creating metrics:
* Configure experiments to track these metrics
* Analyze experiment results using these metrics
* Create additional metrics to measure different aspects of user behavior
# Configure Multiple Comparisons Adjustment
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/stats/configure-multiple-comparisons
Use decision rules in the Stats API to control false positive rates across multiple metrics.
When analyzing experiments with multiple metrics, adjustments control the overall false positive rate. The Stats API uses decision rules to determine how to adjust alpha and power levels for each metric.
## Decision Rules
The analysis plan contains a `decisionRule` object that specifies how metrics combine to form an overall shipping decision. The decision rule uses `AND` and `OR` operators between hypotheses.
### Structure
```json theme={null}
{
"decisionRule": {
"items": [
{ "hypothesis": "success-metric" },
{ "hypothesis": "guardrail-metric" }
],
"operator": "AND"
}
}
```
This example encodes: "Ship if success-metric is significant AND guardrail-metric is significant."
### How Adjustments Work
The multiple comparison adjustments control the false positive rate of the overall decision rule using:
* **Union-intersection testing** for OR conditions (at least one must be significant)
* **Intersection-union testing** for AND conditions (all must be significant)
The adjustment depends on each metric's role within the decision rule. Metrics joined by OR receive stricter alpha adjustments, while metrics joined by AND receive power adjustments.
## Default Behavior
If you don't specify a decision rule, the API applies a Bonferroni multiple testing correction. This divides the alpha level by the number of hypotheses.
```json theme={null}
{
"alpha": 0.05,
"power": 0.8,
"hypotheses": [
{ "id": "metric-1", "..." },
{ "id": "metric-2", "..." },
{ "id": "metric-3", "..." }
]
}
```
With three hypotheses and alpha = 0.05, each hypothesis uses alpha = 0.0167.
## Example: Success and Guardrail Metrics
A common pattern tests for improvement in a success metric while ensuring guardrails pass:
```json theme={null}
{
"alpha": 0.05,
"power": 0.8,
"hypotheses": [
{
"id": "consumption",
"type": {
"superiority": {
"preferredDirection": "INCREASE",
"minimumDetectableEffect": 0.03
}
},
"segments": [
{
"dimensions": {},
"params": {
"zTest": {}
}
}
]
},
{
"id": "crashes",
"type": {
"nonInferiority": {
"preferredDirection": "DECREASE",
"nonInferiorityMargin": 0.01
}
},
"segments": [
{
"dimensions": {},
"params": {
"gstZTest": {
"maxSampleSize": 1000
}
}
}
]
}
],
"decisionRule": {
"items": [
{ "hypothesis": "consumption" },
{ "hypothesis": "crashes" }
],
"operator": "AND"
}
}
```
This tests: "Ship if consumption improves AND crashes don't degrade beyond the margin."
## Related Resources
Learn about multiple comparison concepts
Set up your analysis plan
Understand test types
Understand decision rules
# Configure Sequential Testing
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/stats/configure-sequential-testing
Use sequential testing methods in the Stats API to analyze experiments during data collection.
Sequential tests allow you to analyze experiment results during data collection without invalidating statistical conclusions. The Stats API uses the group sequential test method (`gstZTest`) for sequential analysis.
## Fixed Horizon vs Sequential Tests
The Stats API supports two testing approaches:
| Method | Description | When to Use |
| ---------- | ----------------------- | -------------------------------------------------------------------- |
| `zTest` | Fixed horizon z-test | When you analyze results once at the end of the experiment |
| `gstZTest` | Group sequential z-test | When you want to analyze results continuously during data collection |
## Configure a Group Sequential Test
Group sequential tests provide valid statistical conclusions even when you analyze results multiple times during data collection. You need to specify the expected maximum sample size upfront.
In your analysis plan, specify `gstZTest` in the hypothesis segments:
```json theme={null}
{
"hypotheses": [
{
"id": "my-metric",
"type": {
"superiority": {
"preferredDirection": "INCREASE",
"minimumDetectableEffect": 0.03
}
},
"segments": [
{
"dimensions": {},
"params": {
"gstZTest": {
"maxSampleSize": 10000
}
}
}
]
}
]
}
```
The `maxSampleSize` parameter helps the test allocate the false positive rate optimally across analyses. Your estimate doesn't need to be exact, but a reasonable estimate improves power.
## Configure a Fixed Horizon Test
For metrics you only analyze once at the end of the experiment, use the standard `zTest`:
```json theme={null}
{
"hypotheses": [
{
"id": "my-metric",
"type": {
"superiority": {
"preferredDirection": "INCREASE",
"minimumDetectableEffect": 0.03
}
},
"segments": [
{
"dimensions": {},
"params": {
"zTest": {}
}
}
]
}
]
}
```
## Provide Time-Series Data
Sequential tests require time-series data to track how results evolve. Structure your data with `timeLabel` values containing cumulative statistics up to each time point:
```json theme={null}
{
"id": "my-metric",
"segments": [
{
"dimensions": {},
"groups": [
{
"group": "control",
"data": {
"gstZTest": {
"summary": {
"data": [
{
"timeLabel": "2024-01-01",
"value": {
"mean": 2.0,
"variance": 2.28,
"count": 100
}
},
{
"timeLabel": "2024-01-02",
"value": {
"mean": 1.99,
"variance": 2.29,
"count": 200
}
}
]
}
}
}
}
]
}
]
}
```
## Related Resources
Learn about sequential testing concepts
Complete analysis tutorial
Set up your analysis plan
Understand test types
# Configure Variance Reduction
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/stats/configure-variance-reduction
Use pre-exposure data in the Stats API to reduce variance and increase experiment precision.
Variance reduction uses pre-exposure data to reduce noise in your experiment metrics. This allows you to detect effects with fewer samples or detect smaller effects with the same sample size.
## How It Works
The Stats API applies regression adjustment to reduce variance. You provide pre-exposure measurements alongside your post-exposure data, and the API uses the correlation between them to produce more precise estimates.
## Configure Variance Reduction for Analysis
Include pre-exposure data in the `adjustment` field within your summary data:
```json theme={null}
{
"id": "my-metric",
"segments": [
{
"dimensions": {},
"groups": [
{
"group": "control",
"data": {
"zTest": {
"summary": {
"mean": 10.5,
"variance": 25.0,
"count": 1000,
"adjustment": {
"mean": 9.8,
"variance": 22.0,
"covariance": 15.2
}
}
}
}
},
{
"group": "treatment",
"data": {
"zTest": {
"summary": {
"mean": 11.2,
"variance": 26.0,
"count": 1000,
"adjustment": {
"mean": 9.9,
"variance": 21.5,
"covariance": 14.8
}
}
}
}
}
]
}
]
}
```
### Adjustment Fields
The `adjustment` object contains:
| Field | Description |
| ------------ | -------------------------------------------------------- |
| `mean` | Mean of the pre-exposure measurements |
| `variance` | Variance of the pre-exposure measurements |
| `covariance` | Covariance between pre-exposure and post-exposure values |
## Configure Variance Reduction for Power Analysis
For power analysis, include the expected covariance adjustment in your power data:
```json theme={null}
{
"hypotheses": [
{
"id": "my-metric",
"segments": [
{
"dimensions": {},
"powerData": {
"zTest": {
"baselineMean": 10.0,
"baselineVariance": 25.0,
"adjustment": {
"baselineVariance": 22.0,
"baselineCovariance": 15.0
}
}
}
}
]
}
]
}
```
### Power Analysis Adjustment Fields
| Field | Description |
| -------------------- | ------------------------------------------------------------- |
| `baselineVariance` | Variance of the pre-exposure covariate |
| `baselineCovariance` | Covariance between the pre-exposure covariate and the outcome |
## Interpret Results
The API returns both adjusted and unadjusted estimates. The treatment effect estimate benefits from the reduced variance, providing tighter confidence intervals. The response includes a `varianceReductionRate` field showing the achieved reduction.
## Related Resources
Learn about variance reduction concepts
Complete analysis tutorial
Calculate required sample sizes
# Create an Analysis Plan
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/stats/create-analysis-plan
Learn how to create an analysis plan for your experiment.
The first step when setting up the analysis plan is to decide on the risk management parameters alpha and power, which control the overall false positive and false negative rates of the analysis. For this example, use a false positive rate of `0.05` and a statistical power of `0.8`, implying a false negative rate of `1 - 0.8 = 0.2`.
With these parameters, the analysis plan looks like:
```json theme={null}
{
"alpha": 0.05,
"power": 0.8
}
```
## Create the Groups
In this analysis, you only have two groups: a `control` group and a `treatment` group. Both have the same number of samples on average, so set both of their weights to 1.
```json theme={null}
{
"groups": [
{
"id": "control",
"weight": 1
},
{
"id": "treatment",
"weight": 1
}
]
}
```
## Set Up the Comparisons
To compare the treatment group to the control group, use a `oneVsAll` comparison specification.
```json theme={null}
{
"comparisons": {
"oneVsAll": {
"baseline": "control"
}
}
}
```
## Set Up the Hypotheses
The hypotheses consist of two metrics: a crash rate metric and a consumption metric. Use a non-inferiority hypothesis for the crash rate metric to accept a slight increase. Set the margin to 1%. To avoid waiting until the end of the experiment to learn about increased crash rates, analyze the crash rate sequentially. Use a group sequential test to analyze the data sequentially.
For the consumption metric, use a superiority hypothesis with a minimum detectable effect of 3%. Anything less than 3% is not practically meaningful in this case, so you want to design the test for this effect size. You want to analyze the consumption at the end of the experiment, so select a regular z-test.
Add a single segment entry with an empty dimensions map to signal that the test has no segmentation. Your analysis plan at this stage is:
```json theme={null}
{
"hypotheses": [
{
"id": "crashes",
"type": {
"nonInferiority": {
"preferredDirection": "DECREASE",
"nonInferiorityMargin": 0.01
}
},
"segments": [
{
"dimensions": {},
"params": {
"gstZTest": {
"maxSampleSize": 1000
}
}
}
]
},
{
"id": "consumption",
"type": {
"superiority": {
"preferredDirection": "INCREASE",
"minimumDetectableEffect": 0.03
}
},
"segments": [
{
"dimensions": {},
"params": {
"zTest": {}
}
}
]
}
]
}
```
## Create the Decision Rule
Your decision rule is to ship the experiment if the guardrail is significantly non-inferior and the success metric is significantly superior. The decision rule is logically `crashes AND consumption`. You write it as:
```json theme={null}
{
"decisionRule": {
"items": [
{
"hypothesis": "consumption"
},
{
"hypothesis": "crashes"
}
],
"operator": "AND"
}
}
```
## Bring It All Together
Your analysis plan is now ready! The full plan is:
```json theme={null}
{
"alpha": 0.05,
"power": 0.8,
"groups": [
{
"id": "control",
"weight": 1
},
{
"id": "treatment",
"weight": 1
}
],
"comparisons": {
"oneVsAll": {
"baseline": "control"
}
},
"decisionRule": {
"items": [
{
"hypothesis": "consumption"
},
{
"hypothesis": "crashes"
}
],
"operator": "AND"
},
"hypotheses": [
{
"id": "consumption",
"type": {
"superiority": {
"preferredDirection": "INCREASE",
"minimumDetectableEffect": 0.03
}
},
"segments": [
{
"dimensions": {},
"params": {
"zTest": {}
}
}
]
},
{
"id": "crashes",
"type": {
"nonInferiority": {
"preferredDirection": "DECREASE",
"nonInferiorityMargin": 0.01
}
},
"segments": [
{
"dimensions": {},
"params": {
"gstZTest": {
"maxSampleSize": 1000
}
}
}
]
}
]
}
```
# Run an Analysis
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/stats/run-analysis
Learn how to analyze your experiment.
In this section, you go through the steps to run an analysis. You set up the request for the following example where you analyze an experiment with two metrics: a crash metric used as a guardrail that you analyze sequentially, and a consumption metric that you analyze once. To go through this tutorial you need to first [create an analysis plan](./create-analysis-plan).
## Run the Analysis
To run an analysis, send a POST request to the analysis endpoint with both the analysis plan and the data:
```bash theme={null}
curl -X POST "https://stats.confidence.dev/v1/analysis:run" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"plan": { ... },
"data": { ... }
}'
```
## Create the Analysis Data
When you've set up the analysis plan, you only have to add the data for analysis. The format must match what you specified in the analysis plan.
For your crash metric, you have two days of data. The data for your hypothesis looks like:
```json theme={null}
{
"id": "crashes",
"segments": [
{
"dimensions": {},
"groups": [
{
"group": "control",
"data": {
"gstZTest": {
"summary": {
"data": [
{
"timeLabel": "2020-01-01",
"value": {
"mean": 2,
"variance": 2.282828,
"count": 100
}
},
{
"timeLabel": "2020-01-02",
"value": {
"mean": 1.99,
"variance": 2.291357,
"count": 200
}
}
]
}
}
}
},
{
"group": "treatment",
"data": {
"gstZTest": {
"summary": {
"data": [
{
"timeLabel": "2020-01-01",
"value": {
"mean": 2.95,
"variance": 2.45202,
"count": 100
}
},
{
"timeLabel": "2020-01-02",
"value": {
"mean": 2.97,
"variance": 2.702613,
"count": 200
}
}
]
}
}
}
}
]
}
]
}
```
Your consumption data is for the whole period, and looks like:
```json theme={null}
{
"id": "consumption",
"segments": [
{
"dimensions": {},
"groups": [
{
"group": "control",
"data": {
"zTest": {
"summary": {
"mean": 0.25,
"variance": 0.1893939,
"count": 100
}
}
}
},
{
"group": "treatment",
"data": {
"zTest": {
"summary": {
"mean": 0.3133333,
"variance": 0.2165996,
"count": 150
}
}
}
}
]
}
]
}
```
This completes the data needed for your analysis. See the API reference for full request and response examples.
# Run a Power Analysis
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/stats/run-power-analysis
Learn how to run a power analysis.
In this section, you go through the steps to run a power analysis. You set up the request for the following example where you analyze an experiment with two metrics: a crash metric used as a guardrail that you analyze sequentially, and a consumption metric that you analyze once. To go through this tutorial you need to first [create an analysis plan](./create-analysis-plan).
## Run the Power Analysis
To run a power analysis, send a POST request to the power analysis endpoint with both the analysis plan and the power data:
```bash theme={null}
curl -X POST "https://stats.confidence.dev/v1/powerAnalysis:run" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"plan": { ... },
"data": { ... }
}'
```
## Data for the Power Analysis
To run the power analysis, you need to add power parameters to the analysis plan. For both the z-test and group sequential z-test, you need to include the mean and variance estimates.
Pass the following data for the consumption metric together with the analysis plan:
```json theme={null}
{
"hypotheses": [
{
"id": "consumption",
"segments": [
{
"dimensions": {},
"powerData": {
"zTest": {
"baselineMean": 0.3,
"baselineVariance": 0.63
}
},
"expected_sample_size": 1000
}
]
}
]
}
```
For the crash rate metric, include the following data:
```json theme={null}
{
"hypotheses": [
{
"id": "crashes",
"segments": [
{
"dimensions": {},
"powerData": {
"zTest": {
"baselineMean": 2.5,
"baselineVariance": 1.3
}
},
"expected_sample_size": 1000
}
]
}
]
}
```
This completes the data needed for the power analysis. See the API reference for full request and response examples.
# Add Metrics to a Surface
Source: https://confidence-auth-testing.mintlify.io/docs/api/how-to-guides/surfaces/add-metrics-to-surface
Learn how to add required and associated metrics to a surface using the API.
Add [metrics to a surface](/docs/surfaces/surface-settings#metrics) to make them easier to find for experimenters, or enforce that all experiments on the surface check this metric for regressions.
## Add a Required Metric
Confidence enforces required metrics for all experiments on the surface. Use the following to add a required metric `metrics/` to a surface with resource name `surfaces/`:
```bash theme={null}
curl -X PATCH "https://workflow.confidence.dev/v1/surfaces/?updateMask=metricConfig.mandatoryMetrics" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"metricConfig": {
"mandatoryMetrics": [{
"metric": "metrics/",
"guardrailMetric": {},
"preferredDirection": "INCREASE"
}]
}
}'
```
The `preferredDirection` can be `INCREASE` or `DECREASE`, indicating which direction the metric should move in for a successful experiment.
## Add an Associated Metric
Confidence suggests associated metrics for experiments on the surface but doesn't enforce them. Use the following to add an associated metric:
```bash theme={null}
curl -X PATCH "https://workflow.confidence.dev/v1/surfaces/?updateMask=metricConfig.associatedMetrics" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"metricConfig": {
"associatedMetrics": [{
"metric": "metrics/"
}]
}
}'
```
## Related Resources
Console guide for adding metrics
Deep dive into surface configuration
API reference for creating metrics
# IAM
Source: https://confidence-auth-testing.mintlify.io/docs/api/iam
Confidence IAM is a service for identity and access management.
Confidence IAM is a service for identity and access management of the users and API clients that are part of your Confidence organizations. Confidence IAM provides endpoints for setting roles and permissions of your users and creating API clients for directly accessing the APIs.
All use of the Confidence IAM API is free of charge.
## Use Confidence IAM
Learn how to fetch and list users.
Learn how to create API clients.
Learn about Confidence and third-party apps.
# Metrics Concepts
Source: https://confidence-auth-testing.mintlify.io/docs/api/metrics/concepts
Understand the key concepts in Confidence Metrics.
This section overviews Confidence Metrics key concepts, and should give you a broad understanding of which building blocks exists, and what they can do.
The metric subsystem centers around a `Metric`, which is a description of how to compute a metric. A metric is an aggregation over measurements performed on an `Entity`. The entity is typically a user but can also be something a user interacts with, like an ad. A `ScheduledMetricCalculation` performs the actual computation of the metrics by tying together the entity measurements from a `FactTable` with exposures from an `ExposureTable`. Confidence calculates exposures from a `AssignmentTable` that either Confidence Flags, the customer or some third party system provides. A `ScheduledExposureCalculation` computes exposure by periodically creating a job that takes the assignments and summarizes them into a first exposure for each entity. The rest of this section goes through each key concept in more detail, but still on a high level.
## Key Concepts
### Data Warehouse
The `Data Warehouse` is where you store your data and run your queries. Confidence connects to your data warehouse to run queries, but all the data is in your data warehouse. Confidence supports integration with BigQuery, Databricks, Snowflake and Redshift.
### Entity
An `Entity` is an abstract term for some part of your business that can be uniquely identified, measured and experimented on. The canonical example is a user, but it can also be something your customers interact with, like an ad. In Confidence, both measurements and assignments relate to an entity.
### Tables
Tables describe the structure of the data in your data warehouse so that Confidence can understand it. Three important tables in Confidence are: `AssignmentTable`, `ExposureTable` and `FactTable`.
The `AssignmentTable` points to a table in your data warehouse that stores information on when and which variant was assigned to one of your entities. Either Confidence Flags, your internal feature flagging, or some third party service provides this table. The only requirement is that you map out its columns in the `AssignmentTable`.
The `ExposureTable` points to a table that is an aggregation over the assignments table. It takes the assignments for each entity and determines the point of time of the first assignment of each variant. The time of the first assignment is called first exposure. In Confidence, the `ExposureTable` belongs to a single experiment, meaning that it filters out the assignments from the `AssignmentTable` that originates from the experiment before it aggregates them.
The `FactTable` represents a collection of measurements of an entity involved in some business process. It points to a table in your data warehouse that has the events relating to that business process. It could, for example, be the sales amount and checkout time of each user that placed an order on an ecommerce site. To define a fact table, you need the 1) the timestamp when the event occurred, 2) the entities involved in the event, and 3) a measurement of those entities.
### Metrics
A `Metric` is a description of how to aggregate a measurement across entities. In essence, it consists of a type, an aggregation and a time window. Two types of metrics exist in Confidence: *average* and *ratio* metrics. The average metric takes average of a measurement across entities, and a ratio separately aggregates a numerator and denominator and divides them. The metric aggregation determines how to aggregate values within an entity, typically, an entity has more than one event associated with them in the fact table. Finally, the time window specifies when to measure the metric relative to exposure. For example, sum all sales that occurred during the second week since first exposure. The metric is also computed on a set of dimensions like date, variant group and one or more dimensions of the entity itself.
### Schedules
The metric itself doesn't perform any calculation. A metric is only an abstract representation of how to compute it. A schedule defines from which data source and at what frequency to calculate a metric. Two schedules exist in Confidence: `ScheduledExposureCalculation` to calculate first exposures, and `ScheduledMetricCalculation` to calculate metrics. These schedules operate at a given, possibly variable, rate, and trigger `MetricCalculation`s and `ExposureCalculation`s that represent the calculation of the metric or exposure table within a specific time window.
# Entities Reference
Source: https://confidence-auth-testing.mintlify.io/docs/api/metrics/entities-reference
Technical reference for entities in Confidence Metrics.
This section provides technical specifications and reference information for entities.
For conceptual explanations of entities, see [Entities](./concepts#entity) in the Metrics Concepts page.
## Primary Key Types
Available primary key types for entities:
| Type | Description | Use Case |
| :-------------------- | :--------------------- | :---------------------------------------------- |
| `COLUMN_TYPE_STRING` | String identifiers | UUIDs, usernames, session IDs |
| `COLUMN_TYPE_INTEGER` | Integer identifiers | Autoincrementing user IDs, numeric keys |
| `COLUMN_TYPE_BOOLEAN` | Boolean values | Binary entity identifiers (rarely used) |
| `COLUMN_TYPE_DOUBLE` | Floating point numbers | Numeric identifiers with decimals (rarely used) |
## Entity Constraints
* **Unique names**: Entity display names must be unique within a project
* **Immutable key type**: You cannot change the primary key type after creation
* **Referenced entities**: You cannot delete entities used in fact tables, dimension tables, or metrics
* **Case sensitivity**: Entity names are case-sensitive
## Entity Resource Name Format
Reference entities using the resource name format:
```text theme={null}
entities/{entity_id}
```
Example: `entities/user`, `entities/session`
# Metrics Reference
Source: https://confidence-auth-testing.mintlify.io/docs/api/metrics/metrics-reference
Technical reference for metrics in Confidence Metrics.
This section provides technical specifications and reference information for metrics.
For conceptual explanations of metrics, see [Metrics](./concepts#metrics) in the Metrics Concepts page.
## Metric Types
### Average Metrics
Aggregate a single measurement across entities:
```json theme={null}
{
"typeSpec": {
"averageMetricSpec": {
"measurement": {
"name": "purchase_amount",
"repeated": false
},
"aggregation": {
"type": "AGGREGATION_TYPE_SUM"
}
}
}
}
```
### Ratio Metrics
Compute a ratio of two aggregated measurements:
```json theme={null}
{
"typeSpec": {
"ratioMetricSpec": {
"numerator": {
"measurement": {
"name": "converted",
"repeated": false
},
"aggregation": {
"type": "AGGREGATION_TYPE_SUM"
}
},
"denominator": {
"measurement": {
"name": "eligible",
"repeated": false
},
"aggregation": {
"type": "AGGREGATION_TYPE_SUM"
}
}
}
}
}
```
## Aggregation Types
| Type | Description | Use Case |
| :-------------------------------- | :-------------------------------------- | :--------------------------------------- |
| `AGGREGATION_TYPE_SUM` | Sum all values | Total revenue, total conversions |
| `AGGREGATION_TYPE_COUNT` | Count all occurrences | Number of events, session count |
| `AGGREGATION_TYPE_COUNT_DISTINCT` | Count unique values | Unique items viewed, distinct categories |
| `AGGREGATION_TYPE_MAX` | Maximum value | Highest price, longest session |
| `AGGREGATION_TYPE_MIN` | Minimum value | Lowest price, shortest session |
| `AGGREGATION_TYPE_UNIQUE` | Single unique value (fails if multiple) | Latest status, final state |
## Time Windows
### Aggregation Window
Duration after exposure to aggregate measurements:
```json theme={null}
"aggregationWindow": "604800s" // 7 days in seconds
```
Common windows:
* `86400s`: 1 day
* `604800s`: 7 days (1 week)
* `1209600s`: 14 days (2 weeks)
* `2592000s`: 30 days (\~1 month)
### Exposure Offset
Delay before starting measurement:
```json theme={null}
"exposureOffset": "0s" // Start immediately
```
Use cases:
* `0s`: Immediate effect (UI changes, performance)
* `86400s`: 1-day delay (email campaigns)
* `604800s`: 1-week delay (long-term behavior)
## Variance Reduction (CUPED)
Variance reduction improves statistical power by reducing noise in metric estimates:
```json theme={null}
{
"varianceReductionConfig": {
"disabled": false
}
}
```
Recommended for most metrics. Requires pre-exposure data.
```json theme={null}
{
"varianceReductionConfig": {
"disabled": true
}
}
```
Use when:
* No pre-exposure data available
* Metric is new user only
* Testing CUPED impact
## Metric Resource Name Format
Reference metrics using the resource name format:
```text theme={null}
metrics/{metric_id}
```
Example: `metrics/conversion-rate`, `metrics/average-revenue`
## Required Fields
| Field | Type | Description |
| :------------------ | :----- | :-------------------------------------------------- |
| `displayName` | string | Human-readable metric name |
| `entity` | string | Entity resource name (for example, `entities/user`) |
| `factTable` | string | Fact table resource name |
| `aggregationWindow` | string | Duration in seconds (ISO 8601) |
| `exposureOffset` | string | Duration in seconds (ISO 8601) |
| `typeSpec` | object | Average or ratio metric specification |
## Best Practices
### Choose Metric Types
* **Average metrics**: Use for continuous values (revenue, duration, ratings)
* **Ratio metrics**: Use for rates and percentages (conversion rate, CTR, success rate)
### Aggregation Selection
* **SUM**: Default for most measurements (revenue, clicks, conversions)
* **COUNT**: When you only care about occurrence, not value
* **COUNT\_DISTINCT**: When each unique value matters (unique products, categories)
* **MAX/MIN**: When extremes are important (peak load, worst performance)
### Time Window Considerations
* **Shorter windows** (1-7 days): Faster results, earlier decisions
* **Longer windows** (14-30 days): Capture long-term effects, slower to significance
* **Match business cycle**: Align with purchase cycles, subscription periods
### Variance Reduction
* Enable CUPED for most metrics to improve sensitivity
* Disable for new user metrics or when no historical data exists
* Monitor CUPED effectiveness in analysis results
# Tables Reference
Source: https://confidence-auth-testing.mintlify.io/docs/api/metrics/tables-reference
Technical reference for tables in Confidence Metrics.
This section provides technical specifications and reference information for assignment tables, fact tables, and dimension tables.
For conceptual explanations of tables, see [Tables](./concepts#tables) in the Metrics Concepts page.
## Column Types
All tables use these column type specifications:
| Type | Description | Example Use |
| :---------------------- | :---------------- | :------------------------------------- |
| `COLUMN_TYPE_STRING` | Text values | User IDs, country codes, product names |
| `COLUMN_TYPE_INTEGER` | Whole numbers | Count values, age, quantity |
| `COLUMN_TYPE_DOUBLE` | Decimal numbers | Revenue, ratings, percentages |
| `COLUMN_TYPE_BOOLEAN` | True/false values | Conversion flags, feature enabled |
| `COLUMN_TYPE_TIMESTAMP` | Date and time | Event timestamps, created dates |
| `COLUMN_TYPE_DATE` | Date only | Birth dates, enrollment dates |
## Assignment Tables
### Required Fields
| Field | Type | Description |
| :-------------------- | :------------ | :-------------------------------- |
| `timestampColumn` | Column | When the assignment occurred |
| `entityColumnMapping` | ColumnMapping | Which entity was assigned |
| `exposureKeyColumn` | Column | Identifies the experiment/feature |
| `variantKeyColumn` | Column | Which variant was assigned |
### Data Delivery Strategies
Continuously updates data based on incremental checks:
```json theme={null}
{
"strategy": "AUTOMATIC",
"automaticUpdateConfig": {
"incrementDuration": "P13H",
"commitDelay": "P13H"
}
}
```
* **incrementDuration**: How frequently to check for new data (ISO 8601 duration)
* **commitDelay**: Buffer time before considering data complete
Updates once per day at a scheduled time:
```json theme={null}
{
"strategy": "DAILY",
"dailyUpdateConfig": {}
}
```
## Fact Tables
### Required Fields
| Field | Type | Description |
| :---------------- | :--------------- | :------------------------------------------ |
| `name` | string | Unique identifier for the table |
| `displayName` | string | Human-readable name |
| `sql` | string | SQL query selecting fact rows |
| `timestampColumn` | Column | When the measurement occurred |
| `entities` | ColumnMapping\[] | Entity columns (at least one required) |
| `measurements` | Column\[] | Measurement columns (at least one required) |
### Column Specifications
Each column requires:
* `name`: Column name in query results
* `type`: Column type (from table above)
* `repeated`: Boolean indicating if column contains arrays
### Table States
| State | Description |
| :--------- | :-------------------------------------- |
| `CREATING` | Initial validation in progress |
| `ACTIVE` | Ready for use in metrics |
| `FAILED` | Validation failed, check error messages |
## Dimension Tables
### Required Fields
| Field | Type | Description |
| :-------------------- | :--------------- | :---------------------------------------- |
| `displayName` | string | Human-readable name |
| `sql` | string | SQL query selecting dimension rows |
| `timestampColumn` | Column | When the dimension value applies |
| `entityColumnMapping` | ColumnMapping\[] | Entity columns (at least one required) |
| `dimensions` | Column\[] | Dimension columns (at least one required) |
### Dimension Column Types
Dimensions typically use:
* **STRING**: Categorical values (country, platform, tier)
* **BOOLEAN**: Binary attributes (`is_premium`, `is_active`)
* **INTEGER**: Numeric categories (`age_group_code`, `tier_level`)
### Table States
Same as fact tables: `CREATING`, `ACTIVE`, `FAILED`
## Column Mapping Structure
Column mappings connect query results to entities:
```json theme={null}
{
"column": {
"name": "user_id",
"type": "COLUMN_TYPE_STRING",
"repeated": false
},
"entity": "entities/user"
}
```
## Best Practices
### SQL Queries
* **Use SELECT \* sparingly**: Explicitly specify columns for better performance
* **Add WHERE clauses**: Filter data at query time to reduce processing
* **Use partitioning**: Leverage data warehouse partitioning for efficiency
* **Test queries**: Validate SQL in your data warehouse before creating tables
### Data Quality
* **Validate timestamps**: Ensure timestamp columns are in the correct format
* **Handle NULLs**: Decide how to handle NULL values in measurements
* **Check data types**: Verify column types match your schema expectations
* **Monitor failures**: Review failed tables and fix SQL or schema issues
# Get Started with Confidence APIs
Source: https://confidence-auth-testing.mintlify.io/docs/api/quickstart
This tutorial helps you make your first Confidence API call by retrieving a list of existing feature flags.
The steps to take are the following:
1. Create an API client if you haven't done so
2. Request an access token
3. Use the access token to request a list of flags
## Before You Begin
* This tutorial assumes you have a Confidence account.
* Use cURL to make API calls. You can install cURL using the
package manager of your choice.
## Create an API Client
An API client provides a client ID and client secret needed to request an
access token by implementing any of the authorization flows.
On the bottom of the left sidebar, select **Admin > API Clients**.
Enter `API client` as the **Display name**.
## Request an Access Token
The access token is a string which has the credentials and permissions
that you can use to access a given resource (for example, flags, events, and metrics).
To request the access token you first need your client ID and client secret:
On the bottom of the left sidebar, select **Admin > API Clients**.
Select the API client you just created. You find the client ID immediately on the page. The client secret is available behind the **View client secret** link.
With the credentials in hand, you are ready to request an access token. This
tutorial uses the client credentials flow, so you must:
Set to the `application/json` value.
Include the `clientId` and `clientSecret`, along with the `grantType` parameter set to `client_credentials`.
```bash Request theme={null}
curl -X POST "https://iam.confidence.dev/v1/oauth/token" \
-H "Content-Type: application/json" \
-d '{
"grantType": "client_credentials",
"clientId": "clientId",
"clientSecret": "clientSecret"
}'
```
The response returns an access token valid for 1 hour:
```json Response theme={null}
{
"accessToken": "eyJraWQiOi...zWyvwvtz_oEU1x",
"expiresIn": "86400"
}
```
## Example: Request a List of Flags
For this example, use the list flags endpoint in the Flags API to
request information about existing feature flags.
Your API call must include the access token you have just generated using the
`Authorization` header as follows:
```bash Request theme={null}
curl -X GET "https://flags.confidence.dev/v1/flags" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
The API returns the following JSON response:
```json Response theme={null}
{
"flags": [...]
}
```
Congratulations! You made your first API call to Confidence. Since you probably
have not created any flags yet, the response is empty.
## Summary
The Confidence platform provides different APIs depending on the functionality
you want to access. The API calls must include the Authorization header along
with a valid access token.
This tutorial makes use of the client credentials grant type to retrieve the
access token. That works fine in scenarios where you control the API call to
Confidence, for example where your backend is connecting to the API. It does
not work in cases where your app connects on behalf of a specific user, for
example when getting private playlist or profile data.
## What's Next?
* The tutorial used the Flags API to retrieve a list of flags. An interesting
exercise would be to extend the example by trying to create a flag. Do you
accept the challenge?
* The [API basics](./api-basics) page provides more information on how
to work with Confidence platform APIs.
# Analysis Reference
Source: https://confidence-auth-testing.mintlify.io/docs/api/stats/analysis-reference
Technical reference for statistical analysis in Confidence Stats.
This section provides technical specifications and reference information for analysis plans and statistical testing.
For conceptual explanations of analysis, see [Stats Concepts](./concepts).
## Comparison Specifications
Define how to compare groups in an analysis:
### All to Baseline
Compare all treatment groups to a designated control:
```json theme={null}
{
"comparisonSpec": {
"allToBaseline": {
"baseline": "control"
}
}
}
```
**Use when:** Standard A/B test with one control and multiple treatments
### All Pairs
Compare every group to every other group:
```json theme={null}
{
"comparisonSpec": {
"allPairs": {}
}
}
```
**Use when:** exploring all possible differences, no clear control group
### Specific Pairs
Define exactly which groups to compare:
```json theme={null}
{
"comparisonSpec": {
"pairs": [
{
"baseline": "control",
"treatment": "variant_a"
},
{
"baseline": "control",
"treatment": "variant_b"
}
]
}
}
```
**Use when:** complex designs with specific comparisons of interest
## Hypothesis Types
### Superiority Hypothesis
Test if a treatment improves a metric by a meaningful amount:
```json theme={null}
{
"superiority": {
"preferredDirection": "INCREASE",
"minimumDetectableEffect": 0.03
}
}
```
**Fields:**
* `preferredDirection`: `INCREASE` or `DECREASE`
* `minimumDetectableEffect`: Relative change considered meaningful (for example, 0.03 = 3%)
**Use for:** success metrics, primary outcomes
### Non-Inferiority Hypothesis
Test if a treatment doesn't harm a metric beyond an acceptable margin:
```json theme={null}
{
"nonInferiority": {
"preferredDirection": "INCREASE",
"nonInferiorityMargin": 0.01
}
}
```
**Fields:**
* `preferredDirection`: `INCREASE` or `DECREASE`
* `nonInferiorityMargin`: Maximum acceptable degradation (for example, 0.01 = 1%)
**Use for:** guardrail metrics, cost metrics, performance metrics
## Preferred Direction
| Value | Meaning | Example Metrics |
| :--------- | :--------------- | :----------------------------------- |
| `INCREASE` | Higher is better | Revenue, conversion rate, engagement |
| `DECREASE` | Lower is better | Load time, error rate, bounce rate |
## Decision Rules
Combine multiple hypotheses into a single decision:
### AND Rule
All hypotheses must be significant:
```json theme={null}
{
"operator": "AND",
"items": ["metric1", "metric2", "metric3"]
}
```
### OR Rule
At least one hypothesis must be significant:
```json theme={null}
{
"operator": "OR",
"items": ["metric1", "metric2", "metric3"]
}
```
### Complex Rule
Combine AND/OR logic:
```json theme={null}
{
"operator": "AND",
"items": [
{
"rule": {
"operator": "AND",
"items": ["guardrail1", "guardrail2"]
}
},
{
"rule": {
"operator": "OR",
"items": ["success1", "success2", "success3"]
}
}
]
}
```
**Translates to**: `(guardrail1 AND guardrail2) AND (success1 OR success2 OR success3)`
## Group Structure
Define groups with allocation weights:
```json theme={null}
{
"groups": [
{
"id": "control",
"weight": 1
},
{
"id": "treatment",
"weight": 1
}
]
}
```
**Fields:**
* `id`: Unique identifier for the group
* `weight`: Relative allocation (typically proportional to traffic split)
**Common patterns:**
* Equal split: All weights = 1
* 50/25/25: Weights = 2, 1, 1
* 90/10: Weights = 9, 1
## Statistical Parameters
### Significance Level (Alpha)
Probability of false positive:
```json theme={null}
"alpha": 0.05 // 5% false positive rate
```
Common values:
* `0.05`: Standard significance level
* `0.01`: Stricter threshold
* `0.10`: More lenient threshold
### Statistical Power
Probability of detecting a true effect:
```json theme={null}
"power": 0.80 // 80% power
```
Common values:
* `0.80`: Standard power level
* `0.90`: Higher power (larger sample needed)
* `0.70`: Lower power (smaller sample enough)
## Data Types
### Binary Data
For conversion-like metrics:
```json theme={null}
{
"binaryData": {
"successes": [100, 110],
"trials": [1000, 1000]
}
}
```
**Use for:** conversion rates, click-through rates, success/failure outcomes
### Continuous Data
For numeric measurements:
```json theme={null}
{
"continuousData": {
"means": [42.5, 43.2],
"variances": [12.3, 11.8],
"counts": [1000, 1000]
}
}
```
**Use for:** revenue, duration, ratings, counts
## Analysis Methods
Different methods have different assumptions and use cases:
| Method | Sequential | Data Type | Use Case |
| :------------ | :--------- | :-------- | :-------------------------------------- |
| Fixed horizon | No | Both | Final analysis only |
| Sequential | Yes | Both | Continuous monitoring |
| Bayesian | Yes | Both | Continuous updates with prior knowledge |
### Method Assumptions
All methods assume:
* **Random assignment**: Users randomly assigned to groups
* **Independence**: User outcomes are independent
* **Stable variance**: Variance doesn't change over time
* **No spillover**: Treatment doesn't affect control group
Sequential methods additionally assume:
* **Data arrives continuously**: New data added over time
* **Stopping rules followed**: Don't peek without accounting for it
## Best Practices
### Hypothesis Design
* Set MDE/NIM based on business impact, not statistical convenience
* Use superiority for metrics you want to improve
* Use non-inferiority for metrics you want to protect
* Define hypotheses before looking at data
### Decision Rules
* Require all guardrails to pass (use AND)
* Allow any success metric to trigger (use OR)
* Be explicit about what defines success
* Consider multiple testing adjustments
### Power Analysis
* Run power analysis before experiment
* Ensure adequate sample size for MDE
* Consider seasonal effects on sample collection
* Account for multiple comparisons in power calculation
# Stats Concepts
Source: https://confidence-auth-testing.mintlify.io/docs/api/stats/concepts
Understand the key concepts in Confidence Stats.
This section covers the key concepts in Confidence Stats. The primitives defined by the stats service allow you to both plan and analyze experiments.
An analysis consists of an `AnalysisPlan` which is a planning phase description of how you plan to analyze the data, and `AnalysisData` which is the data that you've collected so far.
## Analysis Plan
An `AnalysisPlan` is a description of how to compare a set of groups (for example, treatments) to each other, a set of hypotheses about these groups, and a decision rule that encodes how you plan to make a decision on the combined set of results from all hypotheses. The API uses `AnalysisPlan` in the planning phase to calculate power, and in the analysis phase to decide how to analyze the data.
## Comparisons and Groups
A `Group` is a unique identifier and an integer weight. The weight is proportional to the group allocation. In an analysis, the `ComparisonSpec` defines how to compare the groups to each other. It has three options: 1) compare all groups to the group designated as baseline, 2) compare all pairs of groups to each other, or 3) compare all listed group pairs. The most common setup is to compare all groups to the control group.
## Hypothesis and Method
A `Hypothesis` represents a testable belief about a metric. For example, the change *A* leads to an increase by *X%* in metric *M*. In more technical terms, a hypothesis is a belief about a specific parameter in a statistical model. You test your hypothesis by constructing a model of the data, estimating its parameters and evaluating whether the estimated parameters are consistent with the hypothesis. The analysis method defines these steps. Depending on how your data arises and your experiment design, different analysis methods are appropriate. For example, if you collect new data every day, a sequential analysis is appropriate, whereas a non-sequential method would result in an increased number of false positives. You should carefully select which method to apply. All methods come with a set of assumptions. If some of these assumptions are not satisfied, you shouldn't trust the results.
Two common categories of hypotheses are superiority and non-inferiority hypotheses. A superiority hypothesis states that a metric changes in a given direction by some practically meaningful amount set by the experimenter, called the minimum detectable effect (MDE). A non-inferiority hypothesis states that a metric doesn't change more, in a given direction, than some acceptable margin, called the non-inferiority margin (NIM).
You define a superiority hypothesis for a relative increase of `3%` as:
```json theme={null}
{
"superiority": {
"preferred_direction": "INCREASE",
"minimum_detectable_effect": 0.03
}
}
```
You define a non-inferiority hypothesis for a situation where you could accept a relative decrease of `1%` but not more as:
```json theme={null}
{
"non_inferiority": {
"preferred_direction": "INCREASE",
"non_inferiority_margin": 0.01
}
}
```
## Decision Rule
A decision rule is a logical expression of the significance of a set of hypotheses that determines when you view an analysis as a success. A typical example would be that at least one success metric is significant, while all guardrails are significantly non-inferior. You encode that rule as:
```json theme={null}
{
"operator": "AND",
"items": [
{
"rule": {
"operator": "AND",
"items": ["guardrail1", "guardrail2"]
}
},
{
"rule": {
"operator": "OR",
"items": ["success1", "success2"]
}
}
]
}
```
That translates to the decision rule `(guardrail1 AND guardrail2) AND (success1 OR success2)`.
The decision rule requires you to be explicit about the decision that you aim to take. The rule is the basis for adjusting both the false positive rate and statistical power per hypothesis so that you get the desired overall false positive rate and statistical power.
## Analysis Data
The data passed to the analysis differs depending on the method used for analysis. The data also has a certain type that may require a change in the method used for analysis, the two most common data types are binary and continuous. See more in [Run an Analysis](/docs/api/how-to-guides/stats/run-analysis).
# Confidence as a Platform (APIs)
Source: https://confidence-auth-testing.mintlify.io/docs/confidence-as-a-platform
Use Confidence as a true platform and build what you need on top of it.
At the foundational level, Confidence provides a set of independent APIs for
performing experimentation-related tasks. These APIs are modular and extensible,
allowing you to build your own experimentation platform on top of Confidence.
The APIs write events, assignments, and metrics to a data warehouse. You can
connect your existing warehouse (warehouse-native), or use a managed warehouse
that Confidence provides. With a managed warehouse, Confidence takes care of
storage for you.
The main APIs are:
* **Flags**: Control experiences using feature flags.
* **Events**: Collect data from your users and store it in a data warehouse.
* **Metrics**: Define and compute metrics.
* **Stats**: Perform statistical testing and analysis.
* **Workflows**: Implement workflows to orchestrate experimentation designs.
## Highlights of the Confidence APIs
* **Extensible**: Confidence is by design extensible. You can build your
own experimentation platform on top of Confidence by using the APIs. Leverage
the parts of Confidence that makes sense for you, and build the rest yourself.
* **Warehouse-centric**: Use your existing warehouse, or a managed warehouse
that Confidence provides and operates. Warehouse-native keeps data in your
infrastructure. A managed warehouse lets Confidence take care of storage for you.
* **Usage-based pricing**: You only pay for what you use. The price you pay depends on what
features you use and how much you consume. You can start small and scale up as
your experimentation program grows, without having to worry about upfront
costs.
* **Built for scale**: Confidence supports large-scale experimentation
programs, helping you with coordination and planning of experiments across teams
and products. Use it with hundreds of teams to run thousands of experiments.
## Learn More
Read more about the independent APIs that allow you to integrate Confidence with your existing tooling and infrastructure:
You can follow the [API quickstart](../docs/api/quickstart) tutorial to learn how to
make your first API call. Read more about how the APIs work in the [API basics](../docs/api/api-basics)
section.
# Warehouse-Native Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/data-warehouse-native
Warehouse-Native Confidence is a modern experimentation platform that runs on top of your warehouse.
Data integrity is a core value of Confidence. Warehouse-Native Confidence runs on top of your data warehouse, so
you can be confident that your data is safe, right, and up to date. You have full transparency
into what Confidence does with your data, and you can audit and reproduce calculations.
## How Confidence Interacts With Your Data Warehouse
Depending on the functionality you want to use, Confidence needs access to read from and write to
your data warehouse.
Confidence needs write access to write:
* general events that you log using the events service
* assignments when you resolve flags
* exposure tables calculated from assignment events
Confidence needs read access to read:
* assignments to calculate exposure
* data you want to use to calculate metrics
The figure below gives a high-level overview of how the events, metrics, and flags services in
Confidence read and write to your data warehouse.
You can use only a subset of these services.
In that case, only the relevant parts of the diagram apply.
## Queries in Your Data Warehouse
To calculate exposure and metrics, Confidence runs queries in your data warehouse.
You can read more on the [metrics page](./metrics/introduction#how-confidence-computes-metrics).
### Permissions
You can grant Confidence access only to the relevant tables in your data warehouse.
You don't need to grant global access to your entire data warehouse.
### Data Cache
Confidence caches aggregated results of metrics in its database to improve performance.
The cache only includes the aggregated results of metrics, not the underlying data.
For basic metrics, this means that Confidence stores the daily count, mean and variance for each
treatment group.
## Transient Data in Confidence Flags
With Confidence Flags and the managed resolver, data temporarily passes through Confidence.
The information that Confidence persists is:
* The number of resolves and the timestamp of the last resolve.
* The number of applies and the timestamp of the last apply.
* The names of the fields available in the context. The values of the fields in the context are
never stored.
Read more on the [data transfer page](./flags/introduction#data-transfer).
# Analyze Results
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/analyze-results
Understand what your results mean and what to do next.
Confidence provides metric results by comparing the treatments using
hypothesis tests to see if the differences are statistically
significant. The exact nature of the tests vary depending
on the type and role of the metrics. Ultimately, Confidence gives an
overall shipping recommendation that summarizes the multidimensional
results to one single recommendation.
## Spotlight
For both running and ended experiments, you need to make a decision—whether a
tested feature was good enough to reach the full market or if an ongoing
experiment should continue or if you should stop it. When deciding what to do,
you should always take a step back and consider all the pros and cons of the
decisions you are making. Involve people with different roles in these
decisions.
To help you in deciding what to do, Confidence provides a recommendation in the **Spotlight**
section on the **Result** tab. The recommendation summarizes the outcomes for the multiple metrics
used in the experiment. For an experiment that is live and configured to display results
continuously, the possible recommendations are:
* **Ship**. Confidence recommends to ship the change if:
* at least one success metric has evidence of improvement
* all guardrail metrics meet their tolerance levels if you use a non-inferiority margin; if not,
they should show no evidence of deterioration
* no evidence of a deterioration in any metric or of a sample ratio mismatch In this case, there
is conclusive evidence that the change you are testing improves at least one metric without
doing so while guardrails are acceptable.
* **Continue**. If there's no evidence that you should ship, Confidence recommends to continue the
experiment as long as there are no signs of deterioration or sample ratio mismatch.
* **End**. Confidence recommends to end the experiment if you use success metrics with minimum
detectable effects and all success metrics have reached powered but none is significant.
* **Abort**. Confidence recommends to stop the experiment if there is evidence of
deterioration or a sample ratio mismatch.
When you end an experiment, the recommendations focus on what to do next. The **Continue**, **End**
and **Abort** recommendations change into a **Don't ship** recommendation, as there is no evidence
of an improvement that would suggest shipping.
For ended experiments, the Spotlight section includes an **Explore** option. Click it to create an
[Exploration](./exploration) directly from the Spotlight recommendation. This lets you dig deeper
into the results that informed the recommendation.
## Health Checks
Confidence provides health checks to help you understand the quality of your experiment.
### Incoming Traffic
The incoming traffic health check verifies that your experiment receives traffic. The check confirms
that the flag rule your experiment controls receives resolves from clients, that these resolves are
also applied, and that all groups in the experiment have exposure calculated.
### Balanced Traffic
The balanced traffic health check verifies that the proportion of exposure attributed to each group
follows the allocation that you set up for the experiment. If there is an imbalance, the results are
not reliable. This check uses what is commonly referred to as a sample ratio mismatch test.
### No Metric Deterioration
The no metric deterioration health check verifies that the metrics you track,
including both success and guardrail metrics, do not show any evidence of
deterioration. If a metric deteriorates, you have a clear sign that the
treatment isn't working as intended.
## Metrics
Confidence presents the results for individual metrics in various ways to help you
learn as much as possible from your tests.
### Significance
Significant means that if there is no effect, then it's unlikely that the
observed result is accidental due to the natural variation in the data. Alpha
specifies the threshold for significance, and thus the expected rate of false
positives. Default alpha for Confidence is 5%, which is further adjusted to
account for [multiple comparisons](/docs/experiments/statistical-settings).
You should interpret significance differently for success and guardrail metrics:
| | Significant | Not significant |
| :----------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------- |
| Success metric | You have statistical evidence for a change/increase/decrease due to the treatment. | You lack statistical evidence for a change/increase/decrease due to the treatment. |
| Guardrail metric (with NIM) | You have statistical evidence that the metric has not increased more than/not decreased more than the specified non-inferior margin (NIM). | You lack statistical evidence that the metric has not increased more than/not decreased more than NIM. |
| Guardrail metric (without NIM) | You have statistical evidence that the metric deteriorates. | You lack statistical evidence that the metric deteriorates. |
### Results Estimates and Confidence Intervals
The results for the comparisons between treatment and control give a point
estimate and a confidence interval. The point estimate always lies in the
middle of the confidence interval. The estimated effect of the treatment is
uncertain, and the confidence interval describes the degree of uncertainty. If
you would repeat the experiment 100 times, the confidence interval should
cover the true effect of the treatment in 95% of the experiments if
alpha is 5%. With the same alpha, a given experiment's
confidence interval covers the true effect of the treatment with 95%
confidence.
Confidence displays the point estimate and confidence interval on the *relative* scale. The effects
are always reported as a % change relative to the control group. This makes it easier to
compare and visualize effect sizes across metrics.
### Result Visualization
You can view the difference between treatments and control on the **Result** tab
of the experiment.
For each comparison between a treatment and control, you see the results visualized using a
confidence interval. If you're analyzing the results sequentially, you can click the expand icon to
see a timeline graph.
### Detailed Results
If you click **Detailed results**, you can see more details about the analysis of the metrics
in your experiment. Here you can find the following information:
* **Powered effect** shows the effect size that you have the power to detect with the
current sample size. For example, if you set the power for the experiment to 80%, then a powered
effect of 10% means that based on the users that have been exposed to the experiment so far,
you have a 80% power to detect a 10% effect size. Note here that
the 10% is *relative* to the control group.
* **Sample size** the number of exposed users in each group.
* **Time** the last time point Confidence analyzes the metric.
* **Adjusted alpha** the multiple testing corrected alpha for the metric.
* **Adjusted power** the decision rule corrected power for the metric.
* **Variance reduction** the percentage of variance that pre-exposure data was able to reduce
(Confidence displays `N/A` if you disable variance reduction).
### Learn More
To learn more about your results, use an
[Exploration](./exploration). Here you can add more metrics and split the results
by various dimensions.
You can also click **Detailed results** to get more details about the metrics in
your experiment. Choose between different types of visualizations, and add more
columns to see more details.
## Record Your Decision
After an experiment ends, you should record the outcome and reasoning. This creates an
institutional record of experiment learnings that your team can reference later.
On the **Result** tab of an ended experiment, the **Decision** section appears at the top of the
page. It contains:
* **Outcome**—a dropdown list where you select what you decided to do based on the results (for
example, ship or don't ship). By default, this shows "Not selected".
* **Conclusion**—a text field where you write a brief summary of the decision and the reasoning
behind it.
## Roll Out a Successful Variant
When your results show a clear winner, you can roll out the winning variant directly from the experiment. This option is available for A/B tests with two treatments (control and one treatment variant). Click **Roll out** in the actions section of a live A/B test to convert it to a [rollout](./workflows/rollouts). This distributes the winning variant to all users without manually configuring the flag.
The rollout preserves the experiment's metrics and configuration so you can continue to monitor the impact as you scale up.
## Related Resources
Deep dive into results analysis
Understand the statistics
Step-by-step exploration guide
# Audience
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/audience
The audience is the set of users or other units that you want to make eligible for an experiment.
An audience definition uses:
* inclusion criteria that decide what units to include
* allocation that sets the percentage of the included population to run the experiment on
* a targeting key that specifies the field in the evaluation context to randomize traffic based on
* exclusivity to control the behavior of the experiment in relation to other
experiments, rules, and segments
## Inclusion Criteria
Inclusion criteria define which units are eligible for the experiment. You can combine multiple criteria types:
* **Attributes**: Target users based on evaluation context fields like country, platform, or version.
* **Segments**: Target users who belong to a pre-defined [segment](/docs/flags/segments), letting you reuse the same audience across experiments.
* **Holdbacks**: Include or exclude a random subset of users defined by a [holdback](/docs/surfaces/surface-settings#holdbacks) on a surface.
* **Groups**: Combine multiple criteria with `AND` or `OR` operators to build complex targeting logic.
Learn how to configure these criteria in the [Define the Audience](/docs/how-to-guides/define-audience-criteria) guide.
For more details about allocation, randomization, and the targeting key, go to the [audience](/docs/flags/audience) page for flags.
## Randomization
Confidence uses randomization to assign variants to users. To randomize,
Confidence needs to know which field in the evaluation context it should take
the value from. Read more in the [flag documentation](/docs/flags/audience#randomization).
## Sticky Assignment
Under **advanced options** in the **audience section** you can enable sticky
assignments. When you enable sticky assignments, Confidence writes all
assignments to a storage that is accessible at resolve time with low latency.
Read more about sticky assignment in the [flags documentation](/docs/flags/audience#sticky-assignments).
## Related Resources
Deep dive into audience configuration
Configure targeting criteria
Set up mutual exclusion
Configure experiment variants
# Comments
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/comments
With comments you can work together on your experiment with team mates. Add comments to the experiment design or its results.
Get pinged on Slack when someone mentions you or replies to your comment by [connecting your personal Slack account with
Confidence](/docs/how-to-guides/connect-personal-slack).
## Overview
Comments and reviews are a great way to work together on your experiment with teammates.
You can comment on "comment zones" on the experiment pages or by submitting a review. To see which comment zones are available, click the comment icon in the top right and you see the zones highlight for a brief second. As you hover over comment zones on the page, they highlight and you can click to add a comment. To comment by submitting a review, click the thumbs up and down icon on the sidebar.
If a comment zone already has a comment, it has a badge with the number of threads on it. Clicking on the badge opens the comment sidebar and selects the thread so that you can add a comment to it.
## Add a Comment
To add a comment to the experiment, click the comment icon in the top right corner of the experiment design.
To add a comment:
1. Go to a workflow such as an A/B test or rollout.
2. Click on the comment icon next to the experiment name. The comment sidebar opens on the right side of the page.
3. Click on a comment zone on the workflow where you want to add a comment. The zones flash briefly when you open the sidebar to indicate where you can add comments.
4. Type your comment and press Shift + Enter to post.
Mention a team mate in a comment by typing @
and selecting their name from the list. A mentioned team mate can be assigned to the thread. Assigned threads shows up on the user's home page under the "Your to-do list" section.
## Comment on a Thread
Threads organize comments for you. A thread is a group of related comments. You can add a comment to a thread by selecting it, which reveals an input field where you can type your comment.
## Resolve Threads
You can resolve a thread by clicking the checkmark icon in the top right corner of the thread. Resolved threads are by default hidden. You can show resolved threads by toggling the `Show resolved threads` switch in the top right corner of the comments section.
## Comments in Reviews
When you review an experiment, you can add a comment with [the review](./reviews). This comment shows up in the comment section like all other comments.
## Related Resources
Request and give reviews
Get comment notifications
Share experiment findings
Configure notification settings
# Minimum Detectable Effects and Non-Inferiority Margins
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/design/effect-sizes
Use minimum detectable effects for your success metrics to run power analyses and understand how much traffic you need. Set non-inferiority margins for your guardrail metrics to define acceptable tolerance levels for potential metric degradation.
When planning an experiment, [power analyses](./power) help you
find the sample size needed for your metrics to reach the
desired power. Power is a function of effect size. All
power analyses calculate the required sample size for the effect sizes
you want to detect. Smaller effects are harder to detect and require larger sample sizes.
## Minimum Detectable Effects
For success metrics, the effect size is the minimum detectable effect
(MDE), which represents the smallest effect you want to be able to detect.
Use the MDE to design your experiment so that it has enough statistical power to detect meaningful
effects.
Picking the MDE is a trade-off between:
* The smallest effect that is still relevant for the business
* The smallest effect that is practically measurable
As an experimenter, use your domain expertise and discuss with
stakeholders to decide the smallest effect you would consider
meaningful. Then, calculate the required sample size.
If the sample size needed to detect your chosen MDE is unrealistically large,
you need to increase the MDE.
Note: the MDE is a required input to the power analysis, but does not impact
the calculation of results.
Set the minimum detectable effect size to the smallest effect that you and
your stakeholders care about. This ensures your A/B test can detect
effects that are meaningful to the business. In other words, if the true effect is smaller
than the MDE and you fail to detect it, it doesn't matter because the improvement
would be too small to justify shipping anyway. Larger effects are easier to
detect than smaller effects, while a smaller MDE requires a larger sample
size.
## Non-Inferiority Margins
Confidence uses a different statistical test for guardrail metrics than for
success metrics. These tests, called [non-inferiority tests](/docs/experiments/stats/stat-tests), verify that the
metric performs better than a specified non-inferiority margin (NIM). The
non-inferiority margin is essentially a tolerance level—you accept a small amount of
degradation in the guardrail metric, but it must not worsen
beyond the NIM.
The non-inferiority margin (NIM) is a tolerance threshold that helps you gather
evidence to rule out the possibility that the metric deteriorates
by more than the NIM. This choice affects both the power analysis
and the results calculations.
Unlike the MDE for success metrics, the statistical tests for guardrail metrics directly use the
NIMs in the hypothesis tests.
Because of this, the NIM serves dual purposes: it's both an effect size for the power analysis and a tolerance threshold in the
statistical test itself.
Smaller NIMs require larger sample sizes because it becomes harder to gather
enough evidence that the metric stays within a tighter tolerance range.
## How to Find the Smallest Practically Measurable Effect
Follow these steps to quickly understand what effect sizes are practically measurable in your experiments.
1. **Assess how large your experiments can be.**
* Do you need to run multiple experiments in parallel? For example, if you need to run 4 experiments
simultaneously on the same population, each experiment can only use 100/4=25% of users.
* Do you want to limit exposure to a new variant because the changes are risky?
For example, if your experiments typically have 10,000 users, calculate what effect size you have
enough power to detect with this sample size.
2. **Decide if the detectable effect size is small enough for your business needs.**
* If the smallest detectable effect is small enough, select a value slightly larger than the
smallest effect you can detect with enough power. Remember that sample size calculations are estimates with inherent uncertainty.
* If the smallest detectable effect is too large, consider these options:
* **Change the metric.** Variance can vary widely between metrics measuring similar aspects
of user behavior. A lower-variance metric is more sensitive, giving you a higher
chance of detecting effects.
* **Adjust alpha and power.** You can detect smaller effect sizes with
the same sample size if you can tolerate more risk. Increase alpha to
accept increased false positive results (shipping changes with no real
effect). Decrease power to accept increased false negative results (failing
to ship changes with positive effects).
## Related Resources
Understand statistical power
Calculate required sample sizes
Configure success and guardrail metrics
Configure alpha and power
# Hypothesis
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/design/hypothesis
Formulate a clear hypothesis that you can test with an experiment.
You should start the design of an experiment by specifying a clear
hypothesis that can be quantitatively tested to inform
your decision-making.
## How to Set Up a Hypothesis
A well-formulated hypothesis is a specific assumption that can be conclusively
tested through an experiment.
Not all hypotheses are equally effective. An effective hypothesis should be:
* a statement, not a question
* clear about what experiment outcomes would support or weaken it
* clear about the key variables
* grounded in past research/learnings
* written with as few assumptions as possible
It's also important to consider what decision you make based on the experiment results.
By articulating the decision you want to make, you ensure that your hypothesis statement
reflects the intentions of your team and drives actionable outcomes.
To convert your idea into a testable hypothesis, use the following template as a
starting point:
> **Doing this/building this feature/creating this experience** for
> **these people/personas** should result in a change in their behavior, as measured by
> **success metrics**. The data supports the hypothesis if the **success metrics**
> change by **the minimum detectable effect**.
You can read more about minimum detectable effects (MDE) on the [effect sizes page](./effect-sizes).
## Example Hypothesis
Imagine your team is building an autoplay feature for the Spotify mobile app.
Your team's goals are:
1. Make it easier for people to continue listening when their content ends.
2. Lead users to listen to more content curated by Spotify.
The autoplay feature depends on Radio, which is one type of curated content.
There happens to be a company objective to increase the percentage of content
hours curated by Spotify, so your team decides to choose this as the success
metric for your experiment. Your team would decide to roll out this feature if
it increases the success metric without harming some general guardrail metrics.
Here's an example hypothesis statement:
> **Continuing to play music or podcasts when a play context ends** for
> **all users** should result in **users listening to more of Spotify's
> curated content rather than searching for something else to play themselves**,
> as measured by **percent programmed content**. The data supports the hypothesis if
> the **change in percent programmed content** increases by **2.5pp**.
## Composite Hypothesis
Many experiments use one or two success metrics and a few guardrail metrics. In
this scenario you should write a hypothesis statement for each success metric,
while for the guardrails it's generally enough to just state the hypothesis
that the treatment does not deteriorate the guardrail metrics more than the
acceptable margins (known as non-inferiority margins).
Consider the earlier example of the Autoplay experiment.
The guardrail metrics are the skip rate of programmed content and the app crash rate.
To also include the guardrail metrics, change the hypothesis statement as follows:
> Continuing to play music or podcasts when a play context ends for all
> users should result in users listening to more of Spotify's curated
> content rather than searching for something else to play themselves, as
> measured by percent programmed content. The data supports the hypothesis if
> the percent programmed content increases by 2.5pp, while the app crash rate
> and the programmed content skip rate don't increase by more than the acceptable margins.
The hypothesis statements for the success metrics intend to capture a change in
user behavior that is measurable by some metric.
For guardrail metrics, expect no change, or only a small one.
In settings like this, you need to define the decision rule for a
successful experiment upfront. For example, if the treatment significantly improves
one success metric, but there is no evidence of non-inferiority on the guardrail
metrics, should you ship this variant or not? Have you found enough evidence that this
variant is better than the current default version?
Read more about the [decision rules](/docs/experiments/analyze-results).
## Related Resources
Configure MDE and NIM settings
Configure success and guardrail metrics
Understand decision rules
Run your experiment
# Alpha and Power
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/design/power
Understand risks, how to control them, and what affects your chances of finding effects.
Experimentation is about understanding and controlling risks. Two concepts are
central to managing risk in experimentation: alpha and power.
## Alpha
Alpha is the false positive rate, which describes how often false positives
occur. A false positive happens when you conclude an effect exists when in
reality it doesn't. For example, suppose you run an experiment and the
results show that conversion has increased. If the truth is that there
is no real effect and conversion didn't actually increase because of the treatment, you
have observed a false positive result. Because data is inherently noisy,
the false positive rate can never be completely zero, so you must choose an
acceptable level of risk.
Alpha is commonly set to 5% in many sciences, which is also the default for Confidence.
The alpha you choose determines the rate at which you are willing to accept false positives
across repeated experiments.
Depending on the consequences of shipping a feature that truly has no effect,
you may want to decrease (more conservative) or increase (less conservative) this value.
In an ideal world, you don't want any false positive results at all, but
setting a low alpha makes it harder to detect effects that truly exist.
Setting alpha is a balancing act between the risk of finding effects when there
are none (false positives) and missing effects that really
do exist (false negatives). Common values for alpha are 1%, 5%, and 10%. Higher alphas
are often used in early stage experiments that seek to identify promising variants for more rigorous testing later.
## Power
Statistical power describes the probability of detecting an effect when there truly *is* an effect of a particular size.
It determines your ability to separate signal from noise, with higher power meaning better chances of finding effects when they exist.
Power is also known as the true positive rate, and equals 1 minus the false negative rate.
Power is commonly set to 80% in many sciences, which is also the default for
Confidence. Depending on the consequences of missing a true effect, you may want
to adjust this value. The power level relates to the risks of magnitude (type-M)
and sign (type-S) errors. When an experiment has low power, there is a higher risk that
significant effects it detects are either overestimated or even have the
wrong sign (positive vs. negative).
## Power Analysis
Power analysis is the process of determining the minimum number of users
required to reach a desired level of statistical power. While it's often called
"sample size calculation," this represents the minimum number of users needed to detect a
desired effect size with a given level of confidence, not necessarily the total number
of users exposed in an experiment.
The analysis takes several inputs as outlined below, and outputs the minimum
number of users required to achieve the desired level of power.
### Alpha and Power
You set alpha and power according to your tolerance for false positive or false
negative errors. By default, Confidence sets alpha (the false positive
rate) to 5% and the power level to 80%, but you can adjust them
based on your risk tolerance. Lowering alpha or increasing the
power level increases your confidence in your measured results and your ability
to detect significant effects, but it also increases the
number of users required for your experiment.
### Experiment Intake
Experiment intake is the number of days at the start of an experiment during
which you include newly exposed users in metric calculations. For example, if
you run your experiment for 14 days and want to measure "Consumption during
Week 1," your intake is 7 days. The intake period is typically
determined by how long the experiment can feasibly run.
To avoid seasonality effects, the intake period should ideally be a multiple of 7 days.
A longer experiment duration delays decision-making, but enables the
experiment to expose more users and helps achieve the desired statistical power.
### Metrics
You should select your metrics according to the [hypothesis](./hypothesis) of the experiment.
The variance of your selected metrics significantly affects
the required sample size—high-variance metrics require many more users to
detect small effect sizes. The number of metrics also affects the required sample size
because [multiple testing corrections](../stats/adjustment-multiple-comparisons) impact the adjusted
levels of alpha and power in the experiment.
The minimum detectable effect (MDE) is the smallest effect size you
want to be able to measure to make a decision. The sample size calculation uses
the MDE to decide how many users you need to detect this effect
with a probability equal to the power level. The number of users required is
inversely proportional to the square of the MDE, which means measuring
small changes requires many more users. The MDE should be both meaningful
and realistic. If you set the MDE too high, you may miss effects that
would impact your decision. If you set the MDE too low, it may be
impossible to achieve the desired power with a realistic number of users in reasonable
time. You should ideally set the MDE based on product
requirements for decision-making and a meta-analysis of effect sizes observed in
prior experiments.
As a last resort, consider Cohen's Recommendations:
* Small effect: 1% of the variance ("too small to detect other than statistically; lower limit of what is clinically relevant")
* Medium effect: 6% of the variance ("clear with careful observation")
* Large effect: 15% of the variance ("clear with a superficial glance; unlikely to be the focus of research because it's too obvious")
The minimum detectable effect (MDE) is the effect size used for success metrics.
For guardrail metrics, the effect size is the non-inferiority margin (NIM).
### Number of Variants
Each variant you add increases the multiple testing correction and,
as a result, the number of users required. The probability of
observing a significant result by chance increases with the number of
comparisons, which requires adjustments for multiple comparisons.
For example, with 2 variants you have 1 comparison, but with
3 variants you have 2 comparisons (each compared to control). It's important to
carefully consider the number of variants before running your experiment—
only include variants you're genuinely interested in testing.
### Treatment Sizes
An equal split between treatment and control minimizes the
number of required users, but carries higher risk
because the experiment exposes more users to the new, unproven variant.
## Related Resources
Calculate required sample sizes
Configure MDE and NIM settings
Configure alpha and power
Step-by-step sample size guide
# Mutually Exclusive Experiments
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/exclusive-experiments
Use exclusivity groups to prevent users from being exposed to multiple conflicting experiments at the same time.
Make experiments exclusive to other experiments by ensuring that
they have at least one exclusivity group in common. For example, make two A/B
tests exclusive to each other by adding the same exclusivity group to both A/B
tests. No overlap means that no user or other entity is in both of these
experiments at the same time. See the [audience](./audience) page on
experiments for more information about exclusivity when running experiments.
Using exclusivity groups, you can coordinate experiments that change the same or similar parts of an
experience and make them exclusive to each other.
For example, use exclusivity if you want to run multiple related experiments, but you want to only
expose users to one of the experiments.
If you don't make the experiments exclusive to each other, all the multiple experiments can expose
a given user.
Watch this video to get a quick overview of how to use exclusivity groups to coordinate experiments in Confidence.
## Exclusivity Groups
By using several exclusivity groups, you can create sophisticated coordination
of experimentation programs. For example, a set of A/B tests can be exclusive to
each other, but be randomly overlapping with another set of experiments, where
the experiments in the second set are also exclusive to each other.
Read more about exclusivity groups in the [surface documentation](/docs/surfaces/introduction).
On the surface page under **Settings**, you can configure
exclusivity groups to be `suggested`. Suggested exclusivity groups are
automatically added to the experiment when you add the experiment to the
surface.
## How Exclusivity Works
If you want two experiments to be exclusive to each other, there must be
enough users to allocate to both experiments. If there aren't enough users to
allocate, the experiments aren't able to run at the same time. You define how
much of the audience to allocate to an experiment by setting the
[allocation](/docs/flags/audience#allocation).
For example, these two experiments can run at the same time, and
be exclusive to each other:
* Experiment A: 50% allocation, exclusivity group `A`
* Experiment B: 50% allocation, exclusivity group `A`
These two experiments aren't able to run at the same time since there aren't
enough users to allocate to both experiments:
* Experiment A: 50% allocation, exclusivity group `A`
* Experiment B: 70% allocation, exclusivity group `A`
Confidence takes inclusion criteria of the audience into account when
allocating users to experiments. If the audience of experiment A is already
exclusive to the audience of experiment B based on their inclusion criteria,
then experiment A is able to run at the same time as experiment B even if
their allocations are larger than 100%. For example:
* Experiment A: 50% allocation, exclusivity group `A`, inclusion criteria `country = US`
* Experiment B: 70% allocation, exclusivity group `A`, inclusion criteria `country = SE`
### When Experiments should be Mutually Exclusive
In certain situations, being in multiple related experiments at the same time can lead to a poor
and unexpected user experience because of dependencies among tests.
Examples of when you should consider using exclusivity to avoid conflicts between experiments:
* If experiments are using the same flag, for example to alter the ranking of
search results. Since only one experiment can decide the search rank for one
user at one time, exclusive experiments allow several experiments on
search rank simultaneously.
* If the variants of the experiments can interfere with each other. For example, if one experiment changes the background color of a page (using flag A), and
another experiment changes the text color of the same page (using flag B).
In other cases it may not be as obvious if you should make the experiments
exclusive to each other or not.
A rule of thumb is to ask the question:
> If these experiments run in sequence after each other, is it fine to
> ship the winning variants from each experiment without testing them
> together first?
If the answer to that question is *"Yes"* then it's fine to let the experiments
overlap and run them at the same time.
If the answer is *"No"*, you should either test the changes in combination or
run them separately and non-overlapping by making sure that the experiments are
using the same exclusivity group.
If you test them separately, make sure that you then run a follow-up experiment
where you test the promising cells from each experiment in combination with
each other. What combinations to test depends on what treatment
cells from the two experiments that can co-exist in a meaningful way.
A rule of thumb is that you should run overlapping experiments when possible to save
space for more simultaneous product evaluations.
## Make an Experiment Exclusive to Other Experiments
To make an experiment exclusive to other experiments with a certain exclusivity group,
follow these steps.
1. Go to Confidence and find the A/B test or rollout you want to make exclusive to other experiments.
2. Select the surface on which the exclusivity group lives.
3. Select the exclusivity group you want to use.
4. Click **Save**.
## Coordinate Experiments Across Multiple Surfaces
Coordinate experiments across multiple surfaces by using exclusivity groups from more than one surface.
Watch this video to build intuition for advanced experiment coordination in Confidence.
## Related Resources
Step-by-step exclusivity guide
Configure exclusivity groups
Configure experiment audience
Run your experiment
# Exploration
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/exploration
Drill down into the results of an experiment to learn more about the impact.
Confidence gives you the ability to run analyses on an experiment for
exploratory purposes, also called Explorations. Because this type of analysis significantly increases the
risk of seeing false positives, you should never use it to decide the success or
failure of an experiment. Rather, use the results to inspire hypotheses to test
in new experiments.
You see all existing explorations in the **Exploration** section on the right sidebar. For ended
experiments, you can also create an exploration directly from **Explore** in the
[Spotlight](./analyze-results#spotlight) section.
## Metrics and Dimensions
To produce results, you need to add metrics to your analysis. You can select any metric that uses
the same entity that you configured in the [metric configuration section](/docs/experiments/metrics#assignments) for your experiment.
For each metric, you can add dimensions if they exist. Dimensions come from
[dimension tables](/docs/metrics/dimension-tables) that use the same entity as the metric.
Use dimensions to find out if the experiment had a different impact
on different subgroups.
### Static Dimensions and Dimensions that Change over Time
Confidence supports both static dimensions and dimensions that change over time.
Static dimensions don't change rapidly over time and
aren't affected by the experiment treatment. For example, user age,
registration country, preferred language, etc.
If the experiment treatment can affect a dimension value, it's a time-changing
dimension. For example, user's subscription status, consumption patterns, feature
usages, etc.
To ensure that segmentation is not misleading for time-changing dimensions,
Confidence always uses the dimension value at the time right before the
user got exposed to the treatment. This way, the treatment doesn't
influence the dimension value itself.
If users don't have dimension values in the table, they get NULL as
dimension values. Make the result interpretations easier by mapping NULL values
to a meaningful value in the dimension table configuration.
### Examples of Dimensions That Change over Time
#### New User vs Existing User
The dimension table has a column `is_existing_user` that has a true value for
all existing users at a given time. Since new users that come in during the
experiment don't have a value on this dimension table before exposure by
construction, they get the value NULL. You can map the NULL value to "New user"
in the dimension-table configuration to make the interpretation of the
exploration easier.
You also need to decide on when to consider a user "New" relative to their first
exposure. For example, you can use a 1-day window if a user needs to have signed
up within 24 hours before exposure to be in the "New" category.
#### User that Took a Certain Action the Week Before the Experiment
Examples of this action could be users that watched a
video, added something to checkout, used a certain feature in your app, etc.
The dimension table has a column `took_action_x_last_7_days` that captures a
rolling seven-day window and only uses the last value. If you produce this data
hourly, you can use a one-hour window in the window setting for the dimension in
the dimension-table configuration.
If your dimension value requires aggregations, perform those directly in the dimension table query or before. Confidence only uses the latest non-NULL value in the time window.
If instead the table has a column `took_action_x` with hourly values in the
dimension table and you configure Confidence to use a 7-day window to
fetch the dimension value, **Confidence uses the last non-NULL value
in that 7-day window, which doesn't correspond to the goal of this example.**
## Stale Analyses
Sometimes your analysis might end up in a `Stale` state. If you hover over the
status, you see a message describing why the analysis is stale. Two reasons are:
1. New data exists, and results might be different if you run
the calculations again. To calculate new results based on all available data, click **Retrigger**
above the results.
2. The conclusion is older than the results. Retriggering result calculations because
of the first reason is a common explanation for this. It means that the results might have changed and the
conclusion is possibly invalid. Update the conclusion to get rid of this warning.
## Related Resources
Step-by-step exploration guide
Configure segmentation dimensions
Results analysis overview
# Introduction to Experiments
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/introduction
Confidence is a platform for running experiments. Experiments are a way to test new ideas and measure their impact on your business. Confidence comes with multiple built-in experiment types, and you can also create your own custom experiment designs using the API.
In Confidence, experiment types define common patterns used for
experimentation. Confidence supports three built-in experiment types:
Experiments with one or more treatment groups, typically used in the product
development phase where you test ideas. Can use both success and guardrail
metrics to evaluate if new ideas are successful. A/B tests have a fixed
allocation that you can't change.
Experiments that rely on feature flagging from a third party. With an analysis,
you can leverage the metrics and analysis capabilities of Confidence
together with feature flagging from a Provider other than Confidence. You can
also re-analyze experiments that you ran in the past.
Experiments with only a single treatment group,
where the purpose is to gradually roll out a new variant. Rollouts have an
adjustable level of reach to make the experience of rolling out fully flexible.
Rollouts only use guardrails.
## Related Resources
Step-by-step A/B test tutorial
Step-by-step rollout tutorial
Learn about the statistics behind experiments
# Links
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/links
Add links to related resources for your experiment like design files, docs, etc.
Click the add button in the links sidebar section to add a link to your
experiment. A link has a name and a URL. Click the edit button in the link
section on the sidebar to edit or delete a link.
Add links to related resources to make it easier for your team to understand the experiment and its context in the future.
## Related Resources
Collaborate with your team
Get feedback on experiments
Share experiment findings
Document your hypothesis
# Metrics
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/metrics
Metrics define what to evaluate an experiment on.
The success of an experiment depends on how the metrics you select move.
Metrics describe the behavior of the entity you're experimenting on.
By first calculating the metrics for the control and treatment groups, you can later statistically
compare the two groups.
Metrics in Confidence belong to one of two classes of metrics: success or guardrail metrics.
You first need to create a metric before you can use it in an experiment.
You can read more about how to create metrics on the [metrics](./metrics) page.
## Assignments
For Confidence to be able to evaluate the experiment, it needs to know what you are experimenting on.
Specify an entity, and where the assignments for these entities exist.
1. Click the **Edit** icon to the right on the **Metrics** section of the experiment edit page to bring up
the metric configuration dialog.
2. Select the entity that you want to analyze.
3. Select the assignment table that has assignment logs for the experiment.
4. Optional. Select an [exposure filter](#exposure-filtering).
5. Select how often you want to compute metrics by specifying the metric interval.
6. Click **Save** to save the metric configuration.
### Exposure Filtering
Exposure filters are methods for narrowing down more closely which users
to include in the exposure definition and the analysis of your experiment.
When you add an exposure filter for your experiment, the analysis only
includes the exposed users that also match the exposure filter. The time of
exposure is the first unit of time after default exposure where the user
matches the exposure filter.
You may want to use exposure filters if the default definition of exposure is
too broad for what you want to measure in your experiment.
[Read more about Exposure Filtering](/docs/metrics/exposure)
## Success Metrics
Success metrics aim to prove the hypothesis of the A/B test. For example,
if your hypothesis is that "more users stream podcasts if you rank
podcasts higher," then an appropriate success metric is a metric that measures podcast
consumption.
In this example, the hypothesis expects the metric to increase.
A significant result means that there is evidence of an effect in the desired direction.
Test for the following change in a success metric.
Test for a significant increase of the metric. For example, an
increase in hours spent listening to Spotify.
Test for a significant decrease of the metric. For example, a
decrease in the number of skipped songs in personalized playlists.
You can optionally set a [minimum detectable effect
(MDE)](./design/effect-sizes#minimum-detectable-effects) for success metrics. The MDE is the effect
size that you want to be able to detect with high certainty. You must set an MDE to be able to run a
[power analysis](/docs/experiments/statistical-settings) for the metric and
learn how much traffic the metric needs.
## Guardrail Metrics
A guardrail metric is a metric that ensures that the experiment doesn't have any
unexpected side effects.
You can use any metric as a guardrail metric.
The meaning of a significant result depends on the type of guardrail metric.
Use guardrail metrics in the following situations:
* When you want to ensure that your A/B test doesn't introduce regressions in
performance or product quality.
* When you want to ensure that your A/B test doesn't have a negative impact on
a metric that some other part of the organization cares about.
Guardrail metrics can use non-inferiority margins that let you seek evidence that the change doesn't
have a negative impact that exceeds the margin you specify. The margin is optional. The interpretation
of the results depends on whether you use non-inferiority margins. All guardrail metrics in the same
experiment must either all use or all not use non-inferiority margins.
### With Non-Inferiority Margin
Use guardrail metrics with non-inferiority margins to look for evidence that the change doesn't
negatively impact the metric more than your specified non-inferiority margin.
A significant result means that there is evidence that the guardrail is within acceptable margins.
Guardrail metrics with non-inferiority margins test for the following.
Test if there is evidence that the metric hasn't increased
by more than the NIM. For example, the number of skipped songs in personalized playlists shouldn't
increase by more than 1%.
Test if there is evidence that the metric hasn't decreased
by more than the NIM. For example, the number of hours spent listening to Spotify shouldn't
decrease by more than 1%.
### Without Non-Inferiority Margin
Use guardrail metrics without non-inferiority margins to look for evidence that the change has a
negative impact on the metric. A significant result means that there is evidence that the guardrail
deteriorates because of the change.
Guardrail metrics with non-inferiority margins test for the following.
Test if there is evidence that the metric has increased. For example, test if the number of skipped
songs in personalized playlists has increased.
Test if there is evidence that the metric has decreased. For example, test if the number of hours
spent listening to Spotify has decreased.
Set non-inferiority margins if you can. With non-inferiority margins, you seek evidence that the change doesn't
lead to a regression by more than a tolerance level you specify. Without a non-inferiority margin, a
lack of evidence of a deterioration doesn't imply a neutral result. Read more about how the two
approaches compare in [the guardrail metric lesson](/docs/experiments/metrics).
## Required Metrics
For experiments that use a surface with required metrics,
Confidence adds these metrics to the bottom of the design page. Read more about
required metrics in the [surface documentation](/docs/surfaces/introduction#required-metrics).
## Related Resources
Step-by-step metric setup
Deep dive into metric configuration
Configure MDE and NIM settings
# Monitor a Live Experiment
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/monitoring
Understand when your experiment isn't going as planned and how to react.
Confidence summarizes all the checks for an experiment on the results page in [the spotlight section](/docs/experiments/analyze-results).
Monitor your experiment to make sure that you set it up correctly so
that the data collection and variant delivery to users work as intended.
Three important questions to ask when verifying an experiment are:
* Is exposure working as intended?
* Are the control or treatment groups biased?
* Are users receiving the intended experience?
## Sample Ratio Mismatch Check
The set up of the experiment defined exposure.
The treatment must not impact exposure for results to be trustworthy.
To verify that exposure works as intended, the observed proportions in all treatment groups
should follow the expected variant allocations.
The sample ratio mismatch check tests if the observed proportions of traffic in each variant match
the expected proportions.
If the test indicates a problem, you have a clear signal that there is a systematic difference across
treatment groups in the probability that users log assignments.
A systematic traffic imbalance invalidates the results, as the groups are often no longer
comparable.
The analysis of the experiment relies on there being no systematic difference between the treatment groups.
Correct randomization makes it possible to attribute any movements in metrics to the treatment.
If there is a sign of a sample ratio mismatch, you should stop the experiment and investigate the issue.
## Deterioration Checks
Confidence always checks the metrics you've selected for deterioration. Regardless
of the [test evaluation frequency](/docs/experiments/statistical-settings)
employed, Confidence tests your metrics for movements in the wrong direction as
often as the metric data supports. If there is evidence that metrics are moving in
the wrong direction, Confidence alerts you and recommends aborting the experiment.
## Stop the Experiment
You should stop the experiment when the experiment reaches its required sample
size. At this point, all metrics have the intended amount of data for powering
the metrics. You should stop the experiment regardless of if results show
significant improvements or not.
Stop your experiment when all metrics meet their required sample size and achieve power.
## Current Powered Effect
If you have to stop your experiment before you reach the required sample size, make sure to present
the current powered effect together with results to reflect this increased
uncertainty. Failing to achieve the necessary sample size to power all metrics
means that the risk of overestimating effects is higher.
## Related Resources
Understand experiment results
Configure experiment parameters
Calculate required sample sizes
# Reports
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/reports
Summarize experiment conclusions and share results with your team using the experiment report tab.
Each experiment has a report tab. Use
reports to summarize the conclusions from your experiment and to share the
results with your team. Reports help you spread the learnings from your
experiment to the rest of your organization.
The reports tab becomes available when you launch your experiment.
## Pre-Populated Content
To make it easier for you to write a report, Confidence pre-populates the report
with some information from your experiment.
* If you have uploaded images for the treatment variants, these images appear in the report automatically.
* If you have written a hypothesis, it's included in the report.
* The report includes all the metric results.
## Edit the report
To edit the report, click the `Edit report` button in the top right corner of
the report tab. In edit mode, Confidence highlights all editable fields.
### Metric Results
Decide which metric results to include by clicking the eye icon on the metric
result card.
### Images
You can change the images that the report includes by removing the existing
images (if there are any) and uploading new images.
### Hide Sections
You can hide the sections summary, results, metrics, and next steps section by
leaving them empty. When you leave edit mode, the empty sections are not visible
in the report. You add them back by clicking the `Edit report` button and adding
content to the empty sections.
## Comments
Comment on any part of a report by clicking the comment icon on the right side
and then clicking one of the highlighted areas to comment on. You can mention
other users in your comments by typing `@` followed by their name.
## Share the report
Copy the URL of the report tab to share it with your organization. Remember that
for someone to view the report, they need to have read permissions to the
experiment. Click 'Permissions' in the top right corner of the report tab to see
who manages the access.
## Related Resources
Understand experiment results
Write effective hypotheses
Add comments to experiments
Manage experiment access
# Reviews
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/reviews
Reviews allow you to give feedback on and signal to your teammates that an experiment is ready for launch or not. Surfaces can optionally require approval before allowing experiments to launch.
## Overview
The review process for A/B tests and rollouts can happen in several ways in
Confidence:
* You can request a review from a teammate by clicking the plus icon
in the **Reviews** section on the sidebar.
* You can request a review from an AI review agent that provides automated
feedback on your experiment design.
* Anyone that has read rights to your experiment can decide to leave a review
by clicking the thumbs-up-and-down icon in the **Reviews** section on the
sidebar.
* A surface that you run your experiment on can suggest or require reviewers on
your experiment. If your experiment runs on surfaces that require reviewers, you can't launch
the experiment unless at least one reviewer from each surface has approved the
experiment.
Get pinged on Slack when someone mentions you or replies to your comment
by [connecting your personal Slack account with
Confidence](/docs/how-to-guides/connect-personal-slack).
### Required and Optional Reviewers
All reviewers that you add on the experiment design page are optional in the
sense that you can launch the experiment without approval. Required reviewers
only come from [surfaces](/docs/surfaces/surface-settings).
AI agent approvals are informational only and do not count toward required
reviews. At least one human reviewer from each surface must approve before
you can launch the experiment.
#### Experiment Changes After Approval
Changes to approved experiments reset the approvals. This means that
required reviewers must approve the experiment again before you can launch it.
Changes that reset the approvals are:
* Changing the flag, flag variants, or adding more variants.
* Changing any settings in the audience section, including the allocation percentage.
### Add Reviewers for a Surface
The surface owner can add suggested or required reviewers to the surface. Configure reviewers on the [surface settings page](/docs/surfaces/surface-settings).
## Request a Review
Request a review from a teammate by clicking the plus sign icon in the **Reviews**
section on the right sidebar of the experiment design page. Reviewers can
approve or reject the experiment design and leave comments.
Request reviews from surface reviewers by clicking **Request review** next to
their name. To request a review from all reviewers on a surface, click **Request
review** next to the surface name.
Reviews from optional reviewers are only informational and do not block
launching experiments. You should discuss with your team how to work with
reviews.
If a reviewer requests your review for an experiment, there is a
banner on top of the experiment page when you visit the experiment to remind
you to leave your review. This banner disappears for already approved
experiments.
## Review an Experiment
To review an experiment, click the comment icon at the top right corner on the
experiment page. After you have added comments on sections you want to discuss
or request changes to, submit the review by making an overall judgment
on the experiment design and approve or reject it.
To review an experiment:
1. Go to an A/B test or a rollout.
2. Click the thumbs-up-and-down icon at the top of the **Reviews** section on the right sidebar.
3. Select the outcome of the review in the review dialog. Add a comment if needed.
4. Click **Save**.
## AI Review Agents Beta
AI review agents are currently in beta. If you don't see the **Agents** page in
your organization, contact
[experimentation-cs@spotify.com](mailto:experimentation-cs@spotify.com) to
request access.
AI review agents provide automated feedback on experiment designs. Create
custom agents with specific instructions tailored to your team's review
criteria. When you request a review from an agent, it analyzes the experiment
design and posts comments on individual sections where it has feedback.
Agents can review both A/B tests and rollouts. Each agent has instructions
that define what to check for in each section of the experiment design, such
as hypothesis quality, metric configuration, audience targeting, and sample
size calculations.
Agent reviews are informational and don't count toward required reviews.
An AI agent approval does not meet the required reviewer requirement
for launching—at least one human reviewer must approve.
Learn how to create and use AI review agents
## Related Resources
Create and use AI review agents
Set up surface review requirements
Add comments to experiments
Get review notifications
Configure surface reviewers
# Sample Size Calculator
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/sample-size-calculator
Understand how the sample size calculator works and what affects required sample size.
The sample size calculator is a tool that assists in planning the length and
size of an experiment. The tool calculates what sample size you need to
achieve the requested level of power given the set-up of the experiment. The
required sample size differs across metrics. The tool also displays the largest
required sample size across all metrics. Having a large enough sample size is
important to ensure that the experiment has enough sensitivity to detect
meaningful effects. For more about what affects the required sample size,
see the [power](./design/power) page.
In addition to the required sample size, the calculator estimates the expected
number of samples and the number of days needed to reach the required sample
size for each metric. A progress chart visualizes this information, making it
easier to plan the experiment's runtime.
The expected sample size, days needed, and progress charts are currently in
beta. If you don't see these features in your organization, contact
[experimentation-cs@spotify.com](mailto:experimentation-cs@spotify.com) to
request access.
The sample size calculator doesn't take audience targeting into account. If
you are targeting a subset of the population, then the variance of the metrics
might be different for different subsets of the population. Some subsets might
have larger variance, which increases the required number of users to power a
certain [MDE/NIM](./design/effect-sizes), while others might have
smaller variances which could then decrease the required sample size for a
certain MDE/NIM.
Learn more about sample size calculations in the [the sample size calculation course](/docs/experiments/sample-size-calculator).
## Sample Size for New Metrics
When calculating the required sample size for an experiment, Confidence
looks at historical data for the metrics in the experiment.
There needs to be at least 14 days (plus the aggregation window and exposure
offsets) of historical data for the metric in order for Confidence to be able
to calculate the required sample size. For example, if you have an experiment
with a metric that has a 7-day aggregation window and a 7-day exposure offset,
you need at least 28 days of historical data. If there is not enough
historical data, Confidence can't calculate the required sample
size.
## Expected Sample Size and Days Needed Beta
The calculator uses the exposure source to estimate how many samples per day
the experiment can expect to receive. Based on this rate, it calculates:
* **Expected samples**: The estimated total samples by the end of the planned
runtime, based on historical exposure rates.
* **Days needed**: The estimated number of days to reach the required sample
size for each metric. The overall days needed shown in the widget header
is the maximum across all metrics.
When the expected samples exceed the required samples, the experiment is on
track to be sufficiently powered within its planned runtime. If the days
needed exceeds the planned runtime, consider extending the runtime or
reducing the required sample size.
Both expected samples and days needed are estimates based on historical
exposure from the selected exposure source. The actual exposure in a live
experiment may be lower if:
* The experiment uses audience targeting that excludes some users who were
included in the historical data.
* The flag has other rules that route some traffic away from the experiment.
Consider these factors when planning the experiment's runtime.
## Progress Chart Beta
The progress chart shows the estimated progress toward the required sample
size for each metric over time. The Y-axis displays the percentage of the
required sample size that the experiment has accumulated, and the X-axis
shows the number of days after the experiment starts. A horizontal line at
100% marks the required sample size target.
A vertical dashed line indicates the current planned runtime. Lines that
extend beyond this point appear as dashed projections based on the estimated
daily sample rate.
You can expand individual metric rows to view a per-metric chart. The
per-metric chart shows the absolute sample size on the Y-axis instead of a
percentage, with separate lines for the expected and required sample sizes.
## Exposure Source
The required sample size calculation consists of three parts:
* Obtaining an exposure source
* Using the exposure source to calculate the mean and variance of the metrics
* Calculating the required sample size for each metric based on the mean and variance
The exposure source is the source of the data used to calculate the mean and variance of the metrics. It can be one of the following:
* **Past assignments**: Use all existing assignments available in your assignment table, or filter these on assignments from specific flags to only include a cohort of users similar to those in your upcoming experiment.
- **Previous experiments**: Use exposure from previous experiments as an exposure source.
## Related Resources
Step-by-step sample size guide
Understand statistical power
Configure MDE and NIM settings
Configure alpha and power
# Statistical Settings
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/statistical-settings
Tweak the statistical settings to how you want to control the risk of false positives and false negatives.
For a successful and well-planned experiment, you should commit beforehand to a
strategy for when and how to evaluate the results to avoid the infamous pitfalls
of peeking. Similarly, knowing how much traffic you require to be able to
identify the effects of interest with a high probability is essential for
results to be trustworthy.
## Test Evaluation Frequency
When setting up an experiment, you need to select how often to evaluate the results of the
test. You have two options:
* View results continuously
* View results upon conclusion
Viewing results continuously means that you get the results updated
and presented hourly or daily using sequential tests.
Viewing results only upon conclusion separates the data
collection and analysis phases of the experiment.
With this choice, you can view the results after you end the experiment using fixed horizon tests.
Read more about the details on the [sequential tests](./stats/sequential-tests) page.
Selecting a strategy before launching the experiment is crucial, as it makes it possible to control
the risk of finding false positives regardless of the choice made.
Failure to handle this issue and looking at the results when you shouldn't is commonly referred to
as the "peeking problem."
If you choose to view results only at the end of the experiment, Confidence still uses sequential
tests to run daily checks on all your metrics to ensure they have not deteriorated. Read more about
that in the [monitoring](./monitoring) section.
The benefit of viewing results at the end of the experiment is that it has
higher precision compared to a test with results that update daily. This
means that sticking to that approach leads to less uncertainty in the final
estimates of the effects of the treatments.
## Alpha and Power
The false positive rate is also known as *alpha* and is by default 5%. The
power level, also known as the true positive rate, has a default value of 80%.
A lower alpha means that false positives are less likely to
happen, but at the same time the chance of finding an effect if there is one is
also less likely to happen. The power level sets the desired probability for
being able to find an effect if there is one. See the next section on [power analysis](./sample-size-calculator#power-analysis-and-the-required-sample-size) for how the required
sample size can help inform you how much traffic you need to achieve the
desired level of power.
Confidence adjusts the selected alpha and power levels for multiple comparisons using a Bonferroni
correction that handles success and guardrail metrics differently.
The corrections ensure that your error rates for the decision to ship the feature is at most the
errors rates implied by the false positive rate, determined by alpha, and the power level you give.
Read more about [adjustment for multiple comparisons](./stats/adjustment-multiple-comparisons).
## Power Analysis and the Required Sample Size
The sample size calculator is a tool that assists in planning the length and
size of an experiment. The tool calculates what sample size you need to
achieve the requested level of power given the set-up of the experiment. The
required sample size differs across metrics. The tool also displays the largest
required sample size across all metrics. Having a large enough sample size is
important to ensure that the experiment has enough sensitivity to detect
meaningful effects. For more about what affects the required sample size,
see the [power](./design/power) page.
The sample size calculator doesn't take audience targeting into account. If
you are targeting a subset of the population, then the variance of the metrics
might be different for different subsets of the population. Some subsets might
have larger variance, which increases the required number of users to power a
certain [MDE/NIM](./design/effect-sizes), while others might have
smaller variances which could then decrease the required sample size for a
certain MDE/NIM.
### Calculate the Required Sample Size
To calculate the required sample size for an experiment:
1. Configure the experiment as described in the earlier sections.
2. In the **Required sample size** section on the right sidebar, click **Calculate**.
It takes some time for it to calculate the required sample size.
When it finishes, you see the required sample size for each metric on the right sidebar.
In the preceding example, the results show that the first metric requires 77,000 users
according to the set up of the experiment. The second metric requires
407,000 users, and so to power all metrics the experiment requires at least 407,000 users.
### Adjust the Required Sample Size
If the required sample size is too large compared to the available population,
you can either try to expand the population or reduce the required sample size.
To reduce the required sample size, you can do one or more of the following:
* **Increase Alpha setting**. Alpha is the probability of a false positive. A higher
alpha requires a smaller sample size, but means the risk of finding significance
when there really is no effect increases.
* **Lower Power setting**. Power is the probability of a true positive. The
higher the power, the lower the probability of a false negative. A lower power
requires a smaller sample size, but lowers the chance of finding a true
effect. Lower power also increases the risk of sign and magnitude errors (type
S and type M errors). In general, a too low power makes it hard to reproduce
the results of an experiment.
* **Increase metric MDEs and NIMs**. The MDE and NIM are the
effect sizes that you and your stakeholders care about. The larger the MDE and
NIM, the smaller the required sample size.
### Sample Size for New Metrics
When calculating the required sample size for an experiment, Confidence
looks at historical data for the metrics in the experiment.
There needs to be at least 14 days (plus the aggregation window and exposure
offsets) of historical data for the metric in order for Confidence to be able
to calculate the required sample size. For example, if you have an experiment
with a metric that has a 7-day aggregation window and a 7-day exposure offset,
you need at least 28 days of historical data. If there is not enough
historical data, Confidence can't calculate the required sample
size.
## Related Resources
Calculate required sample sizes
Configure MDE and NIM settings
Understand statistical power
Learn about the statistics
# Adjustment for Multiple Comparisons
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/stats/adjustment-multiple-comparisons
The overall shipping decision dictates how to adjust alpha and power for multiple comparisons.
Confidence centers the adjustment for multiple comparisons around the idea of a
decision rule. In an experiment, it's the decision to release or not release a
new feature that the experiment design should control the risks for. The
adjustments vary among metrics, because different types of metrics contribute
differently to the decision rule. The adjustments ensure that the observed alpha
for the binary decision to ship or not is at most equal to the original alpha.
Similarly for power, the observed power level is at least equal to the original
power level across repeated experiments.
## The Overall Shipping Decision
An important feature of the statistical analysis in Confidence is that the
errors that can happen, false positive and false negatives, matter on the
experiment level, and not on the individual metric level. In other
words, the rates at which these errors happen is over repeated
experiments.
From a product perspective, false positives and false negatives exist for the decision to ship a
feature or not.
A false positive is when you ship a feature that truly doesn't have an effect, and a false negative
is when you don't ship a feature that truly had an effect.
Confidence uses a composite [decision rule](/docs/experiments/analyze-results) to
produce an overall recommendation for a shipping decision.
The results must pass the following for a recommendation to ship:
* at least one success metric has evidence of improvement
* all guardrail metrics show evidence of being within acceptable margins
Alpha needs only to be corrected for the number of success metrics, since the requirement
on the guardrail metrics is that they are all simultaneously significant.
To properly control the power level for the shipping decision, we need to
correct the power level used for each individual metric for the number of guardrail metrics.
The multiple comparison adjustments used are:
* Alpha is adjusted using a Bonferroni correction, where the original alpha is divided by the number of success metrics.
* The power level is adjusted using `1 - (1 - power)/(number of guardrails)`.
To configure multiple comparisons adjustment via the API, see [Configure Multiple Comparisons Adjustment](/docs/api/how-to-guides/stats/configure-multiple-comparisons).
## References
* A. Dmitrienko, A.C. Tamhane,, and F. Bretz (Eds.) (2009) "Multiple Testing Problems in Pharmaceutical Statistics" (First ed.), Chapman and Hall/CRC.
## Related Resources
Understand decision rules
Configure alpha and power
Configure success and guardrails
Understand test types
# Missing Values
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/stats/missing-values
Configure how Confidence handles entities without measurements in your experiment metrics.
It's common that some of your users don't have measurements. For example, if you measure
the time it takes to load a particular part of your app, a user who never visits that part
has a missing value. This section describes how to handle missing values in Confidence.
## Configure How to Treat Missing Values
In general, there are many ways to handle missing values. For example, you can discard them, replace
them with a specific value, or impute them based on values of other similar users. The best approach
generally depends on what you want to measure. For example, if you measure latency, it's
sensible to just discard the users with missing values. If you measure the minutes of
music played, then users with missing values have played zero minutes of music.
Confidence allows you to configure how to handle missing values for users, or entities, more
generally. By default, Confidence replaces missing values with zero if you are using the `SUM`,
`COUNT` or `COUNT_DISTINCT` aggregations. For other aggregations, Confidence discards users with
missing values by default. You can override this behavior in the **Missing Values** section of a
metric, see below.
Discarding users with missing values can lead to [sample ratio mismatch](../monitoring) for a specific metric.
For example, if the treatment increases the chance that a user visits a particular part of the
app that you want to measure, then the treatment group has fewer users with missing
values than the control group. Because of the sample ratio mismatch, the two
groups might no longer be comparable. In this situation, a bias in which users are included in each group's
metric value can drive the significant differences you detect.
If you want further control over how to handle missing values, you can replace them directly in
the SQL query for the fact table. For example, if you want to replace missing values with a 1
instead of a zero, you can define the measurement column `IFNULL(measurement, 1)` instead of just
passing `measurement` to Confidence.
## Missing Values and Variance Reduction
Confidence estimates the parameters required for variance reduction on the subset of users that
have both a pre-exposure and post-exposure measurement. If this subset of users is a small part of all users
then Confidence disables variance reduction.
## Missing Values and Ratio Metrics
Confidence discards all rows in the fact table that have a missing value for the numerator or
denominator to ensure consistency. If you don't want this behavior, then you can replace
the missing values directly in the SQL query for the fact table.
## Related Resources
Configure fact table SQL queries
Improve metric precision
Detect sample ratio mismatches
Configure metric aggregations
# Sequential Tests
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/stats/sequential-tests
Use sequential testing to view results during the experiment without invalidating the statistical results.
Sequential tests make it possible to analyze results during an experiment without jeopardizing the
statistical integrity. If it's important to analyze results during the experiment, you should
use a sequential test. Sequential tests typically have lower power compared to non-sequential tests.
The downside is that if you end an experiment as soon as a metric is significant in a sequential test, the
resulting effect estimates are often biased. Confidence always runs deterioration checks for your
metrics using sequential tests, even if you choose to view the results after the experiment ends.
Sequential tests allow you to analyze the experiment repeatedly, but typically require more
samples compared to a non-sequential test. Use non-sequential tests for
the primary metrics of an experiment and sequential tests for detecting degradations early to
maximize power. Use a sequential test for analyzing the main results only if it's important
to check the progress during the experiment.
Confidence offers two types of sequential tests: group sequential tests and always-valid
sequential tests.
Group sequential tests are the classical statistical approach to sequential analysis
that adjusts the standard z-test to account for multiple analyses.
Always-valid tests are a new development for sequential tests that
require fewer assumptions. Compared to group sequential tests, always-valid
tests don't require you to give an expected sample size before the start of the experiment.
Because they implicitly correct for an infinite number of analyses, they typically have lower
power compared to group sequential tests.
Group sequential tests tend to have higher power compared to always-valid
tests. They require you to specify an expected sample size up front. Your
estimate of the expected sample size doesn't need to be exact. If you can
give a reasonable estimate, then you should use a group sequential test.
To read more about sequential testing and the trade-off between various methods, read the following
[blog post](https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions/).
## Group Sequential Tests
If you give an expected sample size when setting up the experiment,
Confidence uses group sequential tests to calculate valid statistical results
while an experiment is running.
The group sequential test optimally exploits the dependence between the tests at different points
in time.
It allocates the overall false positive rate that the experiment can spend across the multiple
tests performed over time.
How much of the false positive rate that each analysis spends depends on the amount of information
that's available at that time point relative to the expected amount of information at the end of
the experiment.
## Always-Valid Inference
If you don't give an expected sample size, Confidence can't use the group
sequential test and instead resorts to an always-valid approach. These tests
guarantee that the false positive rate doesn't exceed the intended level, but
usually have lower power than the group sequential tests. This means that it
is harder to find effects.
## Analyze Results Sequentially
Sequential tests are always enabled for rollouts.
To use a sequential testing strategy for an A/B test or an analysis workflow:
1. Go to Confidence and select **A/B Tests** or **Analysis Workflows** on the left sidebar.
2. Select the experiment that you want to analyze sequentially.
3. On the right sidebar, click **Results > Results Settings**.
4. Select **Continuously**.
Use the group sequential test by providing an expected sample size:
* Optional. On the right sidebar, click **Results > Configure Statistics**. Enter the **Expected sample size**.
If the expected sample size is present, deterioration checks use the group sequential test. The same applies even if you choose to view the results after the experiment ends.
To configure sequential testing via the API, see [Configure Sequential Testing](/docs/api/how-to-guides/stats/configure-sequential-testing).
## References
* C. Jennison and B. W. Turnbull (2000) "Group Sequential Methods with Applications to Clinical
Trials," Chapman & Hall/CRC.
* M. Schultzberg and S. Ankargren (2023) "Choosing a Sequential Testing Framework—Comparisons and Discussions," Spotify Engineering Blog, [https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions/](https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions/).
* G. Y. Zou, A. Donner, and N. Klar (2005) "Group sequential methods for cluster randomization trials with binary outcomes." Clinical Trials.
* I. Waudby-Smith, D. Arbour, R. Sinha, E. H. Kennedy, and A. Ramdas (2023) "Time-uniform central limit theory and asymptotic confidence sequences,"
[https://arxiv.org/pdf/2103.06476v8.pdf](https://arxiv.org/pdf/2103.06476v8.pdf).
## Related Resources
Configure sequential testing
Understand test types
Monitor live experiments
Understand experiment results
# Statistical Tests
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/stats/stat-tests
Understand the superiority and non-inferiority tests used in Confidence.
The platform provides tests for differences between means of the treatment
groups and the control group. The success metrics and guardrail metrics tests
are slightly different in their interpretations.
## Superiority Tests
Confidence uses superiority tests for success metrics and for deterioration tests.
A success metric test can be significant or non-significant. Significant means that it's unlikely to
find the observed difference of means between the groups if there were no effect. All success
metric tests are against the null hypothesis of zero. Three types of tests are available for
success metrics.
* **Significant result**: The data shows evidence that the treatment caused a change in the metric.
* **Insignificant result**: The data shows no evidence that the treatment caused a change in the metric.
The statistical hypotheses used in the test are:
* $H_0: \delta = 0$
* $H_1: \delta \neq 0$
where $\delta$ is the treatment effect.
* **Significant result**: The data shows evidence that the treatment caused an increase in the metric.
* **Insignificant result**: The data shows no evidence that the treatment caused an increase in the metric.
The statistical hypotheses used in the test are:
* $H_0: \delta = 0$
* $H_1: \delta > 0$
where $\delta$ is the treatment effect.
* **Significant result**: The data shows evidence that the treatment caused a decrease in the metric.
* **Insignificant result**: The data shows no evidence that the treatment caused a decrease in the metric.
The statistical hypotheses used in the test are:
* $H_0: \delta = 0$
* $H_1: \delta < 0$
where $\delta$ is the treatment effect.
## Non-Inferiority Tests
Confidence uses non-inferiority tests for guardrail metrics.
For non-inferiority tests, the test is against the null hypothesis of NIM (non-inferiority margin).
You must select a direction for a non-inferiority test.
* **Significant result**: The data shows evidence that the metric hasn't decreased by more than NIM in the treatment group.
* **Insignificant result**: The data shows no evidence that the metric hasn't decreased by more than NIM in the treatment group.
The statistical hypotheses used in the test are:
* $H_0: \delta < -NIM$
* $H_1: \delta > -NIM$
where $\delta$ is the treatment effect.
* **Significant result**: The data shows evidence that the metric hasn't increased by more than NIM in the treatment group.
* **Insignificant result**: The data shows no evidence that the metric hasn't increased by more than NIM in the treatment group.
The statistical hypotheses used in the test are:
* $H_0: \delta > NIM$
* $H_1: \delta < NIM$
where $\delta$ is the treatment effect.
## Inferiority Tests
Confidence uses inferiority tests for unintended negative effects in success and guardrail metrics. The inferiority test is testing for a move in the opposite direction than the intended one.
For inferiority tests, the test is against the null hypothesis of zero.
You must select a direction for an inferiority test.
* **Significant result**: The data shows evidence that the treatment caused a decrease in the metric.
* **Insignificant result**: The data shows no evidence that the treatment caused a decrease in the metric.
The statistical hypotheses used in the test are:
* $H_0: \delta = 0$
* $H_1: \delta < 0$
where $\delta$ is the treatment effect.
* **Significant result**: The data shows evidence that the treatment caused an increase in the metric.
* **Insignificant result**: The data shows no evidence that the treatment caused an increase in the metric.
The statistical hypotheses used in the test are:
* $H_0: \delta = 0$
* $H_1: \delta > 0$
where $\delta$ is the treatment effect.
## Relative Values
Confidence performs tests on the absolute values, but lets you give NIMs on a relative scale.
The mean of the baseline group, typically the control group, transforms the relative values into absolute values.
## Tests for Success Metrics
Success metrics always use a superiority test. The test is against the null hypothesis of zero mean difference between the groups.
## Tests for Guardrail Metrics
You can test guardrail metrics in two different ways:
* Use an **inferiority test**. This test evaluates whether there is evidence that the guardrail
metric does **worse** in the treatment group compared to the control group.
* Use a **non-inferioriy test**. This test instead evaluates whether there is evidence that the
guardrail metric does **better than a pre-defined threshold** in the treatment group compared to the
control group.
## Tests for Deterioration
Confidence tests all success and guardrail metrics for deterioration. For
success metrics, this means testing for inferiority and superiority separately.
For guardrail metrics, this means testing for inferiority and non-inferiority if
the guardrail metric uses a non-inferiority test.
## Related Resources
Configure alpha and power
Continuous analysis methods
Improve metric precision
# Variance Reduction (CUPED)
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/stats/variance-reduction
Use pre-exposure data to reduce the variance and increase your velocity.
The variance of your metric plays an important role when analyzing an experiment.
With a higher variance, you require more samples to separate the signal from the noise.
A common approach to reduce the variance of a metric is to predict the current measurement using earlier measurements.
If the earlier measurements come from before the start of the experiment, they can adjust for individual variation
that the treatment itself doesn't affect.
Confidence lets you use historical metric values to reduce the variance.
Statistical comparisons, like when comparing metric outcomes between two or more
groups in an experiment, are uncertain. Statistical theory
describes this uncertainty so that it's possible to
conclude that one treatment was superior to another. A standard
comparison of means overlooks the fact that, typically, a large chunk of the
variation in the means is in fact predictable. Consider an experiment on users,
where the metric of interest is time spent in the app. Across users, the amount
of time spent in the app over consecutive weeks is usually fairly highly correlated.
This means that a user who spends three hours a day in the app this week is
likely to spend a sizable amount of time in the app next week as well.
Experimental studies often apply covariate or regression adjustment to reduce the variance
and increase the precision.
Deng et al (2013) popularized covariate adjustment in online experimentation.
Their method is commonly referred to as CUPED.
For the adjustment approach to be valid, the experiment must not influence the data used for the adjustment.
In online experimentation, such data is often available.
Any data computed before the unit (such as a customer) entered the experiment is valid.
The more strongly the pre-exposure data correlates with the post-exposure outcomes, the larger the reduction in variance.
## Variance Reduction for Comparisons of Means
The variance reduction method implemented in Confidence for comparisons of means is the "full
regression adjustment" estimator discussed by Negi and Wooldridge (2021).
The method is more precise than the original CUPED approach. Negi and Wooldridge propose to:
* Regress $Y$ on $1, X$ separately for treatment and control, where $Y$ is the outcome, and $X$ is the pre-treatment variable.
* Estimate the treatment difference by $\hat{\Delta}_{VR} = (\bar{Y}_1-\bar{Y}_0)+(\bar{X}-\bar{X}_1)\hat{\beta}_1-(\bar{X}-\bar{X}_0)\hat{\beta}_0$,
where $\hat{\beta}_i$ is the estimated slope from the regressions, $\bar{X}_i$ and $\bar{Y}_i$ are the sample means for each group, and $\bar{X}$
is the overall sample mean of $X$.
Confidence reports both the adjusted and unadjusted estimates of the sample
means. The variance-reduced adjusted estimate is
$\bar{Y}_i-\hat{\beta}_i(\bar{X}_i-\bar{X})$ for each group $i$, and the
unadjusted estimate is $\bar{Y}_i$.
In experiments with multiple treatment groups, the control group's
variance-reduced adjusted mean estimate differs between comparisons. This
happens because the overall sample mean of the pre-treatment variable,
$\bar{X}$, uses only the data from the groups involved in the specific
comparison. Depending on which treatment group you are comparing to the control
group, $\bar{X}$ changes. Regardless, the variance-reduced treatment effect
estimate $\hat{\Delta}_{VR}$ is generally a more precise estimate than the
unadjusted estimate, providing a more reliable measure of the treatment's impact
on the metric of interest.
## Variance Reduction for Comparisons of Ratios
The approach for reducing the variance through use of pre-exposure data
resembles the method described in the earlier section when the metric of
interest is a ratio metric. Confidence uses the method described by Jin and Ba
(2023). Let $Y_i$ and $Z_i$ be the values for the numerator and denominator for
unit $i$. For example, $Y_i$ could be the total number of searches for user $i$,
and $Z_i$ their number of sessions. The ratio of interest is the group-level
ratio $\sum_{i=1}^n Y_i/\sum_{i=1}^nZ_i$. A difference in ratios between the
groups estimates the treatment effect:
$$
\hat{\Delta} = \frac{\sum_{i\text { in treatment}}Y_i}{\sum_{i\text { in treatment}}Z_i} - \frac{\sum_{i\text { in control}}Y_i}{\sum_{i\text { in control}}Z_i}
$$
To reduce the variance, Confidence applies regression adjustment separately to
each of the four terms in the expression. Ultimately, this leads to adjusted
estimates of the ratios in the two groups and a new estimate of the treatment
effect. The adjustment reduces the variance of the ratios of the two groups, and
the uncertainty surrounding the treatment effect is lower.
## Variance Reduction Rate
Variance reduction adjusts the comparison and reduces the variance as a result.
Ultimately, the variance reduction rate summarizes the size of the reduction:
$$
\text{Variance reduction rate} = 1 - \frac{\text{variance with variance reduction}}{\text{variance without variance reduction}}.
$$
## Relative Values
When using variance reduction, reported relative values use the unadjusted
estimates in the denominator. For a comparison of means, the reported relative
value is $\hat{\Delta}_{VR}/\bar{Y}_0$.
## Interpret Results with Variance Reduction
Interpret the reported treatment effect for a metric that uses variance
reduction in the same way as for a metric that doesn't. The pre-exposure data
helps produce a better signal of the treatment effect. Because the
variance-reduced treatment effect differs from the unadjusted treatment effect,
it can occasionally lead to a different conclusion than the unadjusted effect.
## Use Variance Reduction
Confidence enables variance reduction by default.
To turn on variance reduction for a metric:
1. Go to Confidence and select **Metrics** on the left sidebar.
2. Select the metric you want to enable variance reduction for and click **Edit metric**, or [create a new metric](../metrics#create-a-metric).
3. Expand **Advanced options**.
4. Ensure that **Variance reduction** is checked.
5. Optional. Select an aggregation window for the pre-exposure data.
The metric uses the same measurement, but before exposure, to reduce the variance. Variance reduction is accounted for
in the required sample size calculation by relying on historical patterns of how much the variance can be reduced.
To configure variance reduction via the API, see [Configure Variance Reduction](/docs/api/how-to-guides/stats/configure-variance-reduction).
## References
* A. Deng, Y. Xu, R. Kohavi and T. Walker (2013) "Improving the sensitivity of online controlled
experiments by utilizing pre-experiment data," WSDM '13: Proceedings of the sixth ACM international
conference on Web search and data mining.
* Y. Jin and S. Ba (2023) "Toward Optimal Variance Reduction in Online Controlled Experiments."
Technometrics.
* A. Negi and J. M. Wooldridge (2021) "Revisiting regression adjustment in
experiments with heterogeneous treatment effects", Econometric Reviews.
## Related Resources
Configure variance reduction settings
Understand test types
Configure experiment parameters
# Treatments
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/treatments
Treatments are the variants that you want to test in your experiment.
To configure treatments for an experiment, you first need to select the flag to use.
To select the flag, click **+ Add control** at the top of the edit page. After you add the first treatment, click **+ Add treatment** to add more.
After you have added a flag you can add treatments to the experiment. The first
variant you add is the control group. The next variants you add are treatment
groups. You can reorder the variants by dragging them left and right to select a
different control. You can read more about flags and variants on the
[flags](/docs/how-to-guides/create-flag) page.
## Treatment Sizes
The treatment size is the size of each variant in the experiment. For example,
if you run an experiment with one control group and two treatment groups, you may split your
sample so that 40% of the users are randomly allocated to the control group, 40% to
the first treatment group, and the remaining 20% to the second treatment group.
Divide the sample evenly between the variants whenever possible. This gives you the highest power.
Do this by clicking **Split weights evenly**.
While an even split is optimal from a statistical perspective, there are cases
where it might be difficult to achieve in practice.
If you're testing a risky change for example, you may worry about
degrading the experience for your most valuable users and want to expose only a
small fraction of them. In this case there are two ways to mitigate the risk
and keep your treatments even:
* You can run the test with an even split on a less risky part of the population
(for example, new users, free users or target a specific country)
* You can lower the allocation of the experiment to reduce the total population
you are targeting and have an even split that would expose fewer users. See
[Audience](./audience) for more information.
You need to be careful to ensure that the results are still generalizable. The statistical
tests let you conclude causally whether a change had an effect on
the metric that you observe for the population you are testing on. Testing on a
subpopulation could be a good first step to evaluate the risk before testing on
the rest of the population.
## Related Resources
Set up flag variants for treatments
Configure experiment audience
Configure experiment parameters
Run your experiment
# Unauthenticated Users
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/unauthenticated-users
Run experiments that randomize unauthenticated users and track their behavior after they authenticate.
Many websites serve both unauthenticated and authenticated users.
A common situation in experimentation is that you want to randomize unauthenticated users, but follow
the same users and measure their behavior after they authenticate.
This type of A/B testing is popular when experimenting on sign-up flows.
For example, you make changes to the sign-up flow and measure the impact on engagement when
authenticated.
To run an experiment like this, you need to be able to:
1. Randomize users that haven't been assigned an identifier yet.
2. Describe the relation between unauthenticated and authenticated users.
3. Ensure that an unauthenticated user gets the same experience after authenticating.
## Randomize on Unauthenticated Users
You can randomize on unauthenticated users by having a *required cookie*, which is a website
cookie that's essential for the site to work. The first time a user visits the site you can
generate a UUID as an identifier for the user, and store it in a required cookie. You then pass this
identifier to Confidence and use it as the randomization unit in your experiments. The user may still
delete the cookie, get a new identifier, and potentially be present in multiple treatments resulting
in biased estimates.
## Measure Unauthenticated and Authenticated Users in the Same Experiment
You can measure both unauthenticated users and authenticated users by adding an entity
relationship. The relationship maps the IDs of unauthenticated users to the IDs of authenticated
users. After you have set up an entity relationship you can add metrics for both authenticated users
and unauthenticated users. For example, the entity relationship allows you to measure user behavior
both before and after sign-up. Read more about [entity relationships](/docs/metrics/entity-relation-tables).
Exposure is the timestamp of the first assignment of the unauthenticated identifier.
Metrics for authenticated users measure behavior relative to the exposure timestamp of the
unauthenticated identifier that the entity-relation table maps the user to.
## Consistent Experience After Sign-up
You can ensure that a user gets a consistent experience after authenticating by continuing to pass the
cookie even after the user has authenticated. This way all rules that target the cookie
continue to match even after authentication, and any rule that targets the authenticated ID can
now randomize on that instead.
## Related Resources
Connect unauthenticated to authenticated users
Step-by-step entity relation guide
Configure entity types
Configure randomization settings
# A/B Tests
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/workflows/abtests
An A/B test is a randomized controlled trial. Units of interest, often users or customers, receive one or more treatment variants through randomization.
Compared to other types of experiments, the distinguishing features of A/B
tests are that they:
* can have multiple [treatments](/docs/experiments/treatments), which is sometimes referred to as an A/B/n test
* use both [success and guardrail metrics](/docs/experiments/metrics) to identify experiences that improve some metrics without negatively impacting others
* let you learn and find promising ideas
* have a fixed [allocation](/docs/experiments/audience) that doesn't change
* can use either a [fixed or sequential design](/docs/experiments/statistical-settings), where you view results upon conclusion or continuously during the experiment
The goal of an A/B test is to decide if the change has a positive or negative effect on the
experience as measured by the test's metrics. If the change has a positive effect, distribute
the variant to everyone using a [rollout](./rollouts). A rollout lets you gradually increase how
widely to distribute the variant.
Most A/B tests aim to test product changes with the goal of understanding whether you should roll
out the changes, or if they need further development.
A learning experiment is another type of A/B test that aims to learn about user behavior or to
measure a strategic baseline for the product.
This learning is typically achieved by removing a product or feature from the experience or degrading the
experience in some other way. Such a test helps inform future product prioritization
by breaking down which parts of the existing product have the most impact on
user behavior or the business.
Learning experiments can also be exploratory and only aim to find if a certain variant has a causal relation to an outcome
regardless of direction.
## The Anatomy of an Experiment
An A/B test has different parts. This section gives a high-level overview of these concepts.
### The Hypothesis is the Product Foundation of the Test
A [hypothesis](/docs/experiments/design/hypothesis) is a specific assumption that can be conclusively tested when
subjected to an experiment, and is the basis for a good experiment. It guides
the experiment from a product perspective, and makes the anticipated impact
and value of the experiment clear.
### A/B Tests Distribute Different Experiences Through Variants
An A/B test evaluates how users react after exposure to a new experience.
Variants describe the different user experiences you test.
For example, there could be different variants of a button color.
One variant sets the button color to red, another to blue.
A variant in an experiment is often referred to as a treatment.
These variants often introduce new features, innovations, or changes that should improve the
experience for the user.
Typically, an experiment has one variant representing the current default (in production)
experience, usually called control or the control treatment.
### Randomization Makes Differences Causal
Users in an experiment are randomly assigned a variant.
The variant is the only difference in the experience between the control and treatment groups.
Because of randomization, the different treatments explain any observed change in behavior.
If the treatment group outperforms the control group on the target metric, the
treatment variant improves the user experience.
Randomization ensures that the groups are similar. External factors, such as seasonality, other
feature launches, and competitor moves, affect control and treatment evenly and have no impact on
the results of the experiment.
The treatment effect estimated in an A/B test is only valid for the time of the test.
The estimated effect doesn't necessarily generalize to other future points in time.
The same treatment can have a widely different impact depending on when you run the test.
For example, recommending Christmas songs in July might not have the same effect as in December.
The randomization only ensures that the groups are similar during the experiment.
### Metrics Measure the Effect of the Treatments
Every A/B test needs at least one metric.
These metrics help prove or disprove the hypothesis and to make a business decision based on the
outcome of the test.
In other words, your metrics help answer whether the change is good enough to release widely.
Confidence supports two types of metrics:
1. [**Success metrics**](/docs/experiments/metrics#success-metrics) are metrics that should improve with the treatment
2. [**Guardrail metrics**](/docs/experiments/metrics#guardrail-metrics) are metrics that don't need to improve, but shouldn't deteriorate
It's common and strongly recommended to use both success and guardrail metrics.
The reason is to guard against, for example, cannibalization.
An experiment may want to increase engagement in a new feature, but not by cannibalizing the engagement in another feature.
In this case, the engagement in the new feature would be the success metric, while the engagement in the related feature is
the guardrail metric.
### Statistical Analysis Tells the Answer
Experimentation uses [statistical analysis](/docs/experiments/stats/stat-tests) to reach a conclusion.
A statistical test is a formal procedure used to assess whether the observed difference between
two groups is sufficiently large to say that there is an effect.
The goal of the statistical test is to distinguish the actual effect of treatment from that
due to noise from random sampling.
The statistical tests analyze each metric, and ultimately
summarize the results using a [recommendation for the product decision](/docs/experiments/analyze-results).
## Roll Out a Successful Experiment
Convert the A/B test to a rollout when you complete the A/B test and you have a winning variant.
The rollout targets the exact same users, with all the metrics and configuration from the A/B test.
You can scale up to more users if the A/B test used less than 100% of the allocation.
To avoid reassigning users, the control and treatment groups must remain at the same proportions.
For example, suppose an A/B test was running at 10% of the population but had a 50/50 split of control and treatment.
When you increase the rollout percentage to 50%, all users are in either control or treatment.
You can't continue to track metrics when you increase the rollout percentage beyond this point.
[**Read more about rollouts**](./rollouts)
## Experiment Lifecycle
An A/B test moves through different states during its lifecycle. Each state has specific actions available.
| State | Available actions |
| :---- | :----------------------------- |
| Draft | Launch, Archive, Delete, Clone |
| Live | Roll out, End, Clone |
| Ended | Archive, Clone |
### Clone an Experiment
You can clone any A/B test to create a new draft with the same configuration. Clone an experiment to run a similar test without configuring it from scratch. To clone an experiment, open the A/B test detail page and select **Clone** from the top of the page.
The cloned experiment starts as a new draft that you can change before launch.
### Planned and Actual Runtime
The **Planning** section on the A/B test detail page shows information about the experiment's runtime:
* **Planned runtime**: The expected duration of the experiment. Click **Edit** to set or update the planned runtime. For draft experiments, planned runtime shows "Not set" until you configure it.
* **Actual runtime**: The time the experiment has been running, calculated automatically from the launch and end dates.
Comparing planned and actual runtime helps you track whether experiments run longer than expected.
### Delete a Draft Experiment
You can permanently delete an A/B test while it's in draft state. To delete a draft, open the experiment and click **Delete** in the actions section. This action is permanent and can't be undone.
For experiments that you already launched, use **Archive** instead. Archiving preserves the experiment's data and removes it from the active list.
## Related Resources
Step-by-step A/B test tutorial
Configure experiment parameters
Configure success and guardrail metrics
# Analysis
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/workflows/analysis
Analysis lets you analyze experiments that run outside of Confidence. For example, you can re-analyze experiments you ran before using Confidence, to gather all experiments in one place.
In Confidence, an analysis resembles an A/B test.
The main difference is that the experiment is not run in Confidence and is already ended.
An analysis is different from an A/B test in the following ways:
* No continuous monitoring of the experiment
* The treatment variants are not variants on a feature flag in Confidence
* No sample size calculator on the Design page. View the powered effect size on the Detailed results view like for A/B tests.
* You don't launch analysis instances, since the experiment is not live. Select the metrics you want and click **Calculate** to see the results.
## Analyze Optimizely Experiments in Confidence
You can analyze past or current experiments in Optimizely with Confidence.
To do so, you need to [export the decision events](https://docs.developers.optimizely.com/experimentation-data/docs/enriched-events-export)
from Optimizely to a table in your data warehouse.
A decision event is an event that Optimizely records when a visitor is exposed to an experiment.
Decision events in Optimizely correspond to assignments in Confidence.
The information Confidence requires is available in the columns:
* `experiment_id`: column with identifiers of the experiments
* `variation_id`: column with identifiers of the variants
* `visitor_id`: column with identifiers of the entities in the experiments, like users and visitors
* `timestamp`: column with timestamps of the events
Before setting up an analysis, make sure you have configured your
[data warehouse connection](/docs/warehouse-setup/bigquery).
The steps to analyze an Optimizely experiment in Confidence are:
1. [Export](https://docs.developers.optimizely.com/experimentation-data/docs/enriched-events-export)
the [decision events](https://docs.developers.optimizely.com/experimentation-data/docs/enriched-events-data-specification#decisions-1)
from Optimizely to a table in your data warehouse. If you want to analyze a running experiment,
you need to schedule the export to happen at a regular cadence.
2. If you don't have one already, [create an entity](/docs/metrics/entities) in Confidence that identifies the
entity that's recorded in the `visitor_id` column of the decision events table.
3. [Create an assignment table](/docs/metrics/assignment-tables) in Confidence that points to the decision events table.
* Set the exposure key column to `experiment_id`.
* Set the variant key column to `variation_id`.
* Set the entity to the entity you created in step 2.
* Set the entity column to `visitor_id`.
* Set the timestamp column to `timestamp`.
The columns `experiment_id` and `variation_id` must be strings to be selectable
as exposure key and variant. The type of the `visitor_id` must match the primary key type of the
entity you created in step 2, such as a string. The `timestamp` column must be a timestamp.
4. Create a new analysis in Confidence.
* Set the exposure key to the identifier of your Optimizely experiment. This identifier filters
the decision events table to only include the events for the experiment you want to analyze based
on the `experiment_id` column.
* Set the analysis start date to the date the experiment started.
* Click **Add treatment** and enter the identifier for the variant you want to add. This
identifier should match what's in the `variation_id` column of the decision events table. You can
add multiple variants. Set the weights to match the weights you used in Optimizely.
* Click **Add metric** and select the entity and assignment table you created in step 2 and 3.
Click continue.
* Add a metric and click **Continue**.
If you don't have any metrics in Confidence, follow the steps in the metrics quickstart to
[create a fact table](/docs/how-to-guides/create-fact-table) and
[a metric](/docs/how-to-guides/create-fact-table). Use the entity you created in step 2.
## Related Resources
Step-by-step analysis tutorial
Set up assignment data sources
Compare with A/B tests
# Rollouts
Source: https://confidence-auth-testing.mintlify.io/docs/experiments/workflows/rollouts
A rollout is a way to gradually roll out a change or feature to a percentage of your users to ensure it doesn't have any negative impact on your users or your systems.
The distinguishing features of a rollout compared to other types of experiments
is that they:
* can only have two [treatment groups](/docs/experiments/treatments)
* only use [guardrail metrics](/docs/experiments/metrics#guardrail-metrics)
* have a variable allocation that lets you gradually roll out
* only use a [sequential design](/docs/experiments/statistical-settings) where you see results continuously when the rollout is live
The main reasons for gradually rolling something out as opposed to
enabling it for all users immediately are:
* It lets you ensure that the experience you are rolling out is not worsening
the user experience before you roll it out to everyone.
* See comparisons between the group that has the feature you are rolling
out and the group that has the current default experience for any metric.
A rollout uses the same basic functionality as an A/B test.
It allocates users into two groups, a control group and a treatment group, and compares them.
The difference is that in a rollout, you want to roll out
a feature gradually. You might start with a 95/5 allocation with 5%
of eligible units in the treatment group to limit the impact of a potential issue. You then
monitor important metrics to see if you worsen the user experience. If the experiment is going
well, you increase the allocation to get more units in the treatment until you have rolled
it out completely.
## Related Resources
Step-by-step rollout tutorial
Configure rollout metrics
Monitor rollout health
Compare with A/B tests
# Audience
Source: https://confidence-auth-testing.mintlify.io/docs/flags/audience
Reference documentation for audience targeting in rules and segments.
The audience is the set of users or other units that are eligible for a rule
or segment.
## Inclusion Criteria
The audience is a set of filters that describe the conditions for inclusion.
The filters must match the values in the evaluation context that clients use when resolving flags.
For example, if you add `country is Sweden` as an inclusion criterion, you must include `country` in the evaluation context.
The value for `country` in the evaluation context must match `Sweden` to be part of the audience.
You can use four types of criteria to define your audience:
* **Attributes**: Match values in the evaluation context, such as country, platform, or version.
* **Segments**: Reference a pre-defined [segment](./segments) to reuse targeting logic across flags and experiments.
* **Holdbacks**: Include or exclude a random subset of users defined by a [holdback](../surfaces/surface-settings#holdbacks) on a surface.
* **Groups**: Combine multiple criteria with `AND` or `OR` operators.
Filters can vary in complexity. They can, for example, exactly match a user ID,
or be more elaborate and match users in a certain country that are using a specific
browser with a specific version. You select how multiple criteria should logically work
together by choosing the operator to be `AND` or `OR`.
You can add multiple criteria to a group. With groups, you can create arbitrarily
complex inclusion criteria by adjusting the logical operator used between and within
groups.
Learn how to configure inclusion criteria in the [Define the Audience](/docs/how-to-guides/define-audience-criteria) guide.
## Allocation
The allocation is the percentage of the targeted audience (after the inclusion criteria) that you
want to allocate.
You set the allocation as a percentage of the total audience size.
For example, if the audience size is potentially 2,000, a 50% allocation allocates 1,000 users.
You can increase the allocation for live A/B tests. Decreasing is not allowed for live A/B tests.
Control the allocation for rollouts by setting the [rollout reach](/docs/quickstarts/launch-rollout).
## Randomization
Confidence uses randomization to assign variants to users. To randomize, Confidence needs to know which field in the evaluation context
it should take the value from. If you do not specify a randomization field,
Confidence uses the value from the `targeting_key` field in the evaluation
context. If the field is not present in the evaluation context, the rule
doesn't match and users are not assigned an variant by the rule.
The randomization field is often an identifier that has a value that is unique
for each user. For example, if you have a field called `user_id` in the
evaluation context, you can use that field for randomization.
For applications that have both authenticated and unauthenticated experiences,
you should pass all known identifiers to Confidence. For example, if you use a
visitor ID for users that haven't authenticated, and a user ID when they
authenticate, pass both the visitor ID and the user ID to Confidence after the
user has authenticated.
When experimenting, select the field that maps best to the user experience
you are experimenting on. For example, if you are experimenting on the
visitors that haven't authenticated, and you want them to have a consistent
experience as they authenticate, you should use the visitor ID for randomization.
Read more about [authenticated and non-authenticated users](/docs/experiments/unauthenticated-users)
The field in the evaluation context that you use for randomization must be a
string or an integer, otherwise Confidence fails to evaluate the rule and
the user is not exposed to the experiment.
## Sticky Assignments
When you enable sticky assignments, Confidence writes all assignments to a
storage that is accessible at resolve time with low latency.
Use sticky assignment for two things
* Pause intake of new entities to an experiment
* Ensure that entities are assigned the same variant throughout an experiment even if some of their targeting
attributes change during the experiment.
With sticky assignment selected, you can select whether you want targeting
criteria to apply after an entity is first assigned a variant.
You control the behavior of the inclusion criteria with the checkbox `Don't enforce inclusion criteria for entities that have been assigned to a
variant`. The images below illustrate the evaluation logic for sticky
assignment with and without this checkbox checked.
You can use sticky assignment with the sidecar resolver by hosting your own sticky assignment table. Read more about the [sidecar resolver](/docs/how-to-guides/setup-local-resolver).
Sticky assignments automatically clean up users after 90 days of inactivity. An entity that resolves again within those 90 days renews its TTL. For sticky assignment with the sidecar resolver, you configure the TTL.
### Paused Intake
When you enable sticky assignment, you can pause and restart intake on your
experiments as many times as you want. With intake paused, Confidence stops
assigning new entities to the experiment, and only the entities that are already
assigned to the experiment keep being assigned to the experiment. This
makes it possible to observe outcomes for the assigned entities over
time without letting any more entities be assigned to the experiment. When you pause intake, Confidence returns the
same variants for the assigned users by reading from the table.
Use sticky assignments to evaluate the long-term impact when the cost of exposing new entities is costly. For example, if there is a cost associated with each user entering the experiment, you can expose the number of users you need to evaluate the long-term impact and then pause intake.
### Entities with Target Attributes that Change
Sometimes you expect the targeting attributes used to define the inclusion criteria to change as a consequence of the treatment in the experiment. In
experiments on, for example, conversion, you might want to only include users
that are not converted to begin with, and use an inclusion criterion
like 'is not converted'. If a user converts during the experiment, you
might want to keep serving them the same variant to be able to measure the
longer term impact of the variant. With sticky assignment, if you check the
`Don't enforce inclusion criteria for entities that have been assigned to a
variant` - checkbox, Confidence does not evaluate the inclusion criteria on
resolves from already assigned users.
## Exclusivity
Make experiments mutually exclusive to each other by using exclusivity groups.
Exclusivity groups lives on surfaces. Read more in the [surface documentation](../surfaces#exclusivity-groups).
## Related Resources
Configure targeting criteria
Understand rule types and evaluation
Create reusable user groups
Set up mutual exclusion
# Clients
Source: https://confidence-auth-testing.mintlify.io/docs/flags/clients
Reference documentation for Confidence clients.
Clients authenticate requests to Confidence from your website, backend service, or mobile app.
Clients are different from API clients. Use **Clients** in your
application to talk to Confidence to resolve flags or publish events. **API
clients** talk to Confidence's management APIs, for tasks like
creating new flags or managing rules.
## Client Credentials
After you have created a client, you also need to create a client secret that
you use to authenticate requests to Confidence.
Use the client secret when you initialize a Confidence client SDK, or when
you make API requests to Confidence.
### Integration Type
When creating a client credential, you select an integration type that matches
how your application uses the credential.
* **Backend**: Use for server-side applications where the credential will be
kept secure. Backend credentials have access to more features.
* **Frontend**: Use for client-side applications (web/mobile apps). Frontend
credentials may have restricted access for security purposes.
## Credential Rotation
For security purposes, you may need to rotate credentials periodically. The process
involves creating a new credential, updating your application to use it, and then
deleting the old credential.
## Environments
Environments let you separate flag behavior across different deployment stages
like development, staging, and production. You can use the same client across
multiple environments while configuring different credentials for each
environment.
### How Environments Work
Environments work through two configuration points:
1. **Client credentials**: You can associate each credential with specific
environments. When you link a credential to an environment, the credential
identifies which environment the resolve request comes from.
2. **Flag rules**: You can limit each rule to specific environments. **Rules
without environment restrictions apply to all environments.**
When a client resolves flags, Confidence checks which environment the
credential belongs to, then evaluates only the rules that apply to that
environment.
### Associate Credentials with Environments
When creating a client credential, you can specify which environments the
credential applies to. A credential without environments only matches rules
that also have no environments specified.
| Rule Environments | Credential Environments | Rule Enabled |
| ----------------- | ----------------------- | ------------ |
| Empty | Empty | Yes |
| Empty | Has values | Yes |
| Has values | Empty | No |
| Has values | Has matching value | Yes |
| Has values | No matching value | No |
This approach allows you to reuse a single client across environments.
Create separate credentials for each environment so that flags resolve based
on the environment of the requesting credential.
Navigate to [Clients](https://app.confidence.spotify.com/admin/clients) in the Admin section.
Select the client to show its credentials.
Click the **more options** icon in the top-right corner of the credential.
Select **Edit environments** and enable the environments you want for this credential.
Configure your flag rules to apply only to specific environments in the rule creation flow.
## Related Resources
Step-by-step client management guide
Integrate clients in your application
End-to-end setup tutorial
Run a local resolver for performance
# Context Schema
Source: https://confidence-auth-testing.mintlify.io/docs/flags/context-schema
Map evaluation context fields to names and data types so Confidence can use them for targeting and autocomplete.
You find the context schema under **Admin > Clients** and the **Context Schema** tab.
A context schema lets you map a field in the evaluation context of a feature
flag to name and a data type. Context schemas let Confidence know
what can exist in the evaluation context before Confidence has detected it. Confidence automatically
keeps track of fields you include in your evaluation context and uses them to
populate relevant options in the app. For example, when you define your target audience.
Confidence by default maps the field `visitor_id`
to the entity Visitor as this is an identifier the Confidence flag SDKs automatically emit.
This means that when you create a rollout
targeting the entity Visitor, Confidence randomizes treatment assignment
based on the `visitor_id` value in the evaluation context
of the feature flag.
## Mark Fields as Non-PII
You can mark context fields as not containing personally identifiable information.
When you mark a field as non-PII, Confidence samples the values
that your application sends for that field. This helps you in two ways:
* **Discover field values**: See what values your application sends for a field.
* **Easier targeting**: Get autocomplete suggestions when you create targeting rules.
Only mark fields as non-PII if they truly don't contain personally identifiable information.
Confidence samples and displays these values to help with targeting, so you must ensure
the field doesn't contain sensitive data like email addresses, phone numbers, or user IDs.
### How Confidence Samples Values
Confidence collects sample values from flag resolution requests for fields marked as non-PII.
Confidence only samples values when it detects that a field has low cardinality, which means
the field has a limited set of distinct values. For example, fields like `country`, `platform`,
or `subscription_tier` are good candidates.
To protect user privacy, Confidence only includes a value in the samples if many users send
that same value. If only a few users send a specific value, Confidence doesn't sample it because
the value might be specific to those users and could reveal identifying information.
The sampled values appear as autocomplete suggestions when you create targeting rules.
For example, if you mark the `country` field as non-PII, you see
a list of countries that your application has sent, making it easier to create
targeting rules.
### Mark a Field as Non-PII
Navigate to [Confidence](https://app.confidence.spotify.com).
On the left sidebar, select **Admin** and click **Clients**.
Select the client you want to update.
Go to the **Context Schema** tab.
Find the field you want to mark as non-PII.
Select the checkbox in the **Non-PII** column for that field.
Confidence starts sampling values for this field during flag resolution.
## Related Resources
Use context fields for targeting
Understand how rules use context
Test context-based resolution
Map context fields to entities
# Flags
Source: https://confidence-auth-testing.mintlify.io/docs/flags/create-flags
Reference documentation for flag schemas and variants in Confidence.
Flags let you remotely control the behavior in your application,
like a website or a mobile app.
## Flag Schema
Flags in Confidence don't just describe a boolean decision to enable or disable
a feature. Instead, flags have multiple properties that you use to control
multiple aspects of the experience. The flag schema defines the available
properties and their data type. The schema lets applications consume the flag
value while knowing what to expect.
Confidence supports the following data types in the schema:
| Type | Description | Example values |
| :------ | :--------------- | :--------------------------- |
| String | A string | `"HELO"` |
| Integer | An integer | `42`, `199932` |
| Double | A double | `3.14`, `50.0` |
| Boolean | Boolean | `true`, `false` |
| Struct | Nested structure | `{ age: 23, country: "SE" }` |
## Variants
A variant is a named set of values for the properties of the flag. Workflows, like A/B tests and
rollouts, use variants to give users different experiences.
If a variant doesn't specify the value of a property, the SDK uses the
default value specified when a client resolves the flag.
Each variant displays a status badge to help you understand whether it's actively used:
* **In use**: At least one rule references the variant.
* **Unused**: No rule references the variant.
## Client Association
Not all clients should have access to all flags. For example, some
flags might be sensitive and should only be available to clients that are
running in an trusted environment.
Mobile apps and websites batch resolve flags to reduce the number of requests
made to Confidence. Associate a flag with as few clients as possible to limit
the number of resolved flags. Doing so also reduces costs.
## Permissions
You can manage access permissions for individual flags. Click **Permissions** at the top of the flag detail page to control who can view or edit the flag. Restricting access helps protect sensitive flags in larger teams.
When you share a flag, related resources like segments are automatically shared with the same permissions.
## Update History
Click **Updates** at the top of the flag detail page or **View updates** below the flag heading to see a chronological history of changes made to the flag. The update history shows who changed what and when, which helps with auditing and debugging.
## Flag Activity
The flag detail page sidebar shows information about flag usage.
### Resolves
The **Resolves** section in the sidebar displays a time-series chart of flag resolution activity. It shows how often clients resolve the flag over time. Use the **Day** and **Week** toggle to switch between daily and weekly views.
Use the resolves chart to:
* Monitor flag usage patterns
* Identify unused flags that you can archive
* Troubleshoot resolution issues
### Status
The **Status** section in the sidebar shows when the flag was last applied by a client (for example, "Last apply was 2 hours ago"). Use this to understand flag lifecycle and identify stale or actively used flags.
## Archive Flags
When a flag is no longer needed, you can archive it. Archiving a flag prevents
clients from resolving and using it.
## Related Resources
Step-by-step flag creation guide
Control how variants are assigned
End-to-end flag setup tutorial
Control who can view or edit a flag
Integrate flags in your application
# Data Transfer
Source: https://confidence-auth-testing.mintlify.io/docs/flags/data-transfer
Understand how your data flows through Confidence Flags.
Confidence offers two options for flags: a managed resolver and a sidecar
resolver. With the managed resolver, Confidence operates the flag resolver
service for you. With the sidecar resolver, you operate the resolver. Confidence
doesn't persist any user-identifiable information on the Confidence side. An
identifier and a context transiently passes through the flags service so that
Confidence can give aggregate information.
## Data That Confidence Persists
The information that you pass to the resolver is:
* A unique ID to randomize on.
* A context with information to use for targeting. For example, `country` or `is_employee`.
Information that Confidence persists is:
* The number of resolves and the timestamp of the last resolve.
* The number of applies and the timestamp of the last apply.
* The names of the fields available in the context. The values of the fields in the context are
never stored.
The information that Confidence persists in your data warehouse is:
* The ID used for randomization.
* The context used in the resolve.
* The name of the flag.
* The name of the rule that determined what variant to give.
* The name of the variant that the user was exposed to.
## Managed Resolver
If you use the managed resolver, Confidence operates the flag resolver for you.
The following figure gives more details on how data flows through Confidence Flags in this case.
The sequence of events are:
* A client resolves a flag by passing an ID and a context. It receives a variant.
* Confidence passes the ID and context to other internal services, but without storing the ID or
the values in the context. It only stores the schema of the context and aggregated information about
the number of resolves.
* When the client uses the variant, it also applies it by calling the apply endpoint. The apply
event similarly passes through Confidence, but only storing the number of applies.
* Confidence eventually writes the event to your data warehouse. At this point, neither the ID nor
the context is present in any part of Confidence.
## Sidecar Resolver
When using the sidecar resolver, you operate the flag resolver.
The following figure gives more details on how data flows through Confidence Flags in this case,
and how the sidecar communicates information back to Confidence.
The sequence of events are:
* A client resolves a flag by passing an ID and a context to the sidecar resolver. It receives a variant.
* At regular time intervals, the sidecar resolver sends the number of resolves and the schema of the
context to Confidence.
* When the client uses the variant, it also applies it by calling the apply endpoint. By default,
the sidecar resolver forwards the apply event to Confidence, which only stores the number of applies.
* Confidence eventually writes the event to your data warehouse. At this point, neither the ID nor
the context is present in any part of Confidence.
You can set `CONFIDENCE_SEND_APPLY_LOGS` to `false` to keep apply data within your network. The
sidecar continues to accept apply requests, but doesn't send assignment events, apply distinct
counts to Confidence. Resolve telemetry, aggregate operational metadata such as the apply request
rate, and resolver-state synchronization remain enabled.
Disabling apply logs degrades Confidence functionality. Confidence can't observe which entities
were exposed to each variant. Experiments and rollouts that use flags resolved by this sidecar
won't work correctly: exposure counts, experiment results, and rollout monitoring will be missing
or incomplete. Only disable apply logs if you accept this loss of functionality or provide
equivalent assignment data through another source.
Read more about how to [set up a local sidecar resolver](/docs/how-to-guides/setup-local-resolver).
## Related Resources
Deploy the sidecar resolver
Resolver configuration options
Configure your data warehouse
# Rules
Source: https://confidence-auth-testing.mintlify.io/docs/flags/define-rules
Reference documentation for rules that assign variants to users.
Rules decide which variant to use for a particular client and user
at a given time.
A/B tests and rollouts create rules to assign users to the different treatment groups when you
launch them. You can't change these rules directly in the list of rules.
If no rule matches, the client uses the default value of the flag defined in the client.
Avoid rules that overlap. For example, if you have a rule that
matches all users, and another rule that matches a subset of users, the second
rule never matches.
## Types of Rules
You can create rules directly on a flag, or through A/B tests and rollouts.
### Individual Targeting Rules
Use individual targeting rules to force a specific variant for a set of users. For example,
use individual targeting rules for testing and debugging.
Individual targeting rules let you:
* Select a variant to serve
* Choose an attribute to match against (such as user ID)
* Specify a list of values that should receive the variant
Place individual targeting rules at the top of the list of rules. This ensures they
evaluate first and override other rules. Putting them at the top also
increases their visibility, and lets you quickly turn them off when they are no
longer needed.
### Conditional Targeting Rules
Conditional targeting rules are the most flexible type of rule. Use when you want to assign
a variant to a set of users based on conditions and percentage allocation.
Conditional targeting rules let you:
* Select a variant to serve
* Set what percentage of traffic should receive the variant
* Choose an attribute to use for randomization
* Define matching conditions based on context attributes
You can optionally enable **sticky assignments** to make variant assignments persist for targets even if the rule changes.
### A/B Test and Rollout Rules
When you create an A/B test or rollout, Confidence automatically creates rules to assign users to treatment groups. These managed rules appear in the rules list but you can't edit them directly. Instead, manage them through the A/B test or rollout interface.
* **A/B test rules**: Created when you launch an A/B test. The A/B test manages traffic allocation between control and treatment variants.
* **Rollout rules**: Created when you launch a rollout. The rollout manages the gradual increase of traffic to the new variant.
## Environment Scope
Each rule card displays an environment tag that shows which environments the rule applies to (for example, "All environments"). This tag gives you a quick overview of the rule's scope without opening the rule details.
## Enable Rules
To enable or disable a rule, click the toggle on the rule card.
## Order of Rules
The order of the rules defines in what order to evaluate them. The first rule that matches a client
and a user decides the variant to use. You can reorder the rules by dragging them up and down in
the list.
Newly created rules are at the bottom of the list to not impact already
running tests and rollouts.
Changing the order of rules can potentially have a big impact on the behavior
that users experience. It may also impact the results of A/B tests and rollouts.
Rules for A/B tests and rollouts are always added to the end of the list to not impact already
running tests and rollouts.
## Related Resources
Step-by-step rule creation guide
Configure targeting and allocation
Manage rule priority and order
Verify your rules work correctly
# Flag Cleanup
Source: https://confidence-auth-testing.mintlify.io/docs/flags/flag-cleanup
Reference documentation for flag cleanup scans in Confidence.
Flag cleanup scans your flags and connected repositories to find flags that are no longer pulling their weight. Each scan produces a list of stale flags and takes action on them—archiving them in Confidence, opening cleanup pull requests in your repositories, or both.
## Scans
A scan is a single analysis run. You can trigger scans manually or configure a schedule so they run automatically.
**Manual scans** let you run cleanup on demand—for example, after a feature ships or at the end of a sprint.
**Scheduled scans** run on a recurring interval without manual intervention. Use scheduled scans to keep flag debt from accumulating.
## Signal Types
Each scan looks for three signals:
### Inactive Flags
Flags that haven't seen any apply or resolve call in the last 30 days. Because no client is using them, they're safe to remove from code and archive.
### Archived Flags in Code
Flags you've already archived in Confidence. Clients can no longer resolve them, but their references can remain in your codebase as dead code. A scan finds those lingering references so you can remove them.
### Single Variant Flags
Flags that have returned only one variant to clients over the last 7 days. Because they're no longer splitting traffic, they behave like a constant—you can replace each reference with the value the flag has been serving.
## What Happens to Detected Flags
Cleanup behavior depends on whether the flag appears in your connected repositories.
### Automatic Archive
If an inactive flag has no references in your connected repositories, there's nothing to remove from code. With **Auto archive** enabled, Confidence archives the flag for you automatically.
Automatic archiving only applies to inactive flags with no code references. It does not archive flags that are still referenced, or single variant flags.
### Cleanup Pull Request
If a stale flag is still referenced in code, a coding agent opens a pull request that removes the flag and replaces each reference with the appropriate constant value:
* **Single variant flag**: replaced with the variant's value
* **Inactive or archived flag**: replaced with the call-site default
## Integrations
Scans work without any integrations. Without a GitHub connection, Confidence detects stale flags and—with automatic archiving enabled—archives inactive ones automatically.
Connecting a GitHub repository adds code analysis, so scans can find references in your source code and trigger cleanup pull requests. Opening those pull requests requires a coding agent.
## Related Resources
Run a scan and review the results
Manually archive a flag that's no longer needed
Flag schemas, variants, and lifecycle
# Introduction to Flags
Source: https://confidence-auth-testing.mintlify.io/docs/flags/introduction
Confidence Flags is a managed service for feature flagging. It provides sophisticated targeting and coordination capabilities.
In Confidence, a flag is a way to control what experience a user should
receive. A flag has multiple variants, one for each intended experience, and
these variants describe the specifics of the experience.
This video gives a quick overview of how feature flags work in 2 minutes and 2 seconds.
Clients resolve flags to decide which experience to serve to a user. Common
types of clients are mobile apps, websites, and backend services. A set of
rules define what variant to assign to which users in what situations. The
resolver evaluates the rules and decides which variant to return to the client
using data in the evaluation context. The evaluation context is data about the
user, such as their country, age, or subscription status, and the environment
the client is running in, such as the browser type. The client feeds the
context to the resolver in the resolve request. If the context meets a rule's
targeting condition, the client receives the variant specified in the rule.
## Flag Anatomy
A flag has a name and a value. In Confidence, the flag value is a structure
with named properties. Think of it as a JSON object. This makes it possible to
control multiple aspects of the behavior of a client with a single flag. Flags
have a schema that describes the structure of the value: available properties
and their data types. Variants give a name to a value of the flag which defines
a possible behavior of the thing the flag is controlling.
Imagine a flag that controls the various aspects of Spotify's home screen.
The flag has a name, `home-screen`, and the value has properties:
* The size of the title (`title-font-size`).
* Show the settings button or not (`show-settings`).
* Show shortcuts or not (`show-shortcuts`).
* Number of shortcuts to show (`shortcut-count`).
The schema for this flag is as follows:
| Property | Type | Description |
| :---------------- | :------ | :----------------------------------- |
| `title-font-size` | String | Size of the title font. |
| `show-settings` | Boolean | Whether to show the settings button. |
| `show-shortcuts` | Boolean | Whether to show shortcuts. |
| `shortcuts-count` | Integer | How many shortcuts to show. |
The flag has two variants: `default` and `large-title`. The `default` variant
has the following values:
| Property | Value |
| :---------------- | :-------- |
| `title-font-size` | `"small"` |
| `show-settings` | `true` |
| `show-shortcuts` | `true` |
| `shortcuts-count` | `9` |
The `large-title` variant has the following values:
| Property | Value |
| :---------------- | :-------- |
| `title-font-size` | `"large"` |
| `show-settings` | `false` |
| `show-shortcuts` | `true` |
| `shortcuts-count` | `6` |
With the flag and its variants in place, a client can resolve the flag to a
value. For example, a website can resolve the flag to the `large-title`
variant, and show a large title, hide the settings button, and show six
shortcuts.
When you use a flag to control an experience, the *flag is applied* and the
client emits a flag applied event. The resolver writes flag applied events, in
the form of assignment decision records, to a data warehouse for later use in
analysis of experiments.
## Client Types
Resolving a flag to a value and applying the flag value to control an experience
are two distinct operations. The type of client decides when these operations
happen.
**Single-user clients**, such as mobile apps or single-page web applications,
resolve multiple flags (1, 2) at the start of a session. They later use locally
cached flag values when using flags throughout the session. The reason
for this is to reduce flickering of the user experience during the session.
When using a flag, the client emits an event (3) that says the flag
value was applied. If a client resolves a flag at the start of the session but
never uses it, it doesn't emit an event. The resolver writes a flag applied event
to a data warehouse for later use in analysis of experiments (4).
**Multi-user clients**, such as backend services, resolve flags as requests
come in (1). Since the clients use flag values directly when rendering the
response, the resolve and apply operations happen at the same time.
Use the option (`apply: true`) in the resolve request to Confidence to
apply the flag as the client resolves it (1, 3). As a consequence, the resolver
writes a flag applied event to a data warehouse for later use in analysis of experiments
(2).
If you are using one of Confidence Flags SDKs then you don't have to worry
about the details of the resolve and apply operations. The SDKs take care of
this for you.
Learn more about the concepts of Confidence Flags:
FlagsClientsRulesAudiencesSegments
## Confidence SDKs
Confidence provides SDKs for flag resolving with support for multiple languages and platforms.
More information about the Confidence SDKs is available in the [dedicated section](../sdks).
# Materialized Segments
Source: https://confidence-auth-testing.mintlify.io/docs/flags/materialized-segments
Reference documentation for materialized segments in Confidence.
A materialized segment is a [segment](/docs/flags/segments) that loads its list of units from your BigQuery instance.
You write SQL, Confidence executes it as the service account you have configured, and you can then use this list of units as targeting for A/B tests, rollouts, or flag rules.
Materialized segments are only available for BigQuery.
## When to Use Materialized Segments
Use materialized segments when you need to target a specific list of users defined by complex queries in your data warehouse. Common use cases include:
* **Beta testers**: Target users who have opted into beta programs
* **High-value customers**: Target users based on purchase history or engagement metrics
* **Cohort analysis**: Target users who signed up during a specific period
* **Custom segments**: Any segment that's easier to define in SQL than with attribute criteria
## How Materialized Segments Work
1. You define a SQL query that returns a list of entity IDs
2. Confidence runs this query against your BigQuery instance
3. Confidence loads the resulting IDs into a segment and builds a [bloom filter](#bloom-filters-and-local-resolution)
4. You can use this segment for targeting in rules, experiments, and rollouts
At resolve time, how Confidence checks segment membership depends on the segment size:
* **Up to \~100,000 entities**: The [local resolve providers](/docs/flags/local-resolver) checks membership locally via a bloom filter in the resolver state. No network call needed.
* **Larger segments or exactness-critical**: When the bloom filter exceeds 512 KB, Confidence excludes it from the resolver state. The online resolver (Confidence backend) queries a key-value store for an exact membership check.
## Bloom Filters and Local Resolution
When a load job completes, Confidence automatically builds a bloom filter, a compact, probabilistic data structure that can answer "is this entity in the segment?" locally.
### How Bloom Filters Work
A bloom filter provides:
* **Definitive no**: If the filter says "not in set," the entity is definitely not in the segment.
* **Probable yes**: If the filter says "in set," there's a negligible chance it's a false positive (typically 1×10⁻⁸).
The [local resolve providers](/docs/flags/local-resolver) receive the bloom filter as part of the resolver state. When a flag rule references a materialized segment, the provider checks the bloom filter locally instead of making a network call.
### False Positives
A bloom filter never produces false negatives. It can produce false positives—a negligible fraction of non-members may match as segment members. For experiments, this introduces no systematic bias since these users still receive randomly assigned variants. For feature rollouts, a negligible number of extra users see the feature early. The Confidence UI shows the false positive rate on the materialized segment page.
### Privacy
Bloom filters use a one-way, non-reversible data structure. You can check whether a key is in the filter, but you can't extract the keys. This means Confidence delivers segment membership information to resolvers without exposing the underlying entity identifiers.
### Automatic Delivery
Confidence builds and delivers bloom filters automatically—no configuration required. Every completed load job produces a bloom filter. The resolver state that local resolve providers fetch includes the filter.
Confidence excludes individual bloom filters exceeding 512 KB from the resolver state. These segments fall back to require a key-value store lookup. There's no cap on the number of bloom filters in a state.
### Backward Compatibility
Older SDK versions that don't support bloom filters silently ignore them and continue requiring a key-value store lookup. Newer SDK versions that don't find a bloom filter for a segment fall back to the same behavior.
## Related Resources
Step-by-step creation guide
Local flag resolution with OpenFeature providers
Learn about standard segments
Configure BigQuery connection
# Segments
Source: https://confidence-auth-testing.mintlify.io/docs/flags/segments
Reference documentation for segments in Confidence.
A segment is a definition of a subpopulation, like a certain cohort
of users.
A segment has no targeting key. Use segments with all targeting keys.
## Segment Components
Segments consist of:
* **Inclusion criteria**: Filters that describe which users belong to the segment
* **Allocation**: The percentage of users matching the criteria to include
* **Exclusivity**: Optional setting to make segments mutually exclusive
Segments are reusable across multiple flags and experiments, making them a powerful
tool for consistent targeting across your feature flag infrastructure.
## Related Resources
Step-by-step segment creation guide
Configure targeting and allocation
Add targeting criteria
Target segments in experiments
# Add Confidence to a Slack Channel
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/add-confidence-to-slack-channel
Learn how to add the Confidence app to a Slack channel so it can post notifications.
You need to add the Confidence app to a channel if you want to let Confidence post notifications to it.
Your organization must have [integrated Slack with Confidence](./integrate-slack) first.
## Add Confidence to Channel
In Slack, go to the channel where you want Confidence to post notifications.
Type `/add` and click **Add app to this channel**. Select Confidence from the list.
## Related Resources
Connect Slack to Confidence
Set up notifications for surfaces
Set up experiment activity notifications
Deep dive into notification settings
# Add an Experiment to a Surface
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/add-experiment-to-surface
Learn how to add an experiment to a surface.
Adding an experiment to a [surface](../surfaces/introduction) helps organize experiments and allows others working on that surface to see your experiment.
## Add an Experiment to a Surface
Go to Confidence and select **A/B Tests**, **Rollouts** or **Analyses** on the left sidebar.
Select the experiment you want to add to a surface.
On the right sidebar, click the edit icon in the **Surfaces and coordination** section and select the surface. Click **Save**.
## Related Resources
Deep dive into surface configuration
Set up a new surface
Best practices for experiment organization
Configure exclusivity groups
# Add Metrics to a Surface
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/add-metrics-to-surface
Learn how to add metrics to a surface to make them easier to find or enforce them as required.
Add [metrics to a surface](../surfaces/surface-settings#metrics) to make them easier to find for experimenters experimenting on this surface, or enforce that all experiments on the surface check this metric for regressions by making it `required`.
## Add a Metric to a Surface
Go to Confidence and select **Surfaces** on the left sidebar.
Click on a surface to open its detail page.
Click **Settings** to open the surface settings page.
In the **Metrics** section, click **Add metric** and select the metrics to add.
After selecting metrics, choose which ones should be required for experiments on this surface. Required metrics are checked for regressions in all experiments.
## Related Resources
Deep dive into surface configuration
Set up metrics for your surfaces
Set up a new surface
API reference for adding metrics
# Archive a Flag
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/archive-flag
Learn how to archive a flag when it's no longer needed.
[Archiving a flag](../flags/create-flags#archiving-flags) prevents clients from resolving and using it.
## Archive a Flag
You must disable or delete all active rules on a flag before you can archive it.
## Related Resources
Set up a new feature flag
Learn about flag lifecycle and archiving
Configure clients that resolve flags
API reference for archiving
# Calculate Required Sample Size
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/calculate-sample-size
Learn how to calculate the required sample size for your experiment.
The [sample size calculator](../experiments/sample-size-calculator) helps you plan the length and size of an experiment by calculating what sample size you need to achieve the requested level of power.
When calculating the required sample size for an experiment, Confidence looks at historical data for the metrics in the experiment. There needs to be at least 14 days (plus the aggregation window and exposure offsets) of historical data for the metric. For example, if you have a metric with a 7-day aggregation window and a 7-day exposure offset, you need at least 28 days of historical data.
## Calculate the Required Sample Size
Configure the experiment with your treatments, audience, and metrics.
In the **Required sample size** section on the right sidebar, click the top-right widget icon.
Click **Calculate** on the widget to calculate the required sample size.
When the calculation finishes, the widget displays a results table and a
progress chart. The results table shows the following columns for each
metric:
The **Days needed**, **Expected samples**, and progress charts are beta
features. Contact
[experimentation-cs@spotify.com](mailto:experimentation-cs@spotify.com)
if you don't see them.
* **Days needed**: The estimated number of days to reach the required
sample size.
* **Required samples**: The sample size needed to achieve the configured
level of power.
* **Expected samples**: The projected total samples based on historical
exposure rates.
* **Mean**: The estimated mean of the metric from historical data.
* **Variance**: The estimated variance of the metric from historical data.
The widget header shows the maximum days needed across all metrics. The
progress chart above the table visualizes the estimated progress toward
100% of the required sample size for each metric over time. A vertical
dashed line marks the current planned runtime.
You can expand individual metric rows to view a per-metric chart that
shows the expected and required sample sizes over time.
## Define the Exposure Source
The exposure source is the source of the data used to calculate the mean and variance of the metrics.
Choose one of the following:
* **Assignments**: Use all existing assignments available in your assignment table, or filter these on assignments from specific flags
* **Previous experiment**: Use exposure from a previous experiment
If you selected **Assignments**, you can [filter assignments on flags](./filter-assignments-for-sample-size) to only include a cohort of users similar to those in your upcoming experiment.
## Adjust the Required Sample Size
If the required sample size is too large compared to the available population, you can either try to expand the population or reduce the required sample size.
To reduce the required sample size, you can do one or more of the following:
* **Increase Alpha setting**: Alpha is the probability of a false positive. A higher alpha requires a smaller sample size, but means the risk of finding significance when there really is no effect increases.
* **Lower Power setting**: Power is the probability of a true positive. The higher the power, the lower the probability of a false negative. A lower power requires a smaller sample size, but lowers the chance of finding a true effect. Lower power also increases the risk of sign and magnitude errors (type S and type M errors). In general, a too low power makes it hard to reproduce the results of an experiment.
* **Increase metric MDEs and NIMs**: The MDE and NIM are the effect sizes that you and your stakeholders care about. The larger the MDE and NIM, the smaller the required sample size.
## Related Resources
Deep dive into sample size calculations
Understand statistical power
Learn about MDE and NIM settings
Improve sample size estimates
# Clean Up Flags
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/clean-up-flag
Learn how to run flag cleanup scans to find and remove stale flags.
[Flag cleanup](../flags/flag-cleanup) scans your flags and connected repositories to find flags that are no longer in use. Each scan detects three kinds of stale flags—inactive flags, archived flags still referenced in code, and single variant flags—and can archive them or open cleanup pull requests automatically.
The Confidence home page also shows a **Clean up your workspace** card when
any of your flags haven't been applied in over a week. It's a quick way to
archive individual flags without running a full scan.
## Run a Manual Scan
Select **Flags > Cleanup** in the sidebar.
The scan analyzes your flags and any connected repositories. Results appear
when the scan completes.
Each row shows the flag name, the signal type (inactive, archived in code,
or single variant), and any code references found.
* For flags without code references, click **Archive** to archive the flag
directly, or **Dismiss** to skip it.
* For flags with code references, click **Open PR** to have a coding agent
open a cleanup pull request in the affected repository.
## Set a Scan Schedule
Select **Flags > Cleanup** in the sidebar.
Click **Schedule** in the top right.
Choose how often the scan runs—daily, weekly, or monthly—and click
**Save**.
Scheduled scans run automatically at the configured interval. You can still run a manual scan at any time.
## Enable Auto Archive
With auto archive enabled, Confidence automatically archives inactive flags that have no references in your connected repositories. You don't need to review and manually archive each one.
Select **Flags > Cleanup** in the sidebar.
Click **Settings** in the top right.
Enable **Auto archive inactive flags** and click **Save**.
Auto archive only applies to inactive flags with no code references. Flags
that are still referenced in code are never archived automatically—they appear
in scan results for you to review.
## Related Resources
Signal types, automatic archive, and cleanup pull requests explained
Manually archive a flag that's no longer needed
Flag schemas, variants, and lifecycle
# Configure Surface Notifications
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/configure-surface-notifications
Learn how to send notifications for activities on a surface.
Send [notifications](../surfaces/surface-settings#notifications) for activities on a surface to Slack or email.
## Configure Notifications
Go to **Surfaces** and select your surface.
Click **Settings**.
On the right sidebar, click the edit icon in the **Notifications** section to configure how to send notifications to Slack or email.
[Add the Confidence app to your Slack channel](./add-confidence-to-slack-channel) so it can post notifications.
To enable Slack notifications, you need to [integrate your Slack workspace with Confidence](./integrate-slack).
## Related Resources
Deep dive into surface configuration
Connect Slack to Confidence
Get personal notifications in Slack
Set up review processes for your surface
# Configure Surface Reviews
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/configure-surface-reviews
Learn how to add optional or required reviewers to all experiments on a surface.
Add optional or required reviewers to all experiments on a [surface](../surfaces/surface-settings#reviews). Required reviews block the launch of experiments until at least one of the required reviewers has approved the experiment setup.
## Configure Reviews
Go to Confidence and select **Surfaces** on the left sidebar, then select your surface.
Click **Settings**.
In the **Reviews** section, click the edit button to open the review setup dialog.
Choose a review option:
* **Don't add reviewers**: Experimenters can add reviewers manually, but the surface won't add any by default.
* **Add required reviewers**: At least one reviewer assigned by the surface must approve for experiments to launch.
* **Add optional reviewers**: Add recommended reviewers to all experiments, but experiments can launch without explicit approval.
Click **Next** to continue.
Select users or groups as reviewers from the dropdown. Click **Next** to continue.
Add a Slack channel or email address to centralize review request notifications. Reviewers always get personal notifications regardless of these settings. Click **Save** to apply your changes.
Send all requests to a common Slack channel if there are several reviewers on a surface. This helps keep track of all the review requests and who is reviewing which experiment.
For reviewers to get personal Slack notifications about review requests, they need to [integrate their personal Slack account with Confidence](../notifications/introduction#integrate-your-slack-account-with-confidence).
## Related Resources
Deep dive into surface configuration
Set up Slack and email notifications
Understand the experiment launch process
Set up a new surface
# Connect Your Personal Slack Account
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/connect-personal-slack
Learn how to connect your personal Slack account to receive notifications in Slack.
[Connect your personal Slack account](../notifications/introduction#connect-your-personal-slack-account) to receive personal notifications in Slack.
Your organization must have integrated Slack with Confidence first. If Slack is not yet integrated, ask your admin to [integrate Slack](./integrate-slack).
## Connect Personal Slack
Click your name in the bottom left corner.
Click `Connect` and follow the steps to connect your personal Slack account.
## Related Resources
Connect Slack to Confidence
Deep dive into notification settings
Get notified about comments and mentions
Set up notifications for surfaces
# Create an Activity Feed
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-activity-feed
Learn how to create an activity feed to follow activities for specific types of resources.
[Activity feeds](../notifications/activity-feeds) let you follow everything that happens for a certain type of resource in Confidence and route notifications to Slack, email, or webhook.
## Create an Activity Feed
Go to **Admin > Activity Feeds**.
Click **Create** and configure the activity feed for the resource types you want to follow (feature flags, A/B tests, or rollouts).
* **Slack**: [Add the Confidence app to your Slack channel](./add-confidence-to-slack-channel) so it can post notifications.
* **Webhook**: Configure your webhook URI and secret. See [webhook configuration](../notifications/webhook-configuration) for details on requirements and security.
Each surface has a built-in activity feed. To send all activities on a surface to Slack or email, go to the [surface settings](./configure-surface-notifications) and configure notifications.
## Related Resources
Deep dive into activity feeds
Configure webhooks for activity notifications
Enable Slack notifications
Set up surface-level notifications
Connect Slack to Confidence
# Create an Assignment Table
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-assignment-table
Learn how to create an assignment table in Confidence.
An [assignment table](../metrics/assignment-tables) stores records of what entities have been assigned what configuration.
## Create an Assignment Table
See the [configuration section](../metrics/assignment-tables#configuration) in the reference documentation for SQL query best practices.
Map the timestamp, entity, exposure key, and variant key columns.
To create assignment tables via the API, see [Create Assignment Tables](../api/how-to-guides/metrics/create-assignment-table) in the API how-to guides.
## Related Resources
Deep dive into assignment table configuration
Set up the entity for your assignments
Use assignment tables for analysis
Understand experiment exposure calculations
# Create a Conditional Targeting Rule
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-conditional-targeting-rule
Learn how to create a conditional targeting rule to assign variants to users based on criteria and percentage allocation.
[Conditional targeting rules](../flags/define-rules#conditional-targeting-rules) assign variants to users based on conditions and percentage allocation.
## Create a Conditional Targeting Rule
To create a conditional targeting rule, follow these steps.
Describe why you're adding this rule. This helps others understand the purpose and intended effect.
* **Serve**: Select the variant to serve to matching users.
* **To**: Set the percentage of traffic that should receive the variant (0-100%).
* **Of**: Select the attribute to use for randomization.
* **Matching**: Define conditions based on context attributes, or select **Any context** to match all users.
Select whether the rule applies to all environments or only specific environments. This step only appears if you have more than one environment.
The rule is created and immediately active. To create without activating, select **Create as draft** from the dropdown menu next to the button.
Traffic matching your conditions receives the variant according to the percentage you defined.
## Related Resources
Deep dive into rule types and configuration
Force specific variants for testing
Manage rule priority and order
Configure targeting criteria
# Create a Custom Role
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-custom-role
Learn how to create custom roles from fine-grained permissions.
Create [custom roles](../iam/roles#custom-roles) from fine-grained permissions to fit your organization's needs.
## Create a Custom Role
Go to **Admin > Roles**.
Click **Create** to create a custom role.
Give the role a name, and add the relevant permissions. For each permission you can select one or several types:
* **Reader** - Can view the type of resource that the permission handles
* **Creator** - Can create the type of resource that the permission handles
* **Editor** - Can edit (which includes reading and creating) the type of resource that the permission handles
* **Admin** - Has all permissions for this type of resource
You can add as many permissions and permission types as you need to one role.
Only the creator and reader permission types are meaningful to combine for the same type of resource. For all other combinations, one type of permission includes the other.
To give the custom role to a user or group, create a [policy](../iam/policies).
## Related Resources
Deep dive into role configuration
Assign roles to users and groups
Add new users to your organization
# Create a Dimension Table
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-dimension-table
Learn how to create a dimension table in Confidence.
A [dimension table](../metrics/dimension-tables) lets you segment your entities.
## Create a Dimension Table
On the left sidebar, select **Admin**, then select **Dimension tables**.
In the **About** section, enter a name for the table in the **Name** field. Optionally, select an **Owner**.
In the **Query** section, enter the SQL query and click **Run query**. See the [SQL query section](../metrics/dimension-tables#sql-query) in the reference documentation for more information.
In the **Configure table** section, map the entity column and dimension columns from the query result.
When you click **Create**, the dimension table goes into state `CREATING`. Confidence then runs a sample query towards the dimension table to verify that the SQL query produces columns that match what you've specified. After this the table either enters the `ACTIVE` or `FAILED` state.
To create dimension tables via the API, see [Create Dimension Tables](../api/how-to-guides/metrics/create-dimension-table) in the API how-to guides.
## Related Resources
Deep dive into dimension table configuration
Set up measurement data sources
Segment results by dimensions
Build metrics from your data
# Create an Entity
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-entity
Learn how to create an entity in Confidence.
An [entity](../metrics/entities) is something that can be uniquely identified and randomized, like a user.
## Create an Entity
On the left sidebar, select **Admin > Entities**.
In the **Entity key** field, enter a unique identifier for the entity. This key is used to reference the entity in the API.
In the **Display name** field, enter a human-readable name for the entity.
Select the data type of the identifier that identifies the entity. For example, if you have a UUID that identifies your entities, your primary key type is a **String**.
To create entities via the API, see [Create Entities](../api/how-to-guides/metrics/create-entity) in the API how-to guides.
## Related Resources
Deep dive into entity configuration
Connect entities to experiments
Set up measurements for your entities
End-to-end metric setup tutorial
# Create an Entity Relation Table
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-entity-relation-table
Learn how to create an entity relation table to connect anonymous users to authenticated users.
[Entity relation tables](../metrics/entity-relation-tables) connect anonymous users to authenticated users for experimentation.
## Create the Entity Relation Table
Open the entity that you randomize on (for example, Visitor).
In the **Entity relation tables** section, click **Create**.
Input a SQL query that outputs two columns that specifies the mapping between the entities.
Select the columns and the target entity (for example, User).
Confidence doesn't clean the data coming from this table, so it's important that it's of high quality
to ensure trustable experiment results. Any required data cleaning can either be done before the data ends up in the
relation table, or inline in the table definition since it can be any SQL query.
Below is a short description of possible error cases and how those would affect the results, using the **Visitor** to **User** case as an example:
* No mapping exists for a visitor ID: For metrics with padding enabled, the user is included in the calculation of the metrics for the experiment, but get 0 as the metric value. Otherwise, the user is excluded.
* Multiple visitor ID's map to the same user: The user would be included once, with first exposure set to the earliest assignment for the visitor ID.
* One visitor ID maps to multiple users: All users mapped to the visitor ID would be considered exposed to the experiment.
## Create the Experiment
With the entity relation table in place, you can now create the experiment.
Create an experiment and choose the entity that owns the relation as the entity to randomize on.
When selecting metrics you should now see metrics from both entities in the metric picker that you can then configure as in any other experiment.
## Related Resources
Deep dive into entity relation configuration
Learn about entity types and configuration
Handle anonymous user experiments
Run experiments with related entities
# Create a Fact Table
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-fact-table
Learn how to create a fact table in Confidence.
A [fact table](../metrics/fact-tables) contains measurements that describe your entities.
## Create a Fact Table
On the left sidebar, select **Admin > Fact tables**.
Click **+ Create** to create a new fact table.
In the **About** section, enter a name and select an owner.
In the **Query** section, enter the SQL that selects your fact rows and click **Run query**. See the [SQL query section](../metrics/fact-tables#sql-query) in the reference documentation for SQL query best practices.
In the **Configure table** section, map the timestamp column, entity columns, measurement columns, and dimension columns.
When you click create, the fact table goes into state `CREATING`. Confidence then runs a sample query towards the fact table to verify that the SQL query produces columns that match what you've specified. After this the table either goes to state `ACTIVE` or `FAILED`.
To create fact tables via the API, see [Create Fact Tables](../api/how-to-guides/metrics/create-fact-table) in the API how-to guides.
## Related Resources
Deep dive into fact table configuration
Build metrics from your fact tables
Add segmentation dimensions
# Create a Flag
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-flag
Learn how to create a flag in Confidence to remotely control the behavior of your application.
A [flag](../flags/create-flags) lets you remotely control the behavior in your application.
## Create a Flag
The flag creation wizard guides you through a series of steps. The number of steps depends on the flag type you select.
Enter the **Flag key** that uniquely identifies your flag. This is the same ID you use in your code. You can't change the flag key after creation.
Optionally add a **Description** to explain what the flag is for and how to use it.
Select an **Owner** for the flag. The owner defaults to the current user.
Select the type of flag:
* **Boolean**: A simple on/off flag to enable or disable a feature. Selecting this option takes you through a 4-step wizard.
* **JSON**: A flag with a schema that can store multiple values. Selecting this option takes you through a 6-step wizard with additional steps for defining the schema and creating an initial variant.
### Boolean Flag Steps
For Boolean flags, the wizard continues with:
Select which [clients](../flags/clients) can access this flag. You can skip this step and add clients later.
Review your flag configuration. Click **Edit** next to any section to make changes. When ready, click **Create** to create the flag.
When you create a Boolean flag, Confidence automatically creates two variants: **enabled** and **disabled**. You can use these variants in rules to turn features on or off for different users.
### JSON Flag Steps
For JSON flags, the wizard includes additional steps to define the schema and optionally create a first variant:
Define the schema for your flag by adding properties. Each property has a name and a type (String, Integer, Double, or Boolean). These properties define the values your flag can return.
Create an initial variant with values for your properties. You can modify these values later or skip this step and create variants after the flag is created.
Select which [clients](../flags/clients) can access this flag. You can skip this step and add clients later.
Review your flag configuration. Click **Edit** next to any section to make changes. When ready, click **Create** to create the flag.
Use a flag key that is understandable and memorable. For example, `new-navbar`.
Avoid long names, and don't include the configuration in the name itself.
For example, don't use `new-navbar-mobile-experience` or `new-navbar-enabled`
as the flag key.
Mobile apps and websites batch resolve flags to reduce the number of requests
made to Confidence. Associate a flag with as few clients as possible to limit
the number of resolved flags. Doing so also reduces costs.
## Associate a Flag with More Clients
You can associate a flag with more clients by following these steps:
This opens the clients dialog.
## View Code Snippets
To find instructions on how to use the SDKs for a particular flag, you can use
Confidence's code snippet feature.
Select client and credential that you want to preview the code for.
The code snippet shows you how to install, initialize, and resolve using
the Confidence SDK.
## Related Resources
Define the properties and types for your flag
Add variants with different values for your flag
Deep dive into flag concepts and architecture
API reference for flags
# Create a Flag Variant
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-flag-variant
Learn how to create variants to implement different experiences for a flag.
A [variant](../flags/create-flags#variants) is a named set of values for the properties of a flag.
## Create a Variant
The variant creation dialog guides you through two steps.
Enter a **Name** for the variant. Choose a descriptive name like `new-design`. The name can't be changed after creation.
Optionally add a **Description** to explain what this variant represents.
Optionally add an **Image** to help visually identify the variant.
Click **Next** to continue.
Set the values this variant returns for each property in the flag's schema.
If you leave a property empty, the SDK returns the default value specified in the client when the flag is resolved.
Click **Create** to create the variant.
Use short and descriptive names, informed by the provided experience. Use
kebab-case, like `blue` instead of `Blue`, and don't include too much
information: use `blue` instead of `blue-button-color`. Avoid including the
name of the flag in the variant name.
If a variant doesn't specify the value of a property, the SDK uses the
default value specified when a client resolves the flag.
## Related Resources
Target users with your variants
Define properties for your variants
Compare variants in an experiment
API reference for variants
# Create an Individual Targeting Rule
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-individual-targeting-rule
Learn how to create an individual targeting rule to force a specific variant for testing and debugging.
[Individual targeting rules](../flags/define-rules#individual-targeting-rules) force a specific variant for a set of users.
## Create an Individual Targeting Rule
To create an individual targeting rule, follow these steps.
Describe why you're adding this rule. This helps others understand the purpose and intended effect.
* **Serve**: Select the variant to assign to matching users.
* **To**: Select which attribute in the evaluation context to match against (such as User ID).
* **Matching**: Enter the specific values that should receive the variant. You can add multiple values or paste a list.
Select whether the rule applies to all environments or only specific environments. This step only appears if you have more than one environment.
The rule is created and immediately active. To create without activating, use the dropdown menu next to the button.
The rule now gives the selected variant to requests that match any of the values you specified.
Place individual targeting rules at the top of the rule list to ensure they evaluate first. See [how to reorder rules](./reorder-rules).
## Related Resources
Deep dive into rule types and configuration
Manage rule priority and order
Target users with flexible criteria
Verify your rules work correctly
# Create Materialized Segment
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-materialized-segment
Learn how to create a materialized segment that loads units from BigQuery.
A [materialized segment](/docs/flags/materialized-segments) loads its list of units from your BigQuery instance using a SQL query.
## Create a Materialized Segment
Navigate to the **Flags** page in the left menu, then click **Segments**.
Click the **Materialized segments** tab.
Click **+ Create** to create a new materialized segment.
Enter a descriptive name for your segment.
Click **+ Load Entities** and enter a SQL query that returns the entity IDs you want to include in the segment.
Run the query and select the correct column from the response that contains your entity IDs.
Click **+ Create load job** to start loading units into the segment.
## Related Resources
Learn when and why to use materialized segments
Create standard attribute-based segments
Configure your BigQuery connection
API reference for materialized segments
# Create a Metric
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-metric
Learn how to create a metric in Confidence.
A [metric](../metrics/metrics) is an aggregation of a measurement across instances of an entity.
## Create a Metric
In the **About** step, enter the name of the metric and optionally a description. You can also set an owner.
In the **Type** step, select whether you want to create a conversion, consumption, or click-through rate metric. Select **Custom** to access average/share or ratio metrics for full customization.
In the **Inputs** step, select the entity you want to measure and the fact table that has the measurements for your metric. Optionally add filters to include only rows that match specific criteria.
For average metrics, select how you want to aggregate data within units. See [the average metrics section](../metrics/metrics#average-metrics) for different types of aggregations.
For ratio metrics, select the numerator and denominator for the ratio. See [the ratio metrics section](../metrics/metrics#ratio-metrics) for more information about the different options available.
In the **Time window** step, select when to include entities in metric results and set the starting point and duration for evaluating the data.
In the **Suggested usage** step, set a preferred direction and threshold for the metric. These appear as defaults for your metric, but users can override them.
In the **Summary** step, review your metric configuration. You can edit any section before clicking **Create**.
To create metrics via the API, see [Create Metrics](../api/how-to-guides/metrics/create-metric) in the API how-to guides.
## Related Resources
Verify your metric is calculating correctly
Deep dive into metric types and configuration
Set up the data source for your metrics
Use your metrics to measure experiments
# Create a Policy
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-policy
Learn how to create a policy to grant roles to users or groups.
A [policy](../iam/policies) connects one or more groups, users, or API clients with a set of roles, granting permissions across your Confidence workspace.
## Create a Policy
Go to **Admin > Policies**.
Click **Create** to open the create policy dialog.
In the **Principals** field, add the people and groups you want to grant roles to.
In the **Roles** field, select one or more roles to assign to the principals.
Click **Create** to save the policy.
Use policies to grant permissions globally. When you give a user or group a role via a policy, that group has that role for all resources that the role governs. For more granular control, use manual permissions on individual resources instead.
## Related Resources
Deep dive into policy configuration
Configure fine-grained permissions
Understand available roles
Add new users to your organization
# Use Review Agents
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-review-agent
Learn how to create and use AI review agents that provide automated feedback on experiment designs.
Beta
AI review agents are currently in beta. If you don't see the **Agents** page in
your organization, contact
[experimentation-cs@spotify.com](mailto:experimentation-cs@spotify.com) to
request access.
Create an AI review agent to provide automated feedback on A/B test and rollout
designs. Agents analyze experiment configurations and post comments on
individual sections based on your instructions.
## Create the Agent
Go to Confidence and select **Admin** on the left sidebar, then select
**Agents**.
Click **Create** and enter a name and description for your agent.
Toggle on the **Review** skill to allow the agent to review experiments.
Enter instructions that define how the agent should review experiments.
See [Write effective instructions](#write-effective-instructions) for
guidance.
Click **Save** to create your agent.
## Write Effective Instructions
Structure your instructions by section to help the agent provide targeted
feedback. The agent reviews different sections depending on the experiment type.
Map your instructions to the sections on the experiment Design page.
### A/B Test Sections
| Section | What to define |
| ---------------- | --------------------------------------------------------------------------------------------- |
| **Display name** | Naming conventions or required prefixes |
| **Hypothesis** | What makes a good hypothesis statement |
| **Flag** | Flag selection and configuration expectations |
| **Treatments** | Requirements for treatment descriptions, images, and allocation |
| **Audience** | Targeting criteria and allocation expectations |
| **Surfaces** | Surface selection requirements |
| **Metrics** | Metric selection guidelines, role assignments, MDE and NIM requirements, preferred directions |
| **Stats** | Alpha, power, test horizon strategy, and exposure filter expectations |
| **Sample size** | Sample size calculation requirements |
| **Planning** | Experiment duration and scheduling expectations |
| **Links** | Required documentation or resources |
### Rollout Sections
| Section | What to define |
| --------------------- | ----------------------------------------------------- |
| **Display name** | Naming conventions or required prefixes |
| **Description** | What the rollout description should include |
| **Feature** | Variant selection and initial reach expectations |
| **Audience** | Targeting criteria and allocation expectations |
| **Surfaces** | Surface selection requirements |
| **Metrics** | Monitoring metric selection guidelines |
| **Stats** | Alpha and power expectations |
| **Sample size** | Sample size calculation requirements |
| **Planning** | Rollout duration and scheduling expectations |
| **Automatic ramp-up** | Ramp-up schedule, step count, and timing expectations |
| **Links** | Required documentation or resources |
Be specific about what the agent should check. For example, instead of "check
the hypothesis," write "verify the hypothesis states a clear expected outcome
with a measurable effect on the primary metric."
## Preview Agent Reviews
Test how your agent reviews experiments before using it on real experiments.
Go to **Admin** > **Agents** and select your agent.
Click **Preview**.
Select a draft A/B test or rollout from the dropdown list.
Click **Review** to see how the agent would review the experiment.
The preview shows the agent's overall judgment (approved or rejected) along
with feedback for each section. Use this to refine your instructions.
## Request an Agent Review
Request a review from an agent on the experiment Design page.
Go to an A/B test or rollout.
In the **Reviews** section on the right sidebar, click the plus icon.
Search for and select the review agent.
Click **Request** next to the agent name.
The agent analyzes the experiment and posts comments on individual sections
where it has feedback.
AI agent reviews are informational and don't count toward required reviews.
An AI agent approval does not meet the required reviewer requirement for
launching—at least one human reviewer must approve the experiment.
## Related Resources
Learn about the review process
Set up required reviewers for surfaces
# Create a Rollout
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-rollout-rule
Learn how to create a rollout to gradually turn on a feature for users.
A rollout gradually assigns a percentage of traffic to a new variant, allowing you to safely release features.
## Create a Rollout from a Flag
You can create a rollout directly from a flag's rules section.
This opens the rollout creation flow.
Follow the rollout creation steps to set up your gradual feature release.
For percentage-based targeting without a full rollout, consider using a [conditional targeting rule](./create-conditional-targeting-rule) instead.
## Related Resources
Deep dive into rule types and configuration
Gradual feature release with monitoring
Configure targeting criteria
Manage rule priority and order
# Create Segments
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-segments
Learn how to create segments to define reusable groups of users.
A [segment](../flags/segments) is a definition of a subpopulation, like a certain cohort
of users.
## Create a Segment
Select **Segments** in the left sidebar.
The name can't be changed later, so choose wisely.
Define the inclusion criteria for the segment by clicking **+ Add attribute**, **+ Add segment**, or **+ Add group** in the **Inclusion criteria** section. You can add multiple criteria.
## Related Resources
Deep dive into segment configuration
Add targeting criteria to your segments
Use segments in flag rules
API reference for segments
# Create a Surface
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/create-surface
Learn how to create a surface to organize experiments.
A [surface](../surfaces/introduction) is a logical representation of some part of your app or website, under which you can organize experiments.
## Create a Surface
Go to Confidence and select **Surfaces** on the left sidebar.
Click **+ Create** to create a new surface.
Give the surface a name, a description, and an owner. Click **Create**.
## Related Resources
Deep dive into surface configuration
Organize experiments on surfaces
Set up exclusivity groups on surfaces
Configure surface notifications and reviews
# Define the Audience
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/define-audience-criteria
Learn how to add inclusion criteria to target specific users in rules and segments.
The [audience](../flags/audience) is the set of users or other units that are eligible for a rule or segment.
The **Inclusion criteria** section offers four ways to define your audience:
* **Add attribute**: Target users based on evaluation context fields like country, platform, or version.
* **Add segment**: Target users who belong to a pre-defined [segment](../flags/segments).
* **Add holdback**: Include or exclude a random subset of users defined by a [holdback](../surfaces/surface-settings#holdbacks) on a surface.
* **Add group**: Combine multiple criteria into a logical group with `AND` or `OR` operators.
## Add an Attribute Criterion
This can be in an already created A/B test, rollout, segment, or rule.
Click **+ Add attribute** in the **Inclusion criteria** section.
Click it to unfold the dropdown list.
If Confidence has seen the field before, it autocompletes and the
type populates automatically. The operators available to choose from is
dependent on the type of the field. For example, if the field is a string, the
operator is one of: `is` (equals), `is not` (not equals), `in` (one of many) or
`not in` (not one of many).
The `in` operator behaves differently depending on how the context value is typed:
* **String**: `in` checks whether the entire string equals one of the listed values. It does **not** do substring matching. For example, if `fruit` is the string `"Apple, Banana, Cherry"`, then `fruit in ["Banana"]` will **not** match, because the full string does not equal `"Banana"`.
* **List of strings**: `in` checks whether any element in the list matches one of the listed values. If `fruit` is the list `["Apple", "Banana", "Cherry"]`, then `fruit in ["Banana"]` **will** match.
If you need to target users based on a comma-separated string, consider changing the context field to send a list of strings instead. Confidence does not support substring matching on strings.
## Add a Segment Criterion
Use a segment criterion to target users who belong to a pre-defined [segment](../flags/segments).
This lets you reuse the same audience definition across multiple experiments, rollouts, and rules.
This can be in an already created A/B test, rollout, segment, or rule.
Click **+ Add segment** in the **Inclusion criteria** section.
Choose a segment from the dropdown list.
Create segments in the **Segments** section of the left sidebar. Learn more in the [Create Segments](./create-segments) guide.
## Add a Holdback Criterion
Use a holdback criterion to include or exclude a random subset of users defined by a [holdback](../surfaces/surface-settings#holdbacks).
Holdbacks are configured on [surfaces](../surfaces/introduction) and represent a stable random subset of users that you can reuse over time.
This can be in an already created A/B test, rollout, segment, or rule.
Click **+ Add holdback** in the **Inclusion criteria** section.
Choose a holdback from the dropdown list. Only holdbacks from surfaces associated with your experiment are available.
Holdbacks are defined on surfaces. To create a new holdback, go to the surface's settings page. Learn more about holdbacks in the [surface settings documentation](../surfaces/surface-settings#holdbacks).
## Add a Group
Use groups to combine multiple criteria with `AND` or `OR` operators.
Groups let you build complex targeting logic beyond individual attribute criteria.
This can be in an already created A/B test, rollout, segment, or rule.
Click **+ Add group** in the **Inclusion criteria** section.
Choose `AND` or `OR` to control how criteria within the group combine.
Add attributes, segments, or holdbacks within the group.
You can nest groups to create arbitrarily complex targeting logic.
For example, you can target users in Sweden `AND` (on iOS `OR` Android).
## Related Resources
Deep dive into targeting and allocation
Define reusable user groups
Configure available targeting attributes
Configure holdbacks and exclusivity groups
API reference for targeting
# Edit Flag Schema
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/edit-flag-schema
Learn how to add or change properties in a flag's schema.
The [flag schema](../flags/create-flags#flag-schema) defines the available properties and their data types for a flag.
## Edit Schema
Boolean flags start with a single `enabled` property of Boolean type. JSON flags start with the properties you defined during creation. You can add more properties to any flag type.
You can always add new properties to a flag's schema.
Program your clients so that they can always handle the presence of new
properties in flag value.
You can change the name of a property if no variant sets a value for that
particular property. The same is true for deleting a property from the schema.
You can only delete a property if no variant sets a value for it. The schema editor disables properties that are in use by variants.
To edit the schema for a flag, follow these steps.
Click the edit schema button (pencil icon) next to the **Variants** heading on the flag detail page.
Click **Add property** and select the property type from the menu: string, bool, int, double, or struct.
Give the new property a descriptive name.
Click **Save** to apply your changes to the schema.
* Use short and descriptive name of the properties. For example, `color` is great.
* Use boolean properties to represent the state of a feature. For example, use
`enable` to represent if you should enable a feature or not.
* Use kebab-case for property names: `result-count` is better than `resultCount`.
* Avoid putting too much information in the name. For example, `color` is better
than `bgColor`.
## Related Resources
Add variants with values for your schema properties
Set up a new feature flag
Learn about flag schemas and configuration
API reference for schemas
# Explore Experiment Results
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/explore-results
Learn how to create explorations to drill down into experiment results.
[Explorations](../experiments/exploration) let you analyze experiment results with additional metrics and dimensions for exploratory purposes.
An experiment needs to have produced exposure for an exploration to be possible.
## Create an Exploration
After your experiment starts to produce exposure, open the experiment and select the **Result** tab.
Click the **+** button next to the **Explorations** heading in the right sidebar.
In the **Create exploratory analysis** dialog, enter a name for your exploration and click **Create**.
## Add Metrics
Click **Add metric** to open the metric selection dialog.
Use the checkboxes to select the metrics you want. You can select any metric that uses the same entity configured in the metric configuration section for your experiment.
Click **Add metrics** to add the selected metrics to your exploration.
Click **Retrigger** to start the calculations for your selected metrics.
## Add Dimensions
For each metric, you can add dimensions (if any exist). Dimensions come from the fact table that the metric uses and dimension tables that use the same entity as the metric.
## Write a Conclusion
After you have seen the results, you should write a conclusion detailing what you learned from this analysis.
The conclusion is important for posterity, and to help others understand what the outcome of
running the analysis was.
A good conclusion states the reason for running the analysis, the interpretation of the results,
and if you took any action based on it.
## Delete an Exploration
Click the exploration in the **Explorations** sidebar section.
Click the **Delete** button and confirm the deletion.
Do not delete an Exploration because results are not significant! Understanding
that there was no effect is as important as finding an interesting effect.
There is also a good chance that someone has the same great idea as you
‐ keeping your analysis around is likely to reduce duplicate efforts in
your organization.
## Related Resources
Deep dive into exploration features
Add dimensions for segmentation
Learn about result analysis
# Filter Assignments for Sample Size Calculations
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/filter-assignments-for-sample-size
Learn how to filter past assignments on flags to improve sample size calculations.
When calculating [required sample size](../experiments/sample-size-calculator), you can filter past assignments on flags to include only users similar to those in your upcoming experiment.
## Why Filter Assignments
Users assigned by a flag that assigns all users behave differently from a flag that only assigns highly engaged users. This means the mean and variance of metrics differ for the two cohorts, and so do the required sample sizes. By filtering assignments in the exposure source on relevant flags, you ensure the exposure source is as similar as possible to your upcoming experiment.
## Enable Assignment Filtering
Add a column called `flag` to your SQL query. The name can be in any case.
In the **Configure table** section, select the column from the **Flag column** dropdown.
## Filter Past Assignments on Flags
Select the experiment you want to run the sample size calculation for.
Click the icon in the **Required sample size** section to open the sample size calculation widget.
Click the edit icon in the Exposure source section to edit the exposure source.
Select **Assignments** as the exposure source and add the flags you want to filter on.
## Related Resources
Run sample size calculations
Deep dive into sample size calculations
Configure assignment data sources
Understand statistical power
# Integrate Slack with Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/integrate-slack
Learn how to integrate your Slack workspace with Confidence to enable Slack notifications.
[Integrate Slack](../notifications/introduction#integrate-your-slack-account-with-confidence) with Confidence to enable Slack notifications for surfaces, activity feeds, and personal notifications.
You need to have admin rights in your Slack workspace to integrate it with Confidence.
## Integrate Slack
Go to **Admin > Notifications** and select **Integrations**.
Click **Add Slack** and follow the steps to complete the integration.
After integrating Slack, you'll need to [add the Confidence app to specific channels](./add-confidence-to-slack-channel) where you want it to post notifications.
## Related Resources
Enable notifications in specific channels
Get personal notifications in Slack
Deep dive into notification settings
Set up notifications for surfaces
# Invite a User
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/invite-user
Learn how to invite users to your Confidence workspace.
If you're not using SSO, you must [invite a user](../iam/users#invite-users) to give them access to Confidence. Inviting a user sends them an email with a link to create an account.
To invite a user to Confidence, you must have the `Admin` role.
## Invite a User
Go to Confidence. On the bottom of the left sidebar, select **Admin > Users**.
In the dialog, enter the email address of the user you want to invite and which roles you want to assign them.
Click **Invite** to invite your colleague.
Invitations expire after 7 days. If the user doesn't accept the invitation within 7 days, you must invite them again. You can manually expire an invitation by clicking the trashcan icon next to it.
## Related Resources
Deep dive into user management
Understand available roles
Configure custom permissions
Deep dive into identity management
# Manage Clients
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/manage-clients
Learn how to create, configure, and manage clients to authenticate requests to Confidence.
To be able to talk to Confidence from your website, backend service or mobile
app, you need to create a [client](../flags/clients).
## Create a Client
Select **Admin > Clients** in the sidebar.
## Create Credentials
After you have created a client, you also need to create a credential that
you use to authenticate requests to Confidence. To create a credential,
follow these steps:
Select **Admin > Clients** in the sidebar.
A new credential is created and the credential secret is copied to
your clipboard. You can also view the credential secret by clicking the show icon.
Use the credential when you initialize a Confidence client SDK, or when
you make API requests to Confidence.
## Rotate Credentials
If you need to rotate credentials, do the following.
Select **Admin > Clients** in the sidebar.
## Related Resources
Set up flags that use your clients
Deep dive into client configuration
End-to-end tutorial with client setup
API reference for clients
# Manage Flag Permissions
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/manage-flag-permissions
Learn how to manage access permissions for individual flags.
Share a flag with other users or groups to give them access to view or edit the flag. When you share a flag, related resources like segments are automatically shared with the same permissions.
## Share a flag
Navigate to the flag you want to share.
Click **Permissions** at the top of the flag detail page.
In the dialog, add the people or groups you want to share the flag with.
Choose the role to assign:
* **Viewer**: Can view the flag and its configuration
* **Editor**: Can view and edit the flag
Review the list of related resources that are shared automatically with the same permissions.
Click **Share** to grant the permissions.
## Related Resources
Flag schemas, variants, and configuration
Learn about available roles
Configure access policies
Manage experiment-level permissions
# Manage labels
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/manage-labels
Learn how to create and manage labels to organize resources in Confidence.
[Labels](/docs/labels/introduction) help you categorize flags, metrics, and experiments
with custom tags. This guide covers creating label definitions and attaching labels to resources.
## Create a Label
Go to **Admin** > **Labels** in the sidebar navigation.
Click the **Create** button in the top right corner.
Fill in the label form:
* **Display name**: A descriptive name for the label (for example, "Team" or "Product Area")
* **Description**: Optional explanation of what this label represents
* **Resource types**: Select which resources can use this label (Flags, Metrics, or Experiments)
* **Allowed values**: Add the valid values for this label (for example, "Frontend", "Backend", "Mobile")
The label key is auto-generated from the display name.
Click **Create** to save the label definition.
## Attach a Label to a Resource
After creating a label, you can attach it to flags, metrics, or experiments.
Navigate to the flag, metric, or experiment you want to label.
In the sidebar, locate the **Labels** section.
Click **Add label** and select a label from the dropdown. Then choose a value from the allowed values list.
## Edit a Label Value
To change an existing label value on a resource:
In the resource sidebar, click on the label chip you want to edit.
Choose a different value from the dropdown menu.
## Remove a Label from a Resource
In the resource sidebar, click on the label chip you want to remove.
Click the delete icon to remove the label from the resource.
## Filter by Labels
Labels appear as filter options in list views for flags, metrics, and experiments.
Navigate to the Flags, Metrics, or Experiments list page.
Click the filter button to open the filter dropdown.
Expand the **Labels** submenu and select the label you want to filter by.
Select one or more values to filter the list.
## Edit a Label Definition
Go to **Admin** > **Labels**.
Click on the label you want to edit.
Modify the display name, description, resource types, or allowed values as needed.
Click **Save** to apply your changes.
Removing an allowed value does not automatically remove that value from resources that already have it assigned.
## Delete a Label
Go to **Admin** > **Labels**.
Click on the label you want to delete.
Click the **Delete** button and confirm the deletion.
Deleting a label definition does not remove the label values from resources that already have them assigned.
The label values remain on those resources but are no longer visible or editable in the UI.
## Related Resources
Learn about labels and their components
Organize feature flags with labels
Categorize metrics with labels
Group experiments with labels
# Reorder Rules
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/reorder-rules
Learn how to control rule evaluation order and understand the impact of reordering.
[Rules](../flags/define-rules#order-of-rules) evaluate in order from top to bottom, and the first rule that matches determines the variant assignment.
## Reorder Rules
You can reorder rules by dragging them up and down in the list.
Click and hold on a rule card, then drag it to the desired position.
## Best Practices
**Place individual targeting rules at the top.** Individual targeting rules should evaluate first to ensure they take precedence over other rules. This also increases their visibility and makes it quick to turn them off when no longer needed.
**Confidence adds new rules to the bottom.** When you create a new rule, Confidence places it at the bottom of the list to avoid impacting already running tests and rollouts.
Changing the order of rules can have a significant impact on user experience and may affect the results of A/B tests and rollouts. The first rule that matches a client and user determines the variant, so moving rules can change which users see which variants.
## Related Resources
Deep dive into rule types and configuration
Force specific variants for testing
Target users with flexible criteria
Verify your rules work correctly
# How to Run Exclusive Experiments
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/run-exclusive-experiments
Learn how to create exclusivity groups and make experiments mutually exclusive.
[Exclusivity groups](../experiments/exclusive-experiments) ensure that experiments don't overlap, so no user is in multiple experiments at the same time.
## Create an Exclusivity Group
If you don't already have an exclusivity group, create one first:
Give it a name and description. Choose whether it should be marked as suggested, which means it will be automatically pre-selected for experiments on the surface.
## Make an Experiment Exclusive
Go to Confidence and find the A/B test or rollout you want to make mutually exclusive with other experiments.
Select the surface on which the exclusivity group lives.
Select the exclusivity group you want to use.
The experiment is now mutually exclusive with other experiments that share the same exclusivity group.
## Related Resources
Deep dive into exclusivity groups and configuration
Organize experiments with surfaces
Run your first exclusive experiment
Learn about surface management
# Set Up Local Resolver Sidecar
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/setup-local-resolver
Learn how to deploy the local resolver sidecar container to resolve flags locally in your network.
The local resolver [sidecar](../flags/local-resolver#sidecar-resolver) is a container that resolves flags locally in your network. For an alternative approach using OpenFeature SDK providers, see [Local Resolver](../flags/local-resolver).
## Create API Client and Credentials
On the bottom of the left sidebar, select **Admin > API Clients**.
For example, "local resolver."
On the bottom of the left sidebar, select **Admin > Policies**.
Add the role **Flags Resolver Sidecar** to the principal with the name you just created, "local resolver".
Make note of the Client ID and client secret.
## Run with Docker
Set the following environment variables:
| Variable | Required | Description |
| -------------------------- | -------- | ------------------------------------------------ |
| `CONFIDENCE_REGION` | Yes | Your account region (`EU` or `US`). |
| `CONFIDENCE_CLIENT_ID` | Yes | The client ID of the API client you created. |
| `CONFIDENCE_CLIENT_SECRET` | Yes | The client secret of the API client you created. |
The examples use the Europe Artifact Registry mirror. Use the Asia or United States mirror when
closer to where you deploy the sidecar: `asia-docker.pkg.dev/spotify-confidence/public/flags-resolver-sidecar:latest`
or `us-docker.pkg.dev/spotify-confidence/public/flags-resolver-sidecar:latest`. The image registry region only controls
which mirror serves the image. Keep `CONFIDENCE_REGION` set to your Confidence account region, `EU` or `US`.
Run the local resolver:
```bash theme={null}
docker run -it --rm \
-p 8090:8090 \
-e CONFIDENCE_REGION="$CONFIDENCE_REGION" \
-e CONFIDENCE_CLIENT_ID="$CONFIDENCE_CLIENT_ID" \
-e CONFIDENCE_CLIENT_SECRET="$CONFIDENCE_CLIENT_SECRET" \
europe-docker.pkg.dev/spotify-confidence/public/flags-resolver-sidecar:latest
```
The local resolver is available at `http://localhost:8090`.
## Deploy in Kubernetes
Add the sidecar container configuration to your Kubernetes pod spec:
```yaml theme={null}
spec:
containers:
- name: my-service
image: gcr.io/my-service/image:latest
# whatever you already had configured
# add this sidecar container to your pod
- name: confidence-resolver-sidecar
image: europe-docker.pkg.dev/spotify-confidence/public/flags-resolver-sidecar:latest
env:
- name: CONFIDENCE_REGION
value: ""
- name: CONFIDENCE_CLIENT_ID
value: ""
- name: CONFIDENCE_CLIENT_SECRET
value: ""
ports:
- name: http
containerPort: 8090
readinessProbe:
exec:
command: ["/bin/grpc_health_probe", "-addr=:5990"]
initialDelaySeconds: 5
livenessProbe:
exec:
command: ["/bin/grpc_health_probe", "-addr=:5990"]
initialDelaySeconds: 10
resources:
requests:
cpu: 1
memory: 256M
limits:
cpu: 2
memory: 512M
```
## Do Not Propagate Apply Logs
By default, the sidecar sends apply logs to Confidence for exposure and experiment analysis. In
certain cases, such as when you have critical privacy concerns, you might need to keep apply data
within your network. Set `CONFIDENCE_SEND_APPLY_LOGS` to `false`.
The sidecar continues to accept and validate `ApplyFlags` requests and resolves with `apply=true`.
It doesn't send assignment events or apply distinct counts to Confidence. The sidecar
continues to send aggregate operational metadata, including the apply request rate.
Disabling apply logs degrades Confidence functionality. Confidence can't observe which entities
were exposed to each variant. Experiments and rollouts that use flags resolved by this sidecar
won't work correctly: exposure counts, experiment results, and rollout monitoring will be missing
or incomplete. Only disable apply logs if you accept this loss of functionality or provide
equivalent assignment data through another source.
This setting doesn't disable resolve telemetry or other communication required to fetch resolver
state. For more information, see [Data Transfer](../flags/data-transfer).
## Related Resources
Deep dive into local resolver configuration
Understand flag resolution options
Configure API clients and credentials
Learn about SDK integrations
# Share Experiment Permissions
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/share-experiment-permissions
Learn how to share an experiment with other users or groups.
Share an experiment with other users or groups to give them access to view or edit the experiment. When you share an experiment, related resources (flags, segments, assignment tables, entities, and metrics) are automatically shared with the same permissions.
## Share Permissions
Navigate to the A/B test or rollout you want to share.
Click **Permissions** in the top right corner.
In the dialog, add the people or groups you want to share the experiment with.
Choose the role to assign:
* **Viewer** - Can view the experiment and its results
* **Editor** - Can view and edit the experiment
Review the list of related resources that will be shared automatically with the same permissions.
Click **Share** to grant the permissions.
When you share an experiment, Confidence automatically shares related resources like the flag, segments, assignment table, entities, and metrics used in the experiment.
## Related Resources
Learn about available roles
Configure access policies
Add new users to your organization
# Test Flag Resolution
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/test-flag-resolution
Learn how to test flag resolution using the resolver test.
The resolver test helps you test and build intuition for how [rules](../flags/define-rules) resolve on a flag.
## Test Flag Resolution
Go to **Flags** and select the flag you want to test resolving.
Click the test button (beaker icon) in the **Rules** section of the flag detail page. This opens the Resolver test page.
Choose the client you want to test resolution for from the **Clients** dropdown.
Choose the credentials to use for authentication from the **Credentials** dropdown.
The **Filter by flag** checkbox is enabled by default when you open the resolver test from a specific flag. Disable it to test resolution across all flags.
Click **Add evaluation context** to add context fields. Toggle between key-value editing and JSON input using the buttons above. For example, add `user_id` with your user ID or `country` with value `Sweden`.
Click **Resolve** to test the resolution.
The resolver test returns a detailed response explaining which rules match and which don't, together with the reasons why.
Use **Fill using SDK data** to automatically populate the evaluation context with data from a recent SDK request.
## Related Resources
Deep dive into resolution testing
Learn about rule types and evaluation
Configure evaluation context attributes
# Use Exposure Filters
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/use-exposure-filters
Learn how to add exposure filters to narrow down which users to include in experiment analysis.
[Exposure filters](../metrics/exposure#exposure-filtering) are methods for narrowing down more closely which users to include in the exposure definition and the analysis of your experiment. When you add an exposure filter, the analysis only includes the exposed users that also match the exposure filter.
You first need to create a [fact table](../metrics/fact-tables) that has information about the exposure criteria before you can use it as an exposure filter.
## Add an Exposure Filter
On the experiment design page, find the **Results** section in the right sidebar. Click the menu button next to the heading.
Select **Exposure filters** from the dropdown menu to open the Add exposure filter dialog.
Give the exposure filter a descriptive name.
Select the exposure filter fact table from the dropdown. This fact table should contain information about when users meet your exposure criteria.
Optionally filter rows from the fact table by clicking **Add attribute** or **Add set** to define criteria. For example, if you have multiple page view events in the same fact table, filter to include only views from a specific page.
Click **Add** to save the exposure filter.
For exposure filtering, you can only select fact tables that have a column of type `Entity` that matches the entity for the experiment.
Exposure filters don't affect who actually gets the experiment experience. The filter only narrows down *who counts* as exposed in your A/B test. To limit who actually sees the experiment, use [inclusion criteria](../experiments/audience#inclusion-criteria) instead.
## View Results with Exposure Filters
Confidence produces metric results for each exposure filter separately. You can:
* Select different exposure filters in the results page to see how results vary
* Click **Detailed results** to see all metrics with and without all exposure filters simultaneously
## Related Resources
Deep dive into exposure configuration
Set up exposure filter data sources
Analyze results with different filters
# Validate a New Metric
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/validate-metric
Learn how to validate a newly created metric using exploratory analysis on a past experiment.
After creating a new metric, you should validate it to ensure it's calculating correctly and producing expected results. The best way to validate a metric is to test it against a previous experiment where you already know the outcome.
## Why Validate Metrics?
Validating metrics helps you:
* Confirm the metric calculates as expected
* Verify data joins and aggregations are correct
* Check that the metric responds to known treatment effects
* Inspect the underlying SQL for correctness
* Build confidence before using the metric in production experiments
## Before You Begin
* A completed experiment with known results
* A newly created metric that uses the same entity as the experiment
* Access to create explorations
## Validate Your Metric
Choose an experiment that:
* Has already completed and produced results
* Uses the same entity as your new metric
Navigate to the experiment's page in Confidence.
1. Go to the **Explorations** section in the right sidebar
2. Click **+** to create a new exploration
3. Give it a descriptive name like "Validate \[Metric Name]"
1. Click **Add metric** to open the metric selection dialog
2. Select the checkbox next to your newly created metric
3. Click **Add metrics** to add it to the exploration
4. Click **Calculate** to start the analysis
You can only select metrics that use the same entity configured for the experiment.
If you want to validate how your metric behaves across different segments:
1. Click **Add dimension** next to your metric
2. Select relevant dimensions from dimension tables
3. This helps you verify the metric works correctly across different user segments
Once the calculation completes, examine the results:
* **Check the values**: Do the metric values look reasonable?
* **Compare to expectations**: If you know the experiment outcome, does your metric show similar patterns?
* **Review dimensions**: Do dimensional breakdowns make sense?
If you need to debug the calculation itself, you can inspect the SQL query that was used to calculate the metric. To do this:
1. Find the status indicator showing the query job status
2. Click on the status to view details
3. Review the generated SQL query
You can copy the SQL and run it directly in your data warehouse to investigate further if needed.
Write a conclusion in the exploration describing:
* What you were validating
* Whether the metric behaves as expected
* Any issues discovered and how you resolved them
* Confirmation that the metric is ready for production use
This documentation helps others understand the validation process.
## Common Validation Checks
When reviewing your metric results, check for:
### Data Volume
* Does the metric produce results?
* Is the sample size similar to other metrics on this experiment?
### Treatment Effect Direction
* If the experiment had a positive effect, does your metric show that?
* Does the magnitude seem reasonable?
* Do confidence intervals make sense?
### Dimensional Consistency
* Do dimension breakdowns align with known patterns?
* Are there any segments with suspiciously different results?
### SQL Validation
* Are aggregations (SUM, AVG, COUNT) applied correctly?
* Are filters applied correctly?
## Related Resources
Learn how to create metrics in Confidence
Understand how explorations work
Detailed guide on using the exploration feature
Technical reference for metrics
# Verify a domain
Source: https://confidence-auth-testing.mintlify.io/docs/how-to-guides/verify-domain
Prove ownership of your organization's domain by adding a DNS TXT record.
Verifying a [domain](/docs/iam/domains) proves that your organization owns it. Once verified, Confidence associates the domain with your workspace.
To manage domains, you need the **Admin** role.
## Add a domain
Go to Confidence. On the bottom of the left sidebar, select **Admin > Domains**.
Click **Add domain** and enter your organization's domain (for example, `acme.com`). Confidence suggests domains based on the email addresses already associated with your workspace.
Free email domains like `gmail.com` or `outlook.com` cannot be registered.
## Verify Ownership With a DNS TXT Record
After adding a domain, you need to verify that your organization owns it by adding a TXT record to your DNS configuration.
Click **Verify** on the domain row. A dialog shows the TXT record you need to add:
* **Host:** Your domain (for example, `acme.com`)
* **Value:** A verification string in the format `confidence-domain-verification=`
Use the copy buttons to copy each value.
Log in to your DNS provider (for example, Cloudflare, GoDaddy, or Route 53) and add a new **TXT** record with the host and value from the previous step.
The record goes on the domain itself. Most DNS providers call this host **@** or let you leave the host field empty.
DNS changes can take anywhere from a few minutes to 48 hours to propagate, depending on your provider.
Back in Confidence, click **Verify**. Confidence checks your DNS for the TXT record and marks the domain as verified.
If verification fails, double-check that the TXT record is published correctly. You can use a tool like [Google Admin Toolbox Dig](https://toolbox.googleapps.com/apps/dig/) to confirm that the record is visible in public DNS.
## Remove a domain
To remove a domain, open the actions on the domain row and select **Remove**. Removing a verified domain also removes the ownership association with your workspace.
## Related Resources
Understand what verified domains are and how they work
Manage user access to your workspace
Overview of identity and access management
# Domains
Source: https://confidence-auth-testing.mintlify.io/docs/iam/domains
Associate verified domains with your Confidence workspace to prove organizational ownership.
## Overview
A verified domain is a domain (for example, `acme.com`) that you have proven your organization owns. Verification works by adding a DNS TXT record that Confidence checks before marking the domain as verified.
Verified domains associate your organization's email domain with your Confidence workspace. This proves that the workspace legitimately represents your organization.
## How Verification Works
1. A workspace admin adds a domain in **Admin > Domains**.
2. Confidence generates a unique verification token for the domain.
3. The admin adds a TXT record to the domain's DNS configuration with the value `confidence-domain-verification=`.
4. The admin clicks **Verify** in Confidence, which checks public DNS for the record.
5. If Confidence finds the record, it marks the domain as verified.
The TXT record must remain in your DNS for the domain to stay verified. Confidence checks the record at the time of verification.
## Domain Restrictions
You cannot register free email provider domains. This includes domains like `gmail.com`, `outlook.com`, `yahoo.com`, and other common personal email providers.
Only one workspace at a time can verify each domain.
## Required Permissions
Managing domains requires the **Admin** role. Users without this role do not see the Domains page in the admin settings.
## Related Resources
Step-by-step guide to verifying a domain
Manage user access
Overview of identity and access management
# Groups
Source: https://confidence-auth-testing.mintlify.io/docs/iam/groups
Bundle users and API clients into groups to simplify access management across your organization.
Groups allow you to bundle users, API clients, and other groups together and treat them as a unit.
Granting teams, instead of individuals, access to resources simplifies access management.
If someone leaves a team, remove them from the group to instantly revoke all their access rights.
You can layer groups such that one group consists of one or more other groups. Using this you can
model your current organizational structure so that one business area group has multiple teams as
members.
## Sync Groups
If you already have all your groups in some other external system, contact the Confidence team to
see if it's possible to automatically sync the groups from this system into Confidence.
## Related Resources
Overview of identity and access management
Manage individual user access
Understand roles and permissions
Assign roles to groups
# Introduction to Confidence Identity & Access Management
Source: https://confidence-auth-testing.mintlify.io/docs/iam/introduction
Learn how Confidence uses fine-grained authorization to control who can view, create, and edit resources.
Confidence implements a Fine-Grained Authorization (FGA) model that lets you
control who can see, create, and edit resources in Confidence.
Every list of resources that you can see throughout Confidence is automatically filtered by the current permissions.
This level of granular control ensures that sensitive information is only accessible to those who need it,
while still allowing for collaboration and experimentation within teams.
With Confidence, users can confidently explore and use resources without worrying about unauthorized access or accidental changes.
This video gives a quick overview of the central concepts and how to work with access management in Confidence in 3 minutes and 21 seconds.
## Key Concepts
Key concepts for the Identity & Access Management (IAM) system in Confidence are:
* [**Users**](./users) - A user is a person who has access to the Confidence workspace.
* [**Groups**](./groups) - A group is a collection of users (and service accounts).
* [**Roles**](./roles) - A role is a set of permissions that can be assigned to a user or a group.
* [**Policies**](./policies) - A policy gives a user or a group one or several roles.
You can grant a user or a group permission to a resource in three ways:
1. Through a [policy](./policies) that gives the user or the group permission to all resources of a certain type
2. By making the users or group the [owner](./roles#owner) of a resource (typically done when creating a resource)
3. Share a specific resource with a user or a group by selecting **Permissions** on the resource page
Manage the access for everyone on a resource by clicking 'Permissions' on the resource page and set the general access.
## Get Started With Access Management
### Default Settings
Your Confidence workspace comes with:
* Predefined roles (full list in [roles documentation](./roles#predefined-roles)):
* `Creator`: Creators have no general read and edit privileges, but can create any resource. They are the owners of the resources they create, and can edit them.
* `Reader`: Can read everything
* `Editor`: Can create, read, and edit everything
* A group called `Everyone` that all users belong to. You cannot remove this group.
* A policy giving the group `Everyone` the roles of `Creator` and `Reader`. This makes it possible for anyone to create any type of resource in Confidence, for example a flag, an A/B test, or a metric, and to see resources others have created. You can edit or remove this policy if you have the role `IAM Admin` or `Admin`.
When you create a resource, including instances of A/B tests and rollouts:
* The group `Everyone` has the role `Reader` by default. You can change this to `Editor`.
* The creator selects an owner, who gets the role of `Owner`. The owner, by extension, receives also the role of `Editor` for that resource.
### When to Use Policies versus Manual Permissions
A policy gives a group or a user a certain role that implies they can do
specific things, often for a specific type of resource. For example, you can
have a policy that gives the group 'Team A' the role of `Flags Editor` which
allows all users in 'Team A' to edit any feature flag. A manual permission gives
a user or a group read or edit rights for a particular instance of a resource.
Policies control permissions globally. When you give a user or group a role via a
policy, that group has that role for all resources that the role governs.
Returning to the example, if 'Team A' gets the role of `Flags Editor`, this
team can edit *all flags* regardless of flag ownership and manual permissions
set on individual instances of flags.
Don't create policies that give groups or users `Editor` roles.
Permission to edit is often better to handle at the resource level.
`Reader` and `Creator` roles are often good to give via policies.
If you want to limit who can take certain actions on specific instances, make
sure that there are no policies for the roles that govern that action. For
example, if you want to limit who can edit an experiment on each experiment
itself, there must be no policy giving the role `Editor` to any
user or group. If you have a central experimentation team,
add an `Experiment Editor` role to the group of that team.
### Handle New Users and Users Without Groups
Use a policy on the group `Everyone` to set the ground rules for what everybody
can do. By default, Confidence has a policy that gives the group `Everyone` the roles of `Creator` and `Reader`.
Make sure to have a policy for the `Everyone` group in place, to handle users
that have no direct or indirect ownership. Confidence comes with a policy that
configures everyone to be a `Creator` by default. Change this under **Admin >
Policies**.
Learn more about the concepts of Confidence Identity & Access Management:
## Related Resources
Manage user access
Organize users into groups
Understand predefined and custom roles
Grant permissions to users and groups
# Policies
Source: https://confidence-auth-testing.mintlify.io/docs/iam/policies
Grant account-wide permissions by connecting users, groups, and API clients to roles through policies.
Confidence checks all permissions on a per-resource level. Even though this is incredibly powerful sometimes
you want to grant permissions to the entire account.
A policy connects one or more groups, users, API Clients with a set of roles.
## Everyone
Confidence manages a special group called *Everyone* which allows you to grant some roles to every user who logs in and
every API Client. Using this special *Everyone* it's possible to set up a default baseline set of permissions.
## Related Resources
Overview of identity and access management
Understand available roles
Organize users into groups
Manage individual user access
# Roles
Source: https://confidence-auth-testing.mintlify.io/docs/iam/roles
Manage access with predefined roles or create custom roles from fine-grained permissions to fit your organization.
In Confidence, you can create nuanced permission schemas using fine-grained roles.
Confidence comes with a set of [predefined roles](#predefined-roles) that are usually enough to achieve the desired access control.
If you need even more specialized roles, you can create [custom roles](#custom-roles).
## Predefined Roles
Confidence comes with the following predefined roles:
| Role | Description |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Admin** | Has the highest level of privileges. Admins have full control over various resources, including workflows, metrics, flags, IAM policies, events, and billing. They hold permissions to administer, edit, create, and view various entities within the system. |
| **Editor** | General role for editing various resources. Editors have permissions to create, update, and manage entities within workflows, metrics, flags, IAM, and events. |
| **Creator** | General role for creating various resources. Creators have permissions to create, and thereby make themselves owner of, entities within workflows, metrics, flags, IAM, and events. |
| **Reader** | General role for read-only access. Readers can view various entities across workflows, metrics, flags, IAM, and events. |
| **Billing Admin** | Specifically handles billing-related administrative tasks. They have permissions related to billing administration. |
| **Events Editor** | Manages and edits event-related resources. They have permissions to create, edit, and manage events, event connections, event definitions, and related cryptographic keys. |
| **Events Reader** | Has read-only access to event-related resources. They can view events, event connections, event definitions, and related cryptographic keys. |
| **Flags Editor** | Manages and edits feature flags and related segments. They can create, edit, and manage flags, segments, and evaluation context schemas. |
| **Flags Reader** | Has read-only access to feature flags and related segments. They can view flags, segments, and evaluation context schemas. |
| **Flags Resolver Logger** | Involved in logging resolve information for flags. They have permissions related to administration of resolve information and flag assignments. |
| **Flags Resolver Sidecar** | Manages the sidecar aspect of flag resolution. They have permissions to read flags, segments, and clients, and administer resolve information and flag assignments. |
| **IAM Editor** | Manages and edits IAM (Identity and Access Management) resources. They can create, edit, and manage clients, roles, user invitations, OAuth apps, cryptographic keys, and IAM policies. |
| **IAM Reader** | Has read-only access to IAM resources. They can view clients, roles, user invitations, OAuth apps, cryptographic keys, and IAM policies. |
| **Metrics Editor** | Manages and edits metrics-related resources. They can create, edit, and manage metrics, metric calculations, scheduled metric calculations, and associated tables and warehouses. |
| **Metrics Reader** | Has read-only access to metrics-related resources. They can view metrics, metric calculations, scheduled metric calculations, and associated tables and warehouses. |
| **Stats API User** | Specific role for users of the Stats API. They have permissions related to the usage of the Stats API. |
| **Workflows Editor** | Manages and edits workflow-related resources. They can create, edit, and manage workflows, workflow instances, workflow logs, workflow secrets, and related surfaces. |
| **Workflows Reader** | Has read-only access to workflow-related resources. They can view workflows, workflow instances, workflow logs, workflow secrets, and related surfaces. |
### Owner
All resources in Confidence have an owner.
The owner role can only be assigned per instance of a resource, for example an A/B test, a metric, or a surface.
The owner of a resource has full control over the resource.
The owner is typically the user that created the resource, but you can set it to any user or group.
## Custom Roles
Create custom roles from the finest grained permissions to fit your organization's needs.
For each permission you can select one or several types:
* **Reader** - Can view the type of resource that the permission handles
* **Creator** - Can create the type of resource that the permission handles
* **Editor** - Can edit (which includes reading and creating) the type of resource that the permission handles
* **Admin** - Has all permissions for this type of resource
Only the creator and reader permissions types are meaningful to combine for the same type of resource. For all other combinations, one type of permissions includes the other.
### Available Permissions
The following permissions are available:
| Permission | Description |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [API client](../api/quickstart#create-an-api-client) | API clients for Confidence services. Users integrating with Confidence via the APIs need these permissions. |
| [Assignment table](/docs/metrics/assignment-tables) | Data tables for feature flag rules applied events. These permissions are mainly needed in the setup phase of the Confidence App. |
| [Client](/docs/sdks/introduction) | Feature flag clients. Users who need to create or delete new clients need these permissions. Feature flag developers need read permissions to be able to select clients for their feature flags. |
| [Data warehouse](../data-warehouse-native) | Data warehouse integration and configuration. Only relevant for [Warehouse Native Confidence](../data-warehouse-native) users. |
| [Dimension table](/docs/metrics/dimension-tables) | Data tables with dimensions to segment results of metrics in [explorations](/docs/experiments/exploration). Users mapping data in your data warehouse to Confidence need these permissions. |
| [Entity](/docs/metrics/entities) | Unique identifiers for randomly assigning treatment and tying metrics to units exposed to an experiment. These permissions are mainly needed in the setup phase of the Confidence App. |
| [Entity relation table](/docs/metrics/entity-relation-tables) | Mappings between entities and input fields. These permissions are mainly needed in the setup phase of the Confidence App. |
| Exposure calculation | Data job for calculating exposure for [explorations](/docs/experiments/exploration) in experiments and for the results in [analyses](/docs/experiments/workflows/analysis). Workflows have permissions to create and edit exposure calculations even if the workflow instance owner doesn't. |
| [Exposure table](/docs/metrics/exposure) | Data tables for experiment exposure. Typically created by a workflow instance. Workflows have permissions to create and edit exposure tables even if the workflow-instance owner doesn't. |
| [Fact table](/docs/metrics/fact-tables) | Data tables containing measurements that describe your entities. Users mapping data in your data warehouse to Confidence need these permissions. |
| [Flag](../flags/introduction) | Feature flags including variants and rules. Developers integrating software services with Confidence need these permissions. Experimenters can only launch experiments if they have create rights for the involved flag, but it's recommended to give create permissions per flag rather than by roles. |
| [Group](./groups) | Groups of users that own resources and have roles. Users administering groups and teams need these permissions. |
| Materialized segment | Materialized segments for the [sticky assignment](../flags/audience#sticky-assignments) functionality. Workflows have permissions to create and edit materialized segments even if the workflow instance owner doesn't. |
| [Metric](../metrics/introduction) | Metrics defined on top of fact tables. Users developing and consuming metrics in Confidence need these permissions. |
| Metric calculation | Metric calculation for [explorations](/docs/experiments/exploration) in experiments and for the results in [analyses](/docs/experiments/workflows/analysis) instances. Workflows have permissions to create and edit metrics calculations even if the workflow instance owner doesn't. |
| Role | A set of permissions that a group or user can get via a [policy](./policies). It's recommended that only the Confidence Admin has these permissions (which they have by default). |
| Scheduled exposure calculation | Data job for calculating exposure for an experiment. Workflows have permissions to create and edit scheduled exposure calculations even if the workflow instance owner doesn't. |
| Scheduled metric calculation | Metrics calculations for experiment results. Workflows have permissions to create and edit scheduled metric calculations even if the workflow instance owner doesn't. |
| Segment | Segments are internal to rules on feature flags, they contain targeting and allocation logic including treatment assignment. Workflows have permissions to create and edit segments even if the workflow instance owner doesn't. |
| SQL job | A user needs creator permission to be able to run preview queries. Workflows have permissions to create and edit SQL jobs even if the workflow instance owner doesn't. |
| Workflow | Workflows are the blueprints for A/B tests, rollouts, and analyses. These permissions are only needed for users hosting their own Confidence instance or using only Confidence APIs. |
| Workflow instance | The instances of workflows are the A/B tests, rollouts, and analyses created by Confidence users. To be able to experiment, users need permissions for workflow instances. |
| Workflow log | Permission to view logs from a workflow, used when developing custom workflows. |
| Workflow secret | Permission to manage secrets for a workflow, used when developing custom workflows. |
## Related Resources
Overview of identity and access management
Assign roles to users and groups
Organize users into groups
Manage individual user access
# Users
Source: https://confidence-auth-testing.mintlify.io/docs/iam/users
Manage access to Confidence by adding and removing users.
## Single Sign-On (SSO)
Confidence supports single sign-on (SSO) using Google as the identity provider.
To enable SSO, reach out to the Confidence team.
With SSO, users can log in directly without first receiving an invitation.
Confidence creates a user for them the first time they log in.
## Invite Users
If you're not using SSO, you must invite a user to give them access to
Confidence. Inviting a user sends them an email with a link to create an
account.
Invitations expire after 7 days. If the user doesn't accept the invitation
within 7 days, you must invite them again. You can manually expire an
invitation by clicking the trashcan icon next to it.
## Related Resources
Overview of identity and access management
Organize users into groups
Understand roles and permissions
Grant permissions to users
# Introduction
Source: https://confidence-auth-testing.mintlify.io/docs/introduction
Welcome to the Confidence documentation
## Welcome to Confidence
Confidence is a comprehensive experimentation and feature management platform that helps teams make data-driven decisions.
Follow our onboarding guide to get started with Confidence
Jump right in with our quickstart guides
Explore our comprehensive API documentation
Learn about our SDKs and integrations
## What is Confidence?
Confidence provides powerful tools for:
* **Feature Flags**: Control feature rollouts with flexible targeting rules
* **Experimentation**: Run A/B tests and analyze results with statistical rigor
* **Metrics**: Define and track business metrics across your organization
* **Data Warehouse Native**: Leverage your existing data infrastructure
## Choose Your Path
Built on top of your data warehouse
Extensible platform for custom workflows
# Introduction to labels
Source: https://confidence-auth-testing.mintlify.io/docs/labels/introduction
Reference documentation for labels in Confidence.
Labels are user-defined tags that help you categorize and organize resources in Confidence.
Use labels to group related flags, metrics, and experiments for easier filtering and discovery.
## Label components
A label definition consists of:
* **Name**: A human-readable name for the label
* **Key**: A unique identifier auto-generated from the display name (lowercase letters, numbers, and hyphens)
* **Description**: Optional text explaining the label's purpose
* **Resource types**: Which resources this label applies to (flags, metrics, or experiments)
* **Allowed values**: A predefined list of valid values for the label
## Supported resources
Labels can be attached to:
* **Flags**: Categorize feature flags by team, project, or any custom dimension
* **Metrics**: Organize metrics by domain, ownership, or data source
* **Experiments**: Group A/B tests, rollouts, and analyses by initiative or product area
## Using labels
After you create a label definition, the label appears in the sidebar of supported resources.
You can assign values to resources and filter list views by label values.
Labels appear as chips in list views, making it easy to see at a glance how resources are categorized.
When space is limited, labels collapse into a "+ X labels" indicator that expands on click.
When adding metrics to an experiment, you can filter the metrics picker by label values to quickly
find relevant metrics. This is useful when you have many metrics and want to narrow down
the selection to a specific team, product area, or category.
## Related resources
Create and configure labels
Learn about feature flags
Learn about metrics
Learn about experiments
# Assignment Tables
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/assignment-tables
Assignment tables have records of what entities have been assigned what configuration.
When using a feature flag, variants are assigned to entities.
Each such assignment is typically recorded somewhere in a data warehouse.
Assignment tables in Confidence point to a table in your data warehouse that
stores the assignments.
Confidence provides this table if you're using Confidence Flags.
If you're not using Confidence Flags, your internal or third-party feature flagging service provides
the table.
The only requirement for an assignment table in Confidence is that the underlying table has
columns with information about when assignments happened, for whom, and where the assignment came
from.
## Structure
The assignment table in Confidence Flags has the following columns:
| Column | Type | Description |
| --------------------------- | --------- | ------------------------------------------------------------------------------------------- |
| `assignment_time` | Timestamp | When the assignment was applied. |
| `resolve_id` | String | Unique identifier for the resolve request. |
| `targeting_key` | String | The identifier of the entity that was assigned. |
| `targeting_key_selector` | String | The type of entity targeted (for example, `user_id`). |
| `flag` | String | The flag that was resolved. |
| `rule` | String | The rule that matched. Null for fall-through assignments. |
| `variant` | String | The variant assigned. Only set for regular assignments, not fall-through. |
| `assignment_id` | String | Identifier for the treatment group. Used as the variant key when configuring an experiment. |
| `client` | String | The identifier for the client. |
| `client_credential` | String | The identifier for the client credential. |
| `segment` | String | The segment that matched. Only set for regular assignments. |
| `default_assignment_reason` | String | Reason for the default assignment (for example, flag archived or no matching rule). |
Here is an example of what the data in an assignment table looks like:
| `assignment_time` | `resolve_id` | `targeting_key` | `targeting_key_selector` | `flag` | `rule` | `variant` | `assignment_id` | `client` | `client_credential` | `segment` | `default_assignment_reason` |
| ------------------- | ------------ | --------------- | ------------------------ | ---------------- | --------------------------- | ---------------------------------- | --------------- | --------------- | -------------------------------------------- | ----------------- | --------------------------- |
| 2023-05-20 00:04:25 | abc-123 | user5125 | user\_id | flags/test-flag | flags/test-flag/rules/rule1 | flags/test-flag/variants/control | control | clients/web-app | clients/web-app/clientCredentials/prod-key | segments/us-users | |
| 2023-05-20 05:01:21 | def-456 | user7231 | user\_id | flags/test-flag | flags/test-flag/rules/rule1 | flags/test-flag/variants/treatment | treatment | clients/web-app | clients/web-app/clientCredentials/prod-key | segments/us-users | |
| 2023-05-21 10:22:01 | jkl-012 | user8844 | user\_id | flags/other-flag | | | | clients/ios-app | clients/ios-app/clientCredentials/mobile-key | | FLAG\_ARCHIVED |
A single resolve request can produce multiple rows. Each applied flag produces one row,
and each fall-through assignment on that flag produces an additional row.
When configuring an assignment table, you need to specify the columns that map to the following:
* A **timestamp** column that records the time of the assignment. The `assignment_time` column.
* One or more **entity** columns that hold the identifier of the entity the assignment happened for. The `targeting_key` column.
* An **exposure key** column to use to filter out the assignments that belong to the
experiment. The `rule` column. You could also use the `flag` column if there is a 1-1 mapping of an experiment to a flag. Confidence
represents an experiment as a [rule](/docs/how-to-guides/create-conditional-targeting-rule) on the flag.
* A **variant key** column that identifies a specific variant. The `assignment_id` column. Use the `assignment_id` column and not the `variant` column as Confidence supports fall-through variants that allow multiple variants to be part of the same assignment.
## Configuration
### SQL Query
You need to input the SQL that selects your assignment rows. After you've
entered your SQL you can click **Run Query**. Confidence then executes your
query with a limit to check that everything is in order and show you some
sample rows.
You should write resource efficient queries. The result frequency of your experiment
determines how often the queries run. For example, an experiment with hourly calculations runs the
assignment table query once an hour.
Here is a checklist of things to consider when writing the query for your assignment data:
* ✅ Always have an index (also called [Time partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#date_timestamp_partitioned_tables) in BigQuery) on the timestamp
column to ensure that Confidence only queries the data within a partition, without
this each query may require a full table scan.
* ✅ If you can, also set up an index on the entity columns (also called [Clustered Table](https://cloud.google.com/bigquery/docs/clustered-tables) in BigQuery),
this helps improve the performance of the join that Confidence does between your exposures
and facts.
* ✅ If you can, perform heavy transformations in an upstream step and store the result in an
intermediate table to avoid potentially redoing the transformations.
You have some placeholders at your disposal if you need to specify what
partition you're querying: `{START_TIME}` and `{END_TIME}`. These two
parameters vary depending on the requirements of the metric that uses
the assignment table. You don't need
to use them as Confidence always filters on your timestamp column, but
sometimes your table format includes the date. These placeholders have the
same Timestamp type as the timestamp column in your data warehouse.
### Data Delivery Cadence
For configuring data delivery cadence, see the
[data delivery cadence](/docs/metrics/delivery-cadence) page.
### Timestamp Column
The timestamp column is the column that identifies the time that the assignment
occurred. This column can either be a timestamp with a specific time point for
the assignment, or you can select a date column. Selecting a date column
means that any exposure calculation that includes the end of this day should include
this assignment. It's the same as specifying a time for the date that is at 23:59:59.
### Entities
You need to specify the entity that this assignment is for. This has to be the
same identifier that you use for the facts.
### Exposure Key and Variant Key Columns
The exposure and variant keys must, at a given time, uniquely identify a single treatment group.
When using Confidence Flags:
* **exposure key** is the name of the rule
* the **variant key** is the `assignment_id`
If you use an external feature flagging system, then often:
* **exposure key** is the name of the flag
* **variant key** is the name of the variant the flag returns (for example, true/false)
## Related Resources
Step-by-step assignment table guide
Configure exposure calculations
Use assignment data for analysis
Configure experiment entities
# Data Delivery Cadence
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/delivery-cadence
Use the data delivery cadence setting to configure how Confidence should detect when your data is ready.
Confidence needs to know when your data is ready for calculating results. This
includes fact tables, assignment tables, and exposure tables. Use the data
delivery cadence setting on each table to configure the behavior. The three
options are:
* **Manually notify Confidence**. Each table has a timestamp that indicates
until when the system has delivered data. You can update this timestamp manually through
the Confidence API. Use this option if you already have a system for keeping
track of data delivery.
* **Data is delivered continuously**. If you continuously write new data to your
tables, such as new events whenever they happen, Confidence divides the data
into smaller time windows. The **commit delay** parameter determines how
long the system must not write new data to a window before Confidence commits the
window. Upon committing the window, Confidence updates the timestamp that
indicates until when the system has delivered data.
* **Data is partitioned hourly or daily**. If you already partition your data,
Confidence automatically detects when a new partition exists. Confidence
considers a partition ready as soon as it detects that there is data in the
partition.
Confidence assumes that you write hourly partitions to even hours in UTC time zone, and that you
write daily partitions at midnight in UTC.
## Related Resources
Configure fact table delivery cadence
Configure assignment data sources
Connect your data warehouse
Overview of the metrics system
# Dimension Tables
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/dimension-tables
Use dimension tables to segment your entities.
Dimensions describe your entities.
Use dimensions to segment your metrics into subgroups.
For example, by adding a `country` dimension to a `User` entity, you can split results into
subgroups based on the country of the user.
If the dimension table has a column of type timestamp, Confidence uses the dimension value from the time right before the user got exposed to the treatment.
## Configuration
### SQL Query
You need to enter the SQL that selects your dimensions. After you've entered your
SQL, you can click **Run Query**. Confidence then executes your query with a
limit to check that everything is in order and displays a few sample rows.
### Entity
You need to specify which entity the dimensions belong to. You can
split the result of metrics for this entity into subgroups based on the
dimensions.
### Dimensions
A dimension is a value that you can use to categorize an entity. Confidence
supports static dimensions and dimensions that change over time. A typical
example is splitting a "User" entity into segments based on a "Country" which is
a static dimension, or "Used feature X the week before the experiment" which is
a time-changing dimension.
For time-changing dimensions, it's important to use dimension values before the
user got exposed to the treatment. This way, the treatment doesn't influence
the dimension value itself, which could invalidate the
exploration.
Confidence distinguishes between static and time-changing dimensions by looking for
a timestamp column in the dimension table schema. Confidence considers all
dimensions in a dimension table with a timestamp column as time-changing
dimensions.
Confidence uses the dimension value from the time right before
the user got exposed to the treatment. This ensures that the dimension
value is not influenced by the treatment itself.
For time-changing dimensions, you need to specify what time window before
exposure to use to fetch the dimension value. If there are multiple
values in this window, Confidence uses the latest value.
#### NULL Values
Sometimes, not all users exposed to an experiment have a value in the dimension table. Add a NULL-value mapping to the dimension field to
make the lack of value render in a meaningful way in the exploration
page. You can configure the NULL-value mapping by entering a replacement in the
**Replace null with** setting for a dimension.
## Related Resources
Step-by-step dimension table guide
Segment results by dimensions
Set up measurement data sources
Configure dimension entities
# Entities
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/entities
An entity is something that can be uniquely identified and randomized.
The entities represent the things you're experimenting on, like your users.
You can use other entities, like ad, product, or visitor.
Each entity has a unique identifier with a specific data type. For example, if you have a UUID that identifies your users, your primary key type is a string.
## Related Resources
Step-by-step entity creation guide
Connect different entity types
Configure experiment assignments
Set up entity measurements
# Entity Relation Tables
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/entity-relation-tables
Use entity relation tables to connect anonymous users to authenticated users for experimentation.
## When to Use Entity Relation Tables
Entity relation tables serve a **specific use case**: connecting anonymous users to authenticated users when:
* You randomize on an anonymous entity (like Visitor)
* Users later authenticate and become a different entity (like User)
* You want to measure authenticated user metrics in experiments that started before authentication
Entity relation tables are not for:
* Creating general mappings between entities (like users to orders)
* Reusing metrics across different randomization units
* One-to-many relationships for analysis purposes
For analyzing at a more granular level than your randomization unit (for example, analyzing orders when randomizing by users), use [Ratio Metrics](./metrics#ratio-metrics) instead.
## Example Use Case
If you have a **Visitor** entity that you randomize on for unauthenticated users, and then you have a **User** entity for authenticated users, you can create a relation table between **Visitor** and **User**.
Confidence can then use this table to calculate results for **User** metrics, even if the experiment randomizes on the **Visitor** entity.
This makes it possible to experiment on the sign-up process and measure how users behave when converting to customers.
## Create the Entity Relation Table
Open the entity that should own the relation (the entity you randomize on). In the **Entity relation tables** section, click **Create**.
Input a SQL query that outputs two columns that specifies the mapping between the entities. Select the columns and the target entity and click the **Create** to create the table.
Confidence doesn't clean the data coming from this table, so it's important that it's of high quality
to ensure trustable experiment results. Any required data cleaning can either be done before the data ends up in the
relation table, or inline in the table definition since it can be any SQL query.
Below is a short description of possible error cases and how those would affect the results, using the **Visitor** to **User** case as an example:
* No mapping exists for a visitor ID: For metrics with padding enabled, the user is included in the calculation of the metrics for the experiment, but get 0 as the metric value. Otherwise, the user is excluded.
* Multiple visitor ID's map to the same user: The user would be included once, with first exposure set to the earliest assignment for the visitor ID.
* One visitor ID maps to multiple users: All users mapped to the visitor ID would be considered exposed to the experiment.
## Create the Experiment
With the entity relation table in place, you can now create the experiment. Create an experiment and choose the entity that owns the relation as the entity to randomize on.
Then, when selecting metrics you should now see both metrics from both the entities in the metric picker that you can then configure as in any other experiment.
## Exposure Time Considerations
Confidence considers the entity exposed at the time when the assignment happens for the randomization entity. Using the **Visitor** to **User** example, that means that if a user visits the site and resolves a flag,
Confidence sets the exposure time to the time of when the flag was resolved and a variant assigned. Any **User** metrics with time windows and/or exposure offsets get calculated relative to that time,
which means that if there is a long time between the visit and the sign-up the metric might not be measuring what you want. Sometimes it might be good to add an extra exposure
offset to the metric to account for this.
## Common Pitfalls
### Wrong Use: General Entity Maps
**Problem**: trying to use entity relation tables to map users to orders for metric reuse across experiments.
**Why it doesn't work**: entity relation tables affect how Confidence calculates statistics. Using them for general mappings produces wrong variance calculations because the system assumes you're tracking the same entity transitioning between states, not separate entities with one-to-many relationships.
**Solution**: use ratio metrics for order-level analysis with user randomization. Create separate metrics for different randomization units. See [Ratio Metrics](./metrics#ratio-metrics) for details.
### Wrong Use of Metric Reuse Across Different Experiments
**Problem**: attempting to use entity relation tables to share metrics between experiments with different randomization units (for example, email-based users versus street addresses).
**Why it doesn't work**: each experiment needs metrics that match its randomization unit to ensure exact statistical analysis.
**Solution**: create separate metrics for each randomization unit. While this requires duplicate metric definitions, it ensures exact results.
## Detailed Example
**Scenario**: e-commerce site testing sign-up flow improvements, measuring both pre-sign-up browsing behavior and post-sign-up buying behavior.
**Implementation**: create an entity relation table mapping Visitor IDs to User IDs when users create accounts. This allows you to:
* Randomize on Visitor entities for all site visitors
* Measure User metrics like buying rate and order value
* Correctly attribute post-authentication behavior to the original experiment exposure
## Related Resources
Step-by-step relation table guide
Handle anonymous user experiments
Configure experiment entities
Alternative for granular analysis
# Exposure
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/exposure
Exposure defines when a user was first exposed to an experiment and to what variant.
To measure the impact of a change, you need to know when a user was first exposed to the change.
This time of exposure is the timestamp of the first assignment for a user.
## How Exposure Calculations Work
Exposure calculations use raw assignment data stored in an [Assignment Table](./assignment-tables).
The exposure calculations use four inputs:
1. An assignment table. Specifies what assignment table in the data warehouse to query.
2. A start and stop time. The times define what interval to query the assignment table for.
3. Exposure Key. The exposure calculation filters rows in the assignment table where the exposure
key is equal to the supplied value.
4. Variant Key. The calculation filters rows in the assignment table where the variant key is
equal to the given value. If you don't specify a variant key, the calculation doesn't filter on
the variant key.
### Storage
Exposure calculations write to your data warehouse.
You configure the destination schema and dataset when you set up the data warehouse connection.
The exposure calculations create a new table in the data warehouse for all new A/B tests and rollout.
### Exposure Time
Time of exposure is the timestamp of the first assignment for a user.
Later assignments don't affect the exposure calculation.
## Exposure Filtering
Exposure filters are also referred to as 'trigger analysis' in the literature and blogs.
Add an exposure filter on the experiment design page by clicking **Results**
on the left sidebar, then **Exposure filters**. You can add several exposure
filters per experiment. To keep interpretation of the results clear, use one or
two exposure filters.
Confidence produces metric results for each exposure filter (including no
filter) separately. You can see the results for any exposure filter (including no
filter) on the results page by selecting the corresponding filter in the top of
the results page.
You can see the results for all metrics with and without all exposure filters
simultaneously by clicking **Detailed results** in the top right corner of the
metrics result section in the results page. Confidence shows the exposure filter
column by default if the experiment has any exposure filters.
Exposure filters are methods for narrowing down more closely which users
to include in the exposure definition and the analysis of your experiment.
When you add an exposure filter for your experiment, the analysis only
includes the exposed users that also match the exposure filter. The time of
exposure is the first unit of time after default exposure where the user
matches the exposure filter.
You may want to use exposure filters if the default definition of exposure is
too broad for what you want to measure in your experiment.
Exposure filters don't affect who actually gets the experiment
experience. The filter only narrows down *who counts* as exposed in your A/B
test. Since the exposure filter event might happen some time after the base
exposure event, an exposure filter also affects when in time exposure happens,
which in turn determines who to include in the experiment results at different
points in time. For example, the exposure count without an exposure filter might
be 1000 at a given time point, but the exposure count with an exposure filter
might be 500 for the same time point. If you would like to limit who actually
sees the A/B test, you should instead use [inclusion criteria](/docs/experiments/audience).
**Example:** You run an A/B test where you add credits and links at the bottom of Spotify's Rap Caviar
playlist. Possible exposure definitions that range from less to more
restrictive include:
1. Launching the app
2. Opening a playlist
3. Opening Rap Caviar
4. Playing from the Rap Caviar playlist
5. Scrolling to the bottom of the Rap Caviar playlist
Which exposure definition is appropriate depends on the purpose of the A/B test.
To set up an exposure filter, you first need to create a fact table that has
information about the exposure criteria. For example, all users who visited a
certain page. You can then use this fact table to filter the exposure of the
experiment. When you add an exposure filter, experiment results include only
users that are both in the default exposure table and in the exposure filter
table. This filtering means that the analysis only includes users that fulfill
the exposure criteria, defined as being present in the fact table.
### Filter the Exposure Filter Fact Table
Select rows from the fact table you use as an exposure filter by
filtering the rows on values from columns in that fact table. If you for example
have several types of page views events in the same fact table, you can filter
out only the page views from a certain page to use as the exposure filter.
For exposure
filtering, you can only select fact tables that have a column of the type
`Entity` that matches the entity for the experiment.
### Health Checks with Exposure filters
Confidence runs all health checks with and without all exposure filters and sends
alert to the experiment owner if a health check fails.
## Schedules
When you run an experiment you typically don't want to calculate exposure just once but multiple times throughout the
runtime of the experiment. This recurring calculation is typically referred to as an Exposure Schedule in Confidence.
Although Confidence supports any variable intervals the built-in workflows come in two modes: hourly and daily.
For both of these modes the actual intervals are variable and slowly ramp up to the interval you've selected.
This ensures that you get quick feedback as soon as you've launched your experiment.
Here's an example of how an hourly schedule might look if launched at 12:46:45.
1. 3 minutes 15 seconds (12:46:45 to 12:50). This ensures that the schedule aligns with even minutes.
2. 5 minutes (12:50 to 12:55).
3. 10 minutes (12:55 to 13:05).
4. 15 minutes (13:05 to 13:20).
5. 40 minutes (13:20 to 14:00).
6. 60 minutes (14:00 to 15:00). The schedule remains at 60 minutes after reaching the last interval.
The built-in workflows manage your schedules for you.
## Related Resources
Step-by-step exposure filter guide
Configure assignment data sources
Set up exposure filter data sources
# Fact Tables
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/fact-tables
Fact tables contain measurements that describe your entities.
The measurements in your fact tables are the foundations of your metrics.
These facts describe your entities and their behavior.
For example, your fact tables can contain events that describe when a user bought an item,
or measurements that describe the number of times a user listened to a song.
## Configuration
### SQL Query
You need to input the SQL that selects your facts. After you've entered your
SQL you can click **Run Query**. Confidence then executes your query with a
limit to check that everything is in order and show you some sample rows. Here is a
checklist of things to consider when writing the query for your fact data:
* ✅ Always have an index (also called [Time partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#date_timestamp_partitioned_tables) in BigQuery) on the timestamp
column to ensure that Confidence only queries the data within a partition, without
this each query may require a full table scan.
* ✅ If you can, also set up an index on the entity columns (also called [Clustered Table](https://cloud.google.com/bigquery/docs/clustered-tables) in BigQuery),
this helps improve the performance of the join that Confidence does between your exposures
and facts.
* ✅ If you can, perform heavy transformations in an upstream step and store the result in an
intermediate table to avoid potentially redoing the transformations.
* ✅ Remove duplicate rows (multiple rows per entity is fine) to avoid double counting in your
metrics.
* ✅ Check that the distribution of nulls, zeros and other anomalous values in your data is as
you expect to ensure that your metrics are precise.
You have some placeholders at your disposal if you need to specify what
partition you're querying: `{START_TIME}` and `{END_TIME}`. These two
parameters vary depending on the requirements of the metric that uses
the fact table. You don't need
to use them as Confidence always filters on your timestamp column, but
sometimes your table format includes the date. These placeholders come as the
Timestamp type of your data warehouse.
### Entities
You need to specify what entities the facts belong to.
You must have at least one, but you can also have more.
Typically, you might have a user that triggered the fact, but may also have related entities in the
same fact if the user interacted with other entities.
### Timestamp Column
The timestamp column is the column that identifies the time that the fact
occurred. This column can either be a timestamp with a specific time point for
the fact, or you can select a date column. Selecting a date column indicates
that any metric calculation that includes this day should include this fact.
It's the same as specifying a time that is at 23:59:59.
### Measurements
A measurement is the value that you aggregate into a metric.
The measurement column can either be numeric, a boolean, or a HyperLogLog (HLL) sketch.
A boolean measurement treats `true` as the number `1`, and `false` as the number `0`.
HLL measurements can only use the "Approximate count distinct" aggregation in metrics.
### Dimensions
A dimension is a value that you use to categorize a fact. A typical
example is assigning a "Device" or "Browser" dimension. Unlike dimensions
defined by a [dimension table dimensions](./dimension-tables), fact table dimensions
are usually determined at runtime and can have different values for the
same entity within a metric.
### Data Delivery Cadence
For configuring data delivery cadence, see the
[data delivery cadence](/docs/metrics/delivery-cadence) page.
## Related Resources
Step-by-step fact table guide
Build metrics from fact tables
Add segmentation dimensions
# Introduction to Metrics
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/introduction
Confidence Metrics is a managed service that computes metrics for experiments. Use Confidence Metrics integrated with your data warehouse if you are a Warehouse-Native Confidence user, or, base your metrics on events that you track and emit directly to Confidence with the Confidence SDKs if you are a Total Confidence user.
## Get Started
To use Confidence you need to have a cloud data warehouse and give permissions
to Confidence to read and write data to the warehouse. You can limit access so
Confidence only has access to the data it needs.
Follow one of the quickstart guides to set up Confidence with your data
warehouse:
Google BigQuery
Snowflake
Redshift
Databricks
After you have connected Confidence with your data warehouse, follow the
metrics quickstart guide to create a metric.
## What is a Metric?
In Confidence, a metric is a way to measure the behavior of a user, a system or
something else of interest. Use metrics to verify the success of an
experiment, and to decide if a change has had a positive or negative impact
on the exposed audience.
The key concepts for defining a metric are:
The unit that you want to measure, such as a user. [Entities](/docs/metrics/entities) are
identified by a unique ID, like a user ID.
A query that returns data about how entities were assigned to experiments and
variants. Confidence uses assignments to compute exposure for an experiment.
A query that returns data
about "facts" or measurements of interest, such as sales revenue, quantity
sold, number of customer orders, or any other measurable data points.
## How Confidence Computes Metrics
To compute a metric, Confidence goes through the following steps:
Exposure aggregates assignment data. The assignment data can have multiple
records of when a user was assigned to a variant. When computing exposure, Confidence
identifies the first time a user was assigned to a variant and uses that as the
exposure record. **First exposure** is the timestamp for when the user was first exposed,
which metric calculations use to get the right facts.
For metrics that are measuring an average across entities, metrics compute a
per-entity aggregation as a first step. This aggregation step reduces multiple facts per
entity into a single value.
You select the aggregation function (for example the sum, count, or average) in the metric
definition.
The metric calculation joins the per-entity measurements, or facts for metrics that do not require
a per-entity aggregation, with exposure.
After the join, the calculation averages the per-entity measurements per variant.
The first exposure timestamp selects the correct per-entity measurement, taking into account the
exposure offset parameter of the metric.
To improve performance, Confidence caches the data by writing the intermediate steps, such as the
exposure tables, to your data warehouse.
## Related Resources
Step-by-step metric setup tutorial
Metric creation guide
Deep dive into metric types
Configure measurement data sources
# Metrics
Source: https://confidence-auth-testing.mintlify.io/docs/metrics/metrics
Metrics define how to aggregate your measurements.
A metric is an aggregation of a measurement across instances of an entity. A measurement typically
occurs when an instance of an entity, such as a user, interacts with some parts of the business.
Each such measurement event is then recorded in a fact table that has all the measurement
events of the business process of interest.
Confidence supports two types of metrics:
* **Average metrics** calculate the means of the treatment groups.
The calculation takes the average of the entity measurements after first aggregating within each entity.
For example, a user can have multiple data points.
In an average metric, these data points are first aggregated separately for each user, for example, by summing the data points.
In an experiment, the calculation averages the within-user aggregated measurements in each group.
The average metric is the most common type of metric.
* **Ratio metrics** calculate the ratios between the numerator and the denominator measurements in the treatment groups.
The calculation separately aggregates the numerator and denominator measurements across entities.
For example, let the numerator measurement be the number of successful searches, and the denominator
measurement be the number of searches.
The two measurements measure the user entity.
The ratio metric calculates the ratio of the total number of successful searches to the total number of searches.
### Choosing Between Average Metrics, Ratio Metrics, and Entity Relations
**Use Average Metrics when:**
* Analyzing at the same level as randomization (user-level metrics with user randomization)
* You want the average value per entity
* Each entity has the same weight in the analysis
**Use Ratio Metrics when:**
* Analyzing at a finer granularity than randomization (order-level metrics with user randomization)
* Calculating rates or per-unit averages (click-through rate, average order value)
* You have one-to-many relationships between randomization and measurement units
**Use Entity Relation Tables when:**
* ONLY for connecting anonymous users who later authenticate
* You need to track the same user transitioning from anonymous to authenticated state
* Never for general entity mapping or metric reuse across different experiments
For more details, see [Entity Relation Tables](./entity-relation-tables).
## Filtered Metrics
Filter rows in fact tables to create more specialized metrics. For example,
create an unfiltered metric `Number of purchases` and a filtered version `Number
of purchases on mobile`. You can create filters using any combination of logical
expressions based on fact table columns.
You can only filter metrics on
measurements and dimensions available in the same fact table that you create the metric
from.
You can create filtered attributes in your own data warehouse or in the metrics
definition in Confidence.
## Time in Metrics
All metric aggregates measurements over some time window. In Confidence, you
can configure the behavior of the time window in three ways:
**Include the user in metrics results**:
1. **At the end of a window.** Example: Second and third days' consumption. Includes all consumption during the second and third days after exposure. Includes an entity in the results **at the end of their third day of exposure**.
2. **Cumulatively during a window.** Example: Second and third days' consumption. Includes all consumption during the second and third days after exposure. Includes an entity in the results **at the beginning of their second day of exposure**.
3. **Cumulatively (no window).** Example: Average order value. Includes all orders after exposure. Includes all exposed entities.
Watch this video for an overview of the different ways to handle time in metrics in 4 minutes and 34 seconds.
For logged-in, or in other ways persistent users, it often makes sense to
consider a time window. Time-window based metrics make interpretation easier,
as you can be sure that they include the same measurements (in relation to
exposure) for all entities included in the metrics results. For experiments on
short-lived entities like short-lived cookies, using windows over for example
several days makes little sense, as one user is unlikely to come back. Even
if they do, they no longer have the same identifier. In such cases,
windows are redundant.
The figure describes how to choose the time window behavior. See more details
about the trade-offs below.
### Windows-Based Metrics
Window-based metrics measure behavior over a time window, where time is relative to exposure.
To define a window-based metric you need to specify an **exposure offset** and an **aggregation window**.
These parameters specify when the aggregation of data should start relative to exposure, and for how long the
aggregation should be applied.
Illustration of exposure offset and windows in metrics.
The benefit of window-based metrics is that they clearly define what you measure. Experiments are
often subject to novelty effects, where the effect of the treatment is strongest immediately after
exposure. By defining a time window, you have better control of when you measure the behavior you're
interested in. It treats all units in your experiment equally. If they were exposed early or late
has no bearing on the metric.
### At the End of a Window or Cumulatively During a Window
You can select two types of window-based metrics in Confidence: **At the end of a
window** and **Cumulatively during a window**.
#### At the End of a Window
Metrics that include entities in the results **At the end of a window** include
the measurements from exposed entities at the end of the window used by the metric.
The figure illustrates how time windows work to aggregate values within
entities. The horizontal bar shows when the user was first exposed to the
experiment. The dashed part of the box is the exposure offset, and the solid
box is the aggregation window. The metric calculation includes the measurements
that fall within the aggregation window.
Illustration of including the user in metrics results at the end of a window. The metrics result at the given day of the experiment include the measurements in the green parts.
'At the end of a window' metrics show no data before `exposure offset + aggregation window - 1` time units after the start of the experiment.
For example, if you create a metric that measures user behavior during the second week after exposure, the first results
you see for the metric are available 14 days after launch. Before that day, no unit has been exposed for two weeks.
#### Cumulatively During a Window
Metrics that
include entities in the results **Cumulatively during a window** include
measurements from exposed entities cumulatively during the window.
Illustration of including the user in metrics results cumulatively during a window. The metrics
result at the given day of the experiment include the measurements in the green parts.
'Cumulatively during a window' metrics show no data before
`exposure offset` time units after the start of the experiment. After that, they
show the data available for each entity in the time window. For example, if you
create a metric that measures user behavior during the second week after
exposure, the first results you see for the metric are available 7 days after
launch, at which the metric results include measurements from the eighth day of exposure
for the users exposed on the first day of the experiment.
#### Trade-Offs Between At the End of a Window and Cumulatively During a Window
Metrics that use **At the end of a window** are the most rigorous, because they include exactly the
same measurements for the entities included in the metric results at any given time. Metrics that
use **Cumulatively during a window** return results earlier because they don't need to wait for the
window to end. The downside is that they are harder to interpret as the result doesn't measure each
entity over the same length of time after exposure.
If all entities are eventually exposed for at least the upper limit of the window, the two types of
window metrics give the same results at that point. For this reason, the **Cumulatively during a
window** option offers a compromise between early results and ease of interpretation.
### Cumulatively (No Window)
Metrics that are not window-based include all measurements from all entities in the metric results
as soon as an entity is exposed. Metrics without windows are suitable when entities are short-lived,
as with for example cookie-based entities. In this case, following the same
entity over time is often difficult which makes metric windows redundant.
Illustration of including the user in metrics cumulatively without any window.
In cumulative metrics (without window), some entities are always measured
over longer periods than others due to that not all entities are exposed at the
same time. This makes these metrics hard to interpret. Only use these metrics
when entities are short-lived.
## Average Metrics
An average metric measures the average value across the different entities.
Typically, you have multiple facts measured within an entity.
You must select what aggregation to apply within the entity before calculating the average across entities.
Examples of average metrics include:
| Metric | Description |
| :------------------------------ | :--------------------------------------------------------------------------------------------------------------------------- |
| Average minutes played per user | Sum the minutes played for each user, then average across users |
| Share of users that were active | Count the number of events per user and map to 1 if the user was active and 0 otherwise, then average across users |
| Conversion rate | Count the number of conversion events per user and map to 1 if the user converted and 0 otherwise, then average across users |
### Within-Entity Aggregation
You can select to aggregate multiple facts within entities using one of:
| Aggregation | Description |
| :------------------------- | :----------------------------------------------------------------------- |
| Average | Average of the non-null values |
| Count | Count the non-null values |
| Count distinct | Count the number of distinct non-null values |
| Approximate count distinct | Approximate count of distinct non-null values from a HyperLogLog sketch. |
| Sum | Sum of the values |
| Max | Maximum of the non-null values |
| Min | Minimum of the non-null values |
The below examples use these facts.
| Entity | Value |
| :----- | :---- |
| A | 1 |
| A | 3 |
| A | 8 |
| A | null |
| B | 3 |
| B | 1 |
| B | 29 |
The average calculates the average of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 4 |
| B | 11 |
The metric value is (4 + 11)/2 = 7.5.
Count counts the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 3 |
| B | 3 |
The metric value is (3 + 3)/2 = 3.
Count distinct counts the number of distinct non-null within-entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 3 |
| B | 3 |
The metric value is (3 + 3)/2 = 3.
Approximate count distinct uses a HyperLogLog sketch to approximate the number of distinct non-null within-entity values. This aggregation is only available for HLL measurement columns. For example, given HLL sketch values `{seen:['id1']}` for entity A and `{seen:['id2', 'id3']}` for entity B, the result would be approximately
| Entity | Value |
| :----- | :---- |
| A | 1 |
| B | 2 |
The metric value is approximately (1 + 2)/2 = 1.5.
Max calculates the max of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 8 |
| B | 29 |
The metric value is (8 + 29)/2 = 18.5.
Min calculates the min of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 1 |
| B | 1 |
The metric value is (1 + 1)/2 = 1.
Sum calculates the sum of the within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 12 |
| B | 33 |
The metric value is (12 + 33)/2 = 22.5.
## Ratio Metrics
Ratio metrics aggregate the numerator and denominator separately across all entities, without first aggregating within entities. Ratio metrics differ from average metrics, which first aggregate within each entity.
### When to Use Ratio Metrics
Use ratio metrics when:
* Your unit of analysis is more granular than your randomization unit
* You want to analyze metrics at the order, session, or page view level while randomizing at the user level
* You need to calculate metrics like average order value or click-through rate
* You have one-to-many relationships between your randomization unit and measurement unit
**Example scenario**: if you randomize by users but want to calculate average pickup time per order:
* Use a ratio metric with `SUM(pickup_time) / COUNT(order_id)`
* This correctly accounts for users having different numbers of orders
* The variance calculation properly handles the user-level randomization
**Note**: Ratio metrics differ from entity relation tables, which only connect anonymous to authenticated users. If you're trying to analyze orders while randomizing on users, use ratio metrics, not entity relation tables.
Examples of ratio metrics include:
| Metric | Description |
| :------------------ | :--------------------------------------------------------------------------------------------------------------------------- |
| Click-through rate | Sum a binary measurement indicating a click in the numerator, count the number of events, or impressions, in the denominator |
| Average order value | Sum the order value in the numerator, count the number of orders in the denominator |
### Numerator and Denominator Aggregations
You can select to aggregate multiple facts for the numerator and denominator using one of:
| Aggregation | Description |
| :------------------------- | :----------------------------------------------------------------------- |
| Average | Average of the non-null values |
| Count | Count the non-null values |
| Count distinct | Count the number of distinct non-null values |
| Approximate count distinct | Approximate count of distinct non-null values from a HyperLogLog sketch. |
| Sum | Sum of the values |
| Max | Maximum of the non-null values |
| Min | Minimum of the non-null values |
The below examples use these facts.
| Entity | Measurement A | Measurement B |
| :----- | :------------ | :------------ |
| A | 1 | 8 |
| A | 3 | 3 |
| A | 8 | 2 |
| A | null | 3 |
| B | 3 | 9 |
| B | 1 | 1 |
| B | 29 | 2 |
#### Numerator
The numerator examples use measurement A from the example table above.
The average calculates the average of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 4 |
| B | 4 |
The numerator value is 4 + 11 = 15.
Count counts the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 3 |
| B | 3 |
The numerator value is 3 + 3 = 6.
Count distinct counts the number of distinct non-null within-entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 3 |
| B | 3 |
The numerator value is 3 + 3 = 6.
Approximate count distinct uses a HyperLogLog sketch to approximate the number of distinct non-null within-entity values. This aggregation is only available for HLL measurement columns. For example, given HLL sketch values `{seen:['id1']}` for entity A and `{seen:['id2', 'id3']}` for entity B, the result would be approximately
| Entity | Value |
| :----- | :---- |
| A | 1 |
| B | 2 |
The numerator value is approximately 1 + 2 = 3.
Max calculates the max of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 8 |
| B | 29 |
The numerator value is 8 + 29 = 37.
Min calculates the min of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 1 |
| B | 1 |
The numerator value is 1 + 1 = 2.
Sum calculates the sum of the within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 12 |
| B | 33 |
The numerator value is 12 + 33 = 45.
#### Denominator
The denominator examples use measurement B from the example table above.
The average calculates the average of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 4 |
| B | 4 |
The denominator value is 4 + 4 = 8.
Count counts the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 4 |
| B | 3 |
The denominator value is 4 + 3 = 7.
Count distinct counts the number of distinct non-null within-entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 3 |
| B | 3 |
The numerator value is 3 + 3 = 6.
Approximate count distinct uses a HyperLogLog sketch to approximate the number of distinct non-null within-entity values. This aggregation is only available for HLL measurement columns. For example, given HLL sketch values `{seen:['id1']}` for entity A and `{seen:['id2', 'id3']}` for entity B, the result would be approximately
| Entity | Value |
| :----- | :---- |
| A | 1 |
| B | 2 |
The denominator value is approximately 1 + 2 = 3.
Max calculates the max of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 8 |
| B | 9 |
The numerator value is 8 + 9 = 17.
Min calculates the min of the non-null within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 2 |
| B | 1 |
The numerator value is 2 + 1 = 3.
Sum calculates the sum of the within entity values. The example table would result in
| Entity | Value |
| :----- | :---- |
| A | 16 |
| B | 12 |
The numerator value is 16 + 12 = 28.
#### Ratio
Assume that measurement A measures some aspect of a click and is non-null if the entity (like a visitor or a user) clicked.
Measurement B measures the corresponding impression and some aspect of it.
To calculate a click-through rate, use a count aggregation for the numerator and a count aggregation for the denominator.
The ratio is the number of clicks divided by the number of impressions, which equals (3 + 3)/(4 + 3) = 6/7.
Assume that measurement A measures the order value, with null values indicating no completed order.
To calculate the average order value, use a sum aggregation for the numerator and a count aggregation for the denominator.
Use measurement A for both. The average order value is (12 + 33)/(3 + 3) = 45/6.
## Metric Parameters
### Aggregation Window
The aggregation window controls the size of the window to include facts from. This time is always relative to
the time of exposure for the entity. Confidence only calculates metrics for an entity at the end of the window.
For example, if the aggregation window is 1 hour, an entity exposed on 14:50 would find facts between
14:50 and 15:50.
Use aggregation windows that are compatible with the frequency of your facts.
For example, you shouldn't use hourly windows if your fact table has facts
with a frequency of one per day.
### Exposure Offset
You can optionally move the window of aggregation with an offset from the time of exposure.
Sometimes the effect that you want to measure is not the effect of recently seeing the new feature, but instead
how your entities react to the new feature after some more time has passed.
### Variance Reduction With CUPED
Confidence applies [variance reduction](/docs/experiments/stats/variance-reduction) (commonly referred to as CUPED) by default.
You can turn it off, or change the aggregation window used.
The facts from before the time of exposure make it possible to reduce the variance of the metric.
By default, Confidence includes the facts that occurred one aggregation window before exposure.
Variance reduction leverages pre-exposure data to reduce the variance and increase
the signals in the metrics you select. [Deng et al. (2013)](https://dl.acm.org/doi/10.1145/2433396.2433413)
popularized this approach, commonly
referred to as CUPED, which is a form of adjustment using data collected before exposure.
The result of this adjustment is a reduction in the variance of the metric and, by extension,
the uncertainty in the estimate of the treatment effect. You can use variance reduction for
both average metrics and ratio metrics.
You should always use variance reduction, as it leads to a better and more
precise answer to the question of what the effect of the treatment was.
### Cap Values
You can cap the values of the metric to a certain range. Use this to reduce the
influence of outliers. The cap trims the metric value for each entity at the
given values. For example, a user metric that caps the maximum value at 100
replaces all user values that exceed 100 with 100. The replacement occurs after
the aggregation of the metric values for each entity, but before aggregating the
metric across entities. For example, consider a metric that sums order value for
each user. If the metric caps the maximum value at 100, the cap applies to each
user's total order value and replaces total order values that exceed 100 with
100\.
## Related Resources
Step-by-step metric creation guide
Verify your metrics work correctly
Configure measurement data sources
Improve metric precision with CUPED
# Migrate from Eppo to Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/migrations/migrate-from-eppo
Learn how to migrate feature flags, allocations, and audiences from Eppo to Confidence with the AI-powered migration kit.
Run this skill to migrate from Eppo to Confidence:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins --skill migrate-eppo
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants, and asks which one to install to. You can also install the full [Confidence plugin](/docs/migrations/overview#install-the-confidence-plugin) with your assistant's own plugin manager.
The kit moves your feature flags, allocations, and audiences to Confidence, then rewrites your application code from the Eppo SDK to the Confidence SDK. See the [migrations overview](/docs/migrations/overview) for how migration kits work.
## Before You Begin
To run this migration kit, you need:
* The [Confidence plugin](/docs/migrations/overview#install-the-confidence-plugin) installed in your AI assistant
* An Eppo API key—not an SDK key—with read access to feature flags, stored in the `EPPO_API_KEY` environment variable. You can create one under **Admin > API Keys** in the Eppo dashboard.
* The Confidence Flags MCP server, authenticated with your account
* The Confidence Documentation MCP server, used during code transformation
## Run the Migration
Invoke the kit with the `/confidence:migrate-eppo` command in your AI assistant.
Run `/confidence:migrate-eppo plan flag`. The kit scans your Eppo flags, asks you to choose a Confidence client and map the subject entity, and writes a migration plan file.
The plan lists every flag with its allocations and targeting rules, and marks anything that can't migrate automatically. Plan files are resumable, so you can pause and continue later.
Run `/confidence:migrate-eppo execute `. The kit recreates your flags in Confidence with one targeting rule per allocation, in the same waterfall order.
Run `/confidence:migrate-eppo plan code`, review the plan, then execute it. If your app already uses OpenFeature, the kit swaps the registered provider and leaves call sites unchanged. Otherwise it rewrites each Eppo SDK call to the Confidence SDK, creating one pull request per flag.
## What Gets Migrated
| Eppo | Confidence |
| ------------------------------- | --------------------------------------------------------------- |
| Feature flag with allocations | Flag with one targeting rule per allocation, in waterfall order |
| Reusable audience | Segment, created once and referenced by multiple flags |
| `subjectKey` randomization unit | Entity field of your choice, such as `user_id` |
| Default allocation | Final catch-all targeting rule at 100% |
| Variation value | Variant assigned within a rule |
Confidence has no server-side flag default. The kit reproduces Eppo defaults as a final catch-all targeting rule that serves the default variant.
## Known Limitations
The kit flags these Eppo features for manual review because Confidence doesn't support them:
* Switchback allocations that rotate exposure in time windows
* General regular expression matching—only prefix, suffix, and alternation patterns convert cleanly
* Contextual bandits (`getBanditAction`), which need a redesign in Confidence
## Analyze Historical Experiments
The migration kit moves your flags and code, not your historical experiment results. To re-analyze experiments that ran on Eppo, import their assignment data into your data warehouse and follow the [Analyze a Past Experiment](/docs/quickstarts/analyze-past-experiment) quickstart.
## Related Resources
How migration kits work and how to install the plugin
Set up the Confidence MCP servers for your AI assistant
Re-analyze experiments that ran on another platform
Integrate Confidence SDKs into your application
# Migrate from Optimizely to Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/migrations/migrate-from-optimizely
Learn how to migrate flags, rules, and audiences from Optimizely Feature Experimentation to Confidence with the AI-powered migration kit.
Run this skill to migrate from Optimizely to Confidence:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins --skill migrate-optimizely
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants, and asks which one to install to. You can also install the full [Confidence plugin](/docs/migrations/overview#install-the-confidence-plugin) with your assistant's own plugin manager.
The kit moves your flags, rules, and audiences from Optimizely Feature Experimentation to Confidence, then rewrites your application code from the Optimizely SDK to the Confidence SDK. See the [migrations overview](/docs/migrations/overview) for how migration kits work.
## Before You Begin
To run this migration kit, you need:
* The [Confidence plugin](/docs/migrations/overview#install-the-confidence-plugin) installed in your AI assistant
* An Optimizely personal access token or service account token with read access to flags, rules, and audiences, stored in the `OPTIMIZELY_API_TOKEN` environment variable. You can create one under **Account Settings > API Access** in the Optimizely app.
* The Confidence Flags MCP server, authenticated with your account
* The Confidence Documentation MCP server, used during code transformation
* Optional: a Confidence REST API token stored in the `CONFIDENCE_TOKEN` environment variable, required for partial traffic allocation, reusable audiences as segments, and exclusion groups
## Run the Migration
Invoke the kit with the `/confidence:migrate-optimizely` command in your AI assistant.
Run `/confidence:migrate-optimizely plan flag`. The kit scans your Optimizely flags, rulesets, and audiences and writes a migration plan file.
The plan lists every flag with its rules and traffic allocation, and marks anything that can't migrate automatically. Plan files are resumable, so you can pause and continue later.
Run `/confidence:migrate-optimizely execute `. The kit recreates your flags in Confidence with their variations, audiences, and rules in priority order.
Run `/confidence:migrate-optimizely plan code`, review the plan, then execute it. If your app already uses OpenFeature, the kit swaps the registered provider and leaves call sites unchanged. Otherwise it rewrites each Optimizely SDK call to the Confidence SDK, creating one pull request per flag.
The kit migrates stable flags only. It excludes live A/B tests because Confidence uses a different bucketing hash, so migrating a running test would reshuffle its users. Conclude running experiments before migrating, or restart them in Confidence.
## What Gets Migrated
| Optimizely | Confidence |
| -------------------------------------- | -------------------------------------------------------------- |
| Flag | Flag |
| Variation with variable values | Variant with a payload |
| Targeted delivery or A/B rule | Targeting rule, one per Optimizely rule in priority order |
| Audience | Segment with the REST API token, or criteria copied into rules |
| Traffic allocation and variation split | Variant allocations inside the rule |
| Default variation | Final catch-all targeting rule |
| Bucketing ID | Entity field of your choice, such as `user_id` |
## Known Limitations
The kit flags these Optimizely features for manual review because Confidence doesn't support them:
* Substring and general regular expression matching
* Presence conditions that check whether an attribute exists
* Audience conditions based on browser, device, query parameter, cookie, or location
* Multi-armed bandits, because Confidence allocations are static
* Partial rollouts with fall-through and exclusion groups, unless you use the REST API token
## Analyze Historical Experiments
The migration kit moves your flags and code, not your historical experiment results. To re-analyze experiments that ran on Optimizely, export your decision events to your data warehouse and follow the [assignment table from Optimizely decision events](/docs/quickstarts/analyze-past-experiment#assignment-table-from-optimizely-decision-events) guide. The [analysis workflow](/docs/experiments/workflows/analysis) page explains the process in more depth.
## Related Resources
How migration kits work and how to install the plugin
Set up the Confidence MCP servers for your AI assistant
Re-analyze experiments that ran on another platform
Integrate Confidence SDKs into your application
# Migrate from PostHog to Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/migrations/migrate-from-posthog
Learn how to migrate feature flags and multivariate flags from PostHog to Confidence with the AI-powered migration kit.
Run this skill to migrate from PostHog to Confidence:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins --skill migrate-posthog
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants, and asks which one to install to. You can also install the full [Confidence plugin](/docs/migrations/overview#install-the-confidence-plugin) with your assistant's own plugin manager.
The kit moves your feature flags and multivariate flags to Confidence, then rewrites your application code from the PostHog SDK to the Confidence SDK. See the [migrations overview](/docs/migrations/overview) for how migration kits work.
## Before You Begin
To run this migration kit, you need:
* The [Confidence plugin](/docs/migrations/overview#install-the-confidence-plugin) installed in your AI assistant
* The PostHog MCP server, authenticated with your PostHog account. For example, in Claude Code:
```bash theme={null}
claude mcp add posthog --transport http --url "https://mcp-eu.posthog.com/mcp"
```
* The Confidence Flags MCP server, authenticated with your account
* The Confidence Documentation MCP server, used during code transformation
## Run the Migration
Invoke the kit with the `/confidence:migrate-posthog` command in your AI assistant.
Run `/confidence:migrate-posthog plan flag`. The kit scans your PostHog feature flags, asks you to choose a Confidence client and entity, and writes a migration plan file.
The plan lists every flag with its targeting conditions and rollout percentages, and marks anything that can't migrate automatically. Plan files are resumable, so you can pause and continue later.
Run `/confidence:migrate-posthog execute `. The kit recreates each flag in Confidence with its targeting rules intact.
Run `/confidence:migrate-posthog plan code`, review the plan, then execute it. If your app already uses OpenFeature, the kit swaps the registered provider and leaves call sites unchanged. Otherwise it rewrites each PostHog SDK call to the Confidence SDK, creating one pull request per flag.
## What Gets Migrated
| PostHog | Confidence |
| ----------------------- | ---------------------------------------------------------------- |
| Feature flag | Flag with matching targeting rules |
| Multivariate flag | Variant assignments within a single rule, with percentage splits |
| `distinct_id` bucketing | Entity field of your choice, such as `user_id` |
| Group-based bucketing | Group entity reference |
| Targeting conditions | Criteria and expression rules |
| Rollout percentage | Rollout percentage on the targeting rule |
## Known Limitations
The kit flags these PostHog features for manual review because Confidence doesn't support them:
* The `icontains` substring operator
* The `is_not_set` condition
* Cohort targeting
## Analyze Historical Experiments
The migration kit moves your flags and code, not your historical experiment results. To re-analyze experiments that ran on PostHog, import their assignment data into your data warehouse and follow the [Analyze a Past Experiment](/docs/quickstarts/analyze-past-experiment) quickstart.
## Related Resources
How migration kits work and how to install the plugin
Set up the Confidence MCP servers for your AI assistant
Re-analyze experiments that ran on another platform
Integrate Confidence SDKs into your application
# Migrate from Statsig to Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/migrations/migrate-from-statsig
Learn how to migrate feature gates, dynamic configs, and experiments from Statsig to Confidence with the AI-powered migration kit.
Run this skill to migrate from Statsig to Confidence:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins --skill migrate-statsig
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants, and asks which one to install to. You can also install the full [Confidence plugin](/docs/migrations/overview#install-the-confidence-plugin) with your assistant's own plugin manager.
The kit moves your feature gates, dynamic configs, and experiments to Confidence, then rewrites your application code from the Statsig SDK to the Confidence SDK. See the [migrations overview](/docs/migrations/overview) for how migration kits work.
## Before You Begin
To run this migration kit, you need:
* The [Confidence plugin](/docs/migrations/overview#install-the-confidence-plugin) installed in your AI assistant
* A Statsig Console API key (starts with `console-`) with read access to gates, dynamic configs, and experiments, stored in the `STATSIG_API_KEY` environment variable. You can create one under **Project Settings > API Keys** in the Statsig console.
* The Confidence Flags MCP server, authenticated with your account
* The Confidence Documentation MCP server, used during code transformation
* Optional: a Confidence REST API token stored in the `CONFIDENCE_TOKEN` environment variable, required for partial experiment allocation, reusable segments, layer mutual exclusion, and holdouts
## Run the Migration
Invoke the kit with the `/confidence:migrate-statsig` command in your AI assistant.
Run `/confidence:migrate-statsig plan flag`. The kit scans your Statsig gates, dynamic configs, and experiments and writes a migration plan file.
The plan lists every flag with its targeting rules and rollout percentages, and marks anything that can't migrate automatically. Plan files are resumable, so you can pause and continue later.
Run `/confidence:migrate-statsig execute `. The kit recreates your gates, configs, and experiments in Confidence with matching rules and variant splits.
Run `/confidence:migrate-statsig plan code`, review the plan, then execute it. The kit rewrites your Statsig SDK call sites to OpenFeature with the Confidence SDK, creating one pull request per flag.
## What Gets Migrated
| Statsig | Confidence |
| ------------------ | ------------------------------------------------------------------------ |
| Feature gate | Boolean flag with `enabled`/`disabled` variants |
| Dynamic config | Struct flag where each rule's return value becomes a variant |
| Experiment | Struct flag where each group becomes a variant with its percentage split |
| Segment | Segment, or conditions copied into targeting rules |
| Layer | Mutual-exclusion group (requires the REST API token) |
| Rollout percentage | Rollout percentage on the targeting rule |
## Known Limitations
The kit flags these Statsig features for manual review because Confidence doesn't support them:
* Substring conditions (`str_contains_any`, `str_contains_none`) and general regular expression matching
* Conditions that depend on another experiment's assignment (`experiment_group`)
* Custom JavaScript conditions
* Custom event logging (`logEvent`)—Confidence doesn't expose a matching API
* Large ID list segments and analysis-only segments without the REST backend
## Analyze Historical Experiments
The migration kit moves your flags and code, not your historical experiment results. To re-analyze experiments that ran on Statsig, import their assignment data into your data warehouse and follow the [Analyze a Past Experiment](/docs/quickstarts/analyze-past-experiment) quickstart.
## Related Resources
How migration kits work and how to install the plugin
Set up the Confidence MCP servers for your AI assistant
Re-analyze experiments that ran on another platform
Integrate Confidence SDKs into your application
# Migrate to Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/migrations/overview
Migrate feature flags and experiments from PostHog, Eppo, Statsig, or Optimizely to Confidence using AI-powered migration kits.
The [Confidence plugin for AI coding assistants](https://github.com/spotify/confidence-ai-plugins) includes migration kits that automate moving from PostHog, Eppo, Statsig, or Optimizely to Confidence. Each kit guides your AI assistant through recreating your flags in Confidence and rewriting your application code to use the Confidence SDK. Install the kits and pick the one for your platform:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants, and asks which skills and assistant to install to. To get the full plugin—including flag management and documentation search—install it with your assistant's own plugin manager instead.
The plugin works with [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Cursor](https://www.cursor.com/), [Codex](https://developers.openai.com/codex/), and [Gemini CLI](https://github.com/google-gemini/gemini-cli).
## How Migration Works
Every migration kit runs in two phases:
1. **Flag definitions**: The kit reads your flags, targeting rules, and rollout percentages from the source platform and recreates them in Confidence.
2. **Code transformation**: The kit rewrites your application code from the source platform's SDK to [OpenFeature](https://openfeature.dev/) with the Confidence SDK, creating one pull request per flag.
Each phase separates planning from execution. The kit first writes a plan file that you review, then executes it:
* `plan flag` generates a migration plan for your flag definitions
* `plan code` generates a migration plan for your code changes
* `execute ` carries out the plan you generated
Plan files are self-contained and resumable. You can pause a migration, review what remains, and continue later—even with a different AI assistant.
## Install the Confidence Plugin
```bash theme={null}
claude plugin install confidence
```
Go to **Settings > Plugins**.
Search for **Confidence** and select **Install**. The plugin is also available on the [Cursor marketplace](https://cursor.com/marketplace/spotify).
```bash theme={null}
codex plugin marketplace add spotify/confidence-ai-plugins
```
```bash theme={null}
gemini extensions install https://github.com/spotify/confidence-ai-plugins
```
## Before You Begin
To run any migration kit, you need:
* A [Confidence](https://app.confidence.spotify.com) account with a client to resolve flags
* The Confidence Flags MCP server, authenticated with your account—see the [Use AI with Confidence](/docs/quickstarts/use-mcp) quickstart
* The Confidence Documentation MCP server, used during code transformation
* Read access to your source platform, typically an API key—see each migration kit page for details
Some advanced features, such as partial traffic allocation and reusable segments, require a Confidence REST API token in addition to the MCP server. The kit tells you when this applies.
## Migration Kits
Move feature flags and multivariate flags with the `/confidence:migrate-posthog` command.
Move feature flags, allocations, and audiences with the `/confidence:migrate-eppo` command.
Move feature gates, dynamic configs, and experiments with the `/confidence:migrate-statsig` command.
Move flags, rulesets, and audiences with the `/confidence:migrate-optimizely` command.
The plugin also includes skills for onboarding new Confidence accounts and setting up a data warehouse connection. See the [plugin repository](https://github.com/spotify/confidence-ai-plugins) for the full list.
## Related Resources
Set up the Confidence MCP servers for your AI assistant
Complete reference for all MCP tools and capabilities
Re-analyze experiments that ran on another platform
Integrate Confidence SDKs into your application
# Activity Feeds
Source: https://confidence-auth-testing.mintlify.io/docs/notifications/activity-feeds
Follow everything that happens for a certain type of resource in Confidence.
Activity feeds let you follow everything that happens for a certain type of resource in Confidence.
You can route feeds to Slack, email, or webhook.
An activity feed notifies you about activities in one or several of the following types of resources:
* Feature flags
* A/B tests
* Rollouts
Read more about connecting to Slack and notifications settings in the [notifications documentation](./introduction).
## Related Resources
Overview of notification types
Configure surface notifications
Set up personal Slack notifications
Learn about A/B tests
# Notifications
Source: https://confidence-auth-testing.mintlify.io/docs/notifications/introduction
Get notified outside of Confidence when something happens.
Confidence notifies about activities in one of the following three categories:
* **High:** Activities that someone needs, or might need, to take an immediate action on. For example experiment health checks failing.
* **Medium:** Activities that affect the end user experience, and activities that are important for the resource owner to be aware of. For example, experiment start/stop, rollout reach increased, new flag variants added, etc.
* **Low:** All other activities.
## Notification Types
Confidence sends notifications in three ways:
* **Surface notifications**: For all activities in a surface. Configure these in the [surface settings](../surfaces/surface-settings#notifications).
* **Personal notifications**: Direct notifications sent to individual users based on their involvement with resources.
* **Activity feeds**: For specific types of resources like flags, A/B tests, or rollouts. See [activity feeds](./activity-feeds).
## Slack Integration
To enable Slack notifications, your organization must integrate Slack with Confidence. Once integrated, individual users can connect their personal Slack accounts to receive personal notifications.
## Integrate Your Slack Account with Confidence
Integrating Slack enables Slack notifications for surfaces, activity feeds, and personal notifications.
## Connect your Personal Slack Account
Connecting your personal Slack account allows you to receive personal notifications in Slack.
It's only possible to connect your personal Slack if your organization has integrated Slack with Confidence.
## Webhook Integration
Webhooks allow you to receive activity notifications at your own HTTPS endpoint, enabling integration with external systems, monitoring tools, or custom notification workflows.
To configure a webhook:
* Provide an HTTPS URL (IP addresses are not allowed)
* Set a secret for HMAC-SHA256 signature verification
* Implement signature verification in your endpoint to ensure authenticity
See the [webhook configuration guide](./webhook-configuration) for detailed setup instructions, security best practices, and code examples in multiple languages.
## Related Resources
Follow resource activities
Configure webhooks for activity notifications
Configure surface notifications
Set up personal Slack notifications
Collaborate with team comments
# Webhook Configuration
Source: https://confidence-auth-testing.mintlify.io/docs/notifications/webhook-configuration
Configure webhooks to receive activity notifications at your HTTPS endpoint
Webhooks allow you to receive real-time notifications about activities in Confidence by sending HTTP POST requests to your specified endpoint. You can configure webhooks as notification channels for [activity feeds](./activity-feeds), enabling integration with external systems, monitoring tools, or custom notification workflows.
## Configure a Webhook
### Create an Activity Feed with Webhook
Choose from Flags, A/B Tests, or Rollouts
For example: `https://example.com/webhooks/confidence`
See the [Security](#security) section for details on signature verification
### Edit a Feed
When editing an existing webhook, you must re-enter the secret. The secret is write-only and never returned from the server for security reasons.
## Webhook Payload
Confidence sends activity notifications as JSON in the `ListActivitiesResponse` format. Each webhook request contains an array of activities.
### Example Payload
```json theme={null}
{
"activities": [
{
"name": "activityTypes/experiment-started/activities/abc123",
"primaryResource": "workflows/abtest/instances/my-experiment",
"relatedResources": ["surfaces/global"],
"priority": "MEDIUM",
"actor": "identities/user@example.com",
"title": "Experiment started",
"body": "The A/B test 'Homepage Button Test' has been started.",
"activityTime": "2024-02-26T10:30:00Z",
"createTime": "2024-02-26T10:30:00Z",
"updateTime": "2024-02-26T10:30:00Z",
"creator": "identities/user@example.com",
"updater": "identities/user@example.com"
}
]
}
```
### Activity Fields
* `name`: Unique identifier for the activity
* `primaryResource`: The main resource this activity relates to
* `relatedResources`: Other resources involved in the activity
* `priority`: Activity priority level (LOW, MEDIUM, HIGH)
* `actor`: The identity that performed the activity
* `title`: Human-readable activity title
* `body`: Detailed description in Markdown format
* `activityTime`: When the activity occurred
* `createTime`: When Confidence created the activity
* `updateTime`: When the activity was last updated
* `creator`: Identity that created the activity record
* `updater`: Identity that last updated the activity record
The `title` and `body` fields are designed for human-readable display and may change without notice. Do not parse or depend on these fields for automation logic. Instead, use the `primaryResource` field to fetch the authoritative resource data via the corresponding API endpoint.
## Security
Webhooks use HMAC-SHA256 signatures to ensure authenticity and integrity of messages. Every webhook request includes three custom headers for verification.
### Request Headers
| Header | Description |
| --------------------------------- | --------------------------------------------------------- |
| `Confidence-Webhook-Signature` | HMAC-SHA256 signature of the payload |
| `Confidence-Webhook-Id-Signature` | Webhook configuration ID |
| `Confidence-Webhook-Timestamp` | Unix timestamp (seconds) when Confidence sent the request |
### Signature Generation
Confidence computes the signature as:
```text theme={null}
HMAC-SHA256(secret, "{timestamp}.{payload}")
```
Where:
* `secret`: The webhook secret you provided during configuration
* `timestamp`: The value from the `Confidence-Webhook-Timestamp` header
* `payload`: The raw JSON request body
### Verify Signatures
To verify a webhook request is authentic:
* `Confidence-Webhook-Signature`
* `Confidence-Webhook-Timestamp`
* Check that the timestamp is recent (within ±5 minutes of current time)
* Reject requests with timestamps too far in the past or future
* Concatenate timestamp and payload: `"{timestamp}.{payload}"`
* Compute HMAC-SHA256 using your webhook secret
* Convert result to hexadecimal string
* Use constant-time comparison to prevent timing attacks
* If signatures match, the request is authentic
### Example Verification
```javascript Node.js theme={null}
const crypto = require('crypto');
function verifyWebhook(request, secret) {
const signature = request.headers['confidence-webhook-signature'];
const timestamp = request.headers['confidence-webhook-timestamp'];
const payload = request.body; // Raw JSON string
// Validate timestamp (within 5 minutes)
const now = Math.floor(Date.now() / 1000);
const requestTime = parseInt(timestamp);
if (Math.abs(now - requestTime) > 300) {
throw new Error('Timestamp too old or too far in the future');
}
// Compute expected signature
const signedPayload = `${timestamp}.${payload}`;
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(signedPayload)
.digest('hex');
// Compare signatures (constant-time)
if (!crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
)) {
throw new Error('Invalid signature');
}
return true;
}
```
```python Python theme={null}
import hmac
import hashlib
import time
def verify_webhook(headers, body, secret):
signature = headers.get('confidence-webhook-signature')
timestamp = headers.get('confidence-webhook-timestamp')
# Validate timestamp (within 5 minutes)
now = int(time.time())
request_time = int(timestamp)
if abs(now - request_time) > 300:
raise ValueError('Timestamp too old or too far in the future')
# Compute expected signature
signed_payload = f"{timestamp}.{body}"
expected_signature = hmac.new(
secret.encode('utf-8'),
signed_payload.encode('utf-8'),
hashlib.sha256
).hexdigest()
# Compare signatures (constant-time)
if not hmac.compare_digest(signature, expected_signature):
raise ValueError('Invalid signature')
return True
```
```java Java theme={null}
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.time.Instant;
public class WebhookVerifier {
private static final String HMAC_SHA256 = "HmacSHA256";
public static boolean verifyWebhook(String signature, String timestamp,
String payload, String secret)
throws Exception {
// Validate timestamp (within 5 minutes)
long now = Instant.now().getEpochSecond();
long requestTime = Long.parseLong(timestamp);
if (Math.abs(now - requestTime) > 300) {
throw new IllegalArgumentException(
"Timestamp too old or too far in the future");
}
// Compute expected signature
String signedPayload = timestamp + "." + payload;
SecretKeySpec secretKey = new SecretKeySpec(
secret.getBytes(StandardCharsets.UTF_8), HMAC_SHA256);
Mac mac = Mac.getInstance(HMAC_SHA256);
mac.init(secretKey);
byte[] hash = mac.doFinal(signedPayload.getBytes(StandardCharsets.UTF_8));
String expectedSignature = bytesToHex(hash);
// Compare signatures (constant-time)
if (!MessageDigest.isEqual(
signature.getBytes(StandardCharsets.UTF_8),
expectedSignature.getBytes(StandardCharsets.UTF_8))) {
throw new IllegalArgumentException("Invalid signature");
}
return true;
}
private static String bytesToHex(byte[] bytes) {
StringBuilder result = new StringBuilder();
for (byte b : bytes) {
result.append(String.format("%02x", b));
}
return result.toString();
}
}
```
```go Go theme={null}
package main
import (
"crypto/hmac"
"crypto/sha256"
"crypto/subtle"
"encoding/hex"
"errors"
"fmt"
"math"
"strconv"
"time"
)
func verifyWebhook(signature, timestamp, payload, secret string) error {
// Validate timestamp (within 5 minutes)
now := time.Now().Unix()
requestTime, err := strconv.ParseInt(timestamp, 10, 64)
if err != nil {
return fmt.Errorf("invalid timestamp: %w", err)
}
if math.Abs(float64(now-requestTime)) > 300 {
return errors.New("timestamp too old or too far in the future")
}
// Compute expected signature
signedPayload := timestamp + "." + payload
mac := hmac.New(sha256.New, []byte(secret))
mac.Write([]byte(signedPayload))
expectedSignature := hex.EncodeToString(mac.Sum(nil))
// Compare signatures (constant-time)
if subtle.ConstantTimeCompare(
[]byte(signature),
[]byte(expectedSignature)) != 1 {
return errors.New("invalid signature")
}
return nil
}
```
```rust Rust theme={null}
use hmac::{Hmac, Mac};
use sha2::Sha256;
use std::time::{SystemTime, UNIX_EPOCH};
type HmacSha256 = Hmac;
fn verify_webhook(
signature: &str,
timestamp: &str,
payload: &str,
secret: &str,
) -> Result> {
// Validate timestamp (within 5 minutes)
let now = SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs();
let request_time: u64 = timestamp.parse()?;
if (now as i64 - request_time as i64).abs() > 300 {
return Err("Timestamp too old or too far in the future".into());
}
// Compute expected signature
let signed_payload = format!("{}.{}", timestamp, payload);
let mut mac = HmacSha256::new_from_slice(secret.as_bytes())?;
mac.update(signed_payload.as_bytes());
let result = mac.finalize();
let expected_signature = hex::encode(result.into_bytes());
// Compare signatures (constant-time)
if expected_signature != signature {
return Err("Invalid signature".into());
}
Ok(true)
}
```
## Best Practices
* **Generate strong secrets**: Use cryptographically random strings (at least 32 characters)
* **Rotate secrets regularly**: Update webhook secrets periodically
* **Store securely**: Never commit secrets to version control or expose in logs
* **Use environment variables**: Store secrets in environment variables or secret management systems
* **Return quickly**: Respond with HTTP 200 within a few seconds to avoid timeouts
* **Process asynchronously**: Queue webhook payloads for background processing
* **Implement retries**: Confidence does not automatically retry failed webhooks
* **Log webhook ID**: Store the `Confidence-Webhook-Id-Signature` for debugging
* **Validate signatures first**: Always verify signatures before processing payload
* **Handle malformed JSON**: Gracefully handle invalid JSON payloads
* **Monitor failures**: Track webhook failures and set up alerts
* **Return appropriate status codes**:
* `200 OK`: Successfully received and validated
* `400 Bad Request`: Invalid signature or malformed payload
* `500 Internal Server Error`: Processing error (avoid if possible)
* **Validate timestamp**: Prevent replay attacks by checking timestamp freshness
* **Use constant-time comparison**: Prevent timing attacks when comparing signatures
* **Rate limiting**: Implement rate limiting on your webhook endpoint
## Troubleshoot Common Issues
* Verify the webhook is **enabled** (toggle switched on)
* Check that the **priority filter** allows the activity level
* Ensure your endpoint is **publicly accessible** via HTTPS
* Verify your endpoint **returns HTTP 200** status
* Confirm you're using the **correct secret**
* Verify you're computing the signature over `"{timestamp}.{payload}"`
* Check that you're using the **raw request body** (not parsed JSON)
* Ensure timestamp format is **Unix seconds** (not milliseconds)
* Check the **activity priority** matches your webhook configuration
* Verify the **resource type** is followed by the activity feed
* Ensure the activity feed is **subscribed** to the relevant resources
## Related Resources
Configure activity feeds for different resources
Overview of notification types and priorities
Configure surface-level notifications
Explore the Confidence API
# Onboarding Guide
Source: https://confidence-auth-testing.mintlify.io/docs/onboarding-guide
This page outlines the most efficient way to onboard your team or organization to Confidence. Use it to configure your organization's initial setup and integration with Confidence.
## Preconfigured Resources
To help with onboarding, Confidence comes with the following already created for you:
* `User` and `Visitor` entities to cover the most common types of experiments
* A `tutorial-feature` flag to use in quickstarts
* A client to use when resolving flags
You can add more and change as needed later.
## Review Admin Roles
Only the user that created the Confidence account has the `Admin` role by
default. When configuring your organization's initial setup, you should
assign more than one user with the `Admin` role.
Give more users the `Admin` role:
## Configure Your Integration
The first team members onboarding to Confidence are often responsible for
configuring the initial setup. This involves some basic configurations and a
first integration end-to-end. Through these initial steps, other team members
can onboard to Confidence more efficiently.
The goals for the first team members onboarding to Confidence are:
* Complete the initial setup so that your organization has the basics in place
* Understand the experimentation process end-to-end by integrating Confidence in
a mock app or in production
* Prepare your organization for more team members to efficiently onboard
As one of the first ones to onboard, you should:
1. Review entities available for experimentation and optionally add new
2. Decide if you want to start testing Confidence in a mock app or in production
3. Create a feature flag to allow confidence to remotely control a part of your product
4. Create some basic metrics for your first experiments
5. Configure a warehouse for use by Confidence
### Review Entities
Entities represent the objects you want to run experiments and create metrics
on. By default, Confidence includes `User` and `Visitor`. If you want to
experiment on and create metrics for other entities, like `Session` or
`Organization`, create them now. Read more about entities in the
[entities reference](/docs/metrics/entities).
### Test In a Mock App or In Production
To get familiar with the experimentation process end-to-end, you can either
start integrating Confidence in a mock app or directly in production. Working
with a mock app can help you get familiar without affecting production. On the
other hand, integrating directly in production can allow you to see real
results and impact.
Most users start with the **production approach**.
Follow the quick starts below. As you run through the quick starts, consider:
* Testing as if you are a new user to the platform with the goal of understanding
how to use Confidence
* Sharing your initial feedback and ideas for improvements with the rest of the
team, so they can onboard more efficiently
* Gathering questions that arise during the onboarding process and then
create a FAQ for your team
Set up your first feature flag
Create your first metric
Gradually release a feature
Run your first experiment
Import historical data
### Configure a Warehouse
To use Confidence, you need to configure a warehouse. For information about how
to set up your warehouse, refer to the guides below.
Configure BigQuery for Confidence
Configure Snowflake for Confidence
Configure Redshift for Confidence
Configure Databricks for Confidence
## Enable More Team Members to Onboard
When you finish the onboarding guide and have the basics in place, you can
invite more team members to Confidence. Read about [inviting users](/docs/iam/users) and assigning users to different [roles](/docs/iam/roles)
using [policies](/docs/iam/policies). Next, you can consider sharing the onboarding checklist
below with your team to help them onboard efficiently.
## Onboarding Checklist for Team Members
After the first team members have completed the initial setup and integration,
other team members can use this checklist to onboard to Confidence.
Go through the [Configure a Flag quickstart](/docs/quickstarts/configure-flag) to understand how to create and use feature flags.
Go through the [Configure a Metric quickstart](/docs/quickstarts/configure-metric) to understand how to create metrics.
Go through the [Launch a Rollout quickstart](/docs/quickstarts/launch-rollout) to understand how to launch a rollout.
Go through the [Launch an A/B Test quickstart](/docs/quickstarts/launch-abtest) to understand how to launch an A/B test.
Review any internal docs your organization has created about using Confidence.
Reach out to your team's Confidence champions with any questions.
# Analysis Quickstart
Source: https://confidence-auth-testing.mintlify.io/docs/quickstarts/analyze-past-experiment
Analyze a past experiment administered through another feature flagging provider in Confidence with the help of this quickstart.
Don't analyze live experiments with an analysis in
Confidence. This experiment type specializes in analyzing past experiments and is
not suitable for tracking a live experiment.
## Before You Begin
To run an analysis, you need:
* an [entity](/docs/metrics/entities)
* a [fact table](/docs/metrics/fact-tables)
* a [metric](../metrics)
Use the [metrics quickstart](/docs/quickstarts/configure-metric) to create an entity, fact table, and a metric.
## Step 1: Create the Assignment Table
An assignment table is a table that has data about assignments of entities to variants in experiments.
For your analysis to be able to read the data, you need to create an assignment table in Confidence.
Follow the steps on the [assignment table page](/docs/metrics/assignment-tables) to
create your table.
### Assignment Table from Optimizely Decision Events
You can analyze past or current experiments in Optimizely with Confidence.
To do so, you need to [export the decision events](https://docs.developers.optimizely.com/experimentation-data/docs/enriched-events-export)
from Optimizely to a table in your data warehouse.
A decision event is an event that Optimizely records when a visitor is exposed to an experiment.
Decision events in Optimizely correspond to assignments in Confidence.
The information Confidence requires is available in the columns:
* `experiment_id`: column with identifiers of the experiments
* `variation_id`: column with identifiers of the variants
* `visitor_id`: column with identifiers of the entities in the experiments, like users and visitors
* `timestamp`: column with timestamps of the events
To set up the assignment table in Confidence, follow these steps:
1. [Export](https://docs.developers.optimizely.com/experimentation-data/docs/enriched-events-export)
the [decision events](https://docs.developers.optimizely.com/experimentation-data/docs/enriched-events-data-specification#decisions-1)
from Optimizely to a table in your data warehouse. If you want to analyze a running experiment,
you need to schedule the export to happen at a regular cadence.
2. If you don't have one already, [create an entity](/docs/metrics/entities) in Confidence that identifies the
entity that's recorded in the `visitor_id` column of the decision events table.
3. [Create an assignment table](/docs/metrics/assignment-tables) in Confidence that points to the decision events table.
* Set the exposure key column to `experiment_id`.
* Set the variant key column to `variation_id`.
* Set the entity to the entity you created in step 2.
* Set the entity column to `visitor_id`.
* Set the timestamp column to `timestamp`.
The columns `experiment_id` and `variation_id` must be strings to be selectable
as exposure key and variant. The type of the `visitor_id` must match the primary key type of the
entity you created in step 2, such as a string. The `timestamp` column must be a timestamp.
## Step 2: Create an Analysis
Open [Confidence](https://app.confidence.spotify.com) and
select **Analyses** on the left sidebar.
The overview page shows all draft, live, and ended analyses that you have permission to view.
Click **+ Create** in the upper right corner to create a new analysis.
## Step 3: Name, Assignment table, Entity and Owner
Specify which entity the experiment used for randomizing the treatment assignment, for example `User`.
Confidence uses the entity to map the metrics to the units in the analysis.
You first need to give your analysis a name and assign an owner. Use a descriptive name that others understand.
For this exercise, use:
* **Name**: `analysis-`
* **Assignment table**: Select which assignment table to use for this analysis. This option only appears if you have more than one assignment table in Confidence.
* **Entity**: Select the entity that represents the unit you experiment on.
* **Owner**: Select yourself
Click **Create**. You're now on the analysis design page.
## Step 4: Dates and Exposure Key
Select the date range for the experiment to avoid scanning unnecessary data.
The analysis start date is the first day of the experiment that you want to analyze.
If the experiment started on 2023-01-01, input 2023-01-01 as the start date to begin
analyzing it on the day it started.
Enter the identifier of the experiment in the assignment data as the exposure key. For example,
if your experiment identifier is `experiment-123`, enter `experiment-123` as the exposure key. If
you're analyzing an experiment based on decision events exported from Optimizely, your exposure key
is the identifier for your experiment that's available in the `experiment_id` column in your decision events table.
## Step 5: Treatments
To configure the treatment groups for your analysis, you need to enter the variant keys for the treatment groups.
Confidence pre-populates the treatment variant list with the unique values found
in the variant key column of the assignment table for the selected exposure key.
The variant key is the identifier for each group in the experiment that, together with the exposure key, uniquely
identifies the relevant group in the assignment table. For example, if the control group is `default-style`, enter `default-style` as the
variant key.
To set up your treatments:
Enter the identifier of the control group in the **Variant key** field. Optionally enter a display name in the **Name** field and upload an image of the variant. Click **Save and add another**.
Adjust the weights of the treatments to match the weights you used in the underlying randomization.
While the provided treatment split doesn't affect traffic, it
is important that you specify the intended treatment split from the original
experiment. Confidence uses the given split to validate that the randomization is
correct, a central validity check in an experiment. If the observed proportions
of the variants don't match the pre-specified split, the analysis triggers the check for
balanced traffic.
If you're analyzing an experiment based on decision events exported from Optimizely, your variant key
is the identifier for the treatment group that's available in the `variation_id` column in your decision events table.
## Step 6: Metrics
To measure the outcome of the experiment, you need to add metrics to the analysis. In Confidence,
metrics are either **Success metrics** or **Guardrail metrics**. Add your metric as a success metric
if you hope to see an improvement in the metric. For example, with your change you hope to see an increase
in the number of purchases per user. Add your metric as a guardrail metric if you don't expect to see a change,
but you want to make sure the change doesn't have a negative impact. For example, with your change you don't want
to see an increase in the number of returned items per user.
To add your metrics to the analysis:
Select the metric you want to add to the analysis. For success metrics, select the **Preferred direction** and enter an **MDE** (minimum detectable effect). The MDE represents the size of the change you're interested in finding. For guardrail metrics, select the **Preferred direction** and enter the **NIM** (non-inferiority margin). The NIM represents your tolerance for a negative change. Click **Add metrics**.
Read more about [minimum detectable effects and non-inferiority margins](/docs/experiments/design/effect-sizes).
Your analysis can have required metrics added from the surface that the analysis belongs to. Read more about [required metrics](../surfaces#required-metrics).
## Step 7: Calculate
Review your setup and click **Calculate** to run the analysis. You are now on the Result page.
You can add exploratory analyses on the Result page. Click **Explore** on the Metrics result section.
You can go back and change settings on the Design tab. If you change
certain settings, like exposure key or treatments, you need to recalculate the
analysis to get back to the Result tab.
## Related Resources
Deep dive into analysis configuration options
Understand how to configure assignment data sources
Run new experiments directly in Confidence
Move your flags and code from PostHog, Eppo, Statsig, or Optimizely
# Configure a Flag
Source: https://confidence-auth-testing.mintlify.io/docs/quickstarts/configure-flag
This tutorial shows you how to configure a flag that controls the design of a website header.
The tutorial consists of the following steps:
1. [Create a client](#create-a-client) for the website so that it can resolve
flags.
2. [Create a flag](#create-a-flag) for the header design, and define its schema.
3. [Use the flag](#use-the-flag) in the website to control the header.
4. [Create variants](#create-variants) of the header.
5. [Force a variant for a user](#force-a-variant-for-a-user) to test a specific
variant.
6. [Test resolving the flag](#resolve-tester) with an evaluation context.
This page targets the following audience:
* Anyone who wants to set up a flag and understand how to use it.
Before you begin:
* You need to have a [Confidence](https://app.confidence.spotify.com) account.
This video gives a quick overview of how feature flags work in 2 minutes and 10 seconds.
Use this guide to set up a flag that you resolve, but without changing anything in your
code. This way, you can use the traffic the flag receives to set up and run A/B tests or rollouts
using the [A/B test quickstart](/docs/quickstarts/launch-abtest) or the [rollout quickstart](/docs/quickstarts/launch-rollout). Since the resolved flag value
isn't used in your code, nothing changes for your users.
You can also configure flags using natural language with [Confidence MCP servers](/docs/quickstarts/use-mcp). MCP lets you create flags, add variants, set up targeting rules, and test resolution directly from Claude Code, Codex, Cursor, or VS Code.
## Choose a Resolution Method That Fits Your Application
You can resolve a Confidence feature flag in two ways: by using
Confidence's managed resolver or hosting your own local resolver. Read
more about the resolution options in [the documentation](/docs/flags/data-transfer).
## Create a Client
You must associate all feature flags with at least one [client](/docs/sdks/introduction). A
client can, for example, be a backend service or a website. Clients use
flags to deliver different user experiences. To resolve flags, a client must
authenticate with Confidence using client credentials.
Confidence includes a default client that has the same name as your Confidence account. You can use this client, or create a more specific one for web feature flags in this quickstart.
Follow these steps to create a client:
You can find it under the Admin section in the sidebar in Confidence.
Name the client `Web client` (unless it already exists).
You have now created a web client, and created the associated client
credentials. These credentials are later used to resolve the flag.
## Create a Flag
Flags let Confidence control the behavior of your application.
For example, use a flag to control which machine learning model serves a recommendation, the number
and size of tiles on a page, or a call to action message and its position on a sign-up page.
For this tutorial, create a flag that controls the color and size of a header.
To create the flag, follow these steps.
Select **Flags** on the left sidebar.
Or select the client with the same name as your account.
In the last step, you associated the flag with a specific client. This means
only **that client** can resolve the `header-redesign` flag. Limiting which flags are available to which
clients is valuable for several reasons. For example, it prevents exposing flags to clients that run in
uncontrolled environments such as mobile apps or server-side web apps. It also saves resources when resolving flags
in batch (for example, at app start) by restricting resolution to only the relevant flags.
Next, you define the **schema** of the flag. The value of a flag is not just a
single value, but rather a key-value map of properties. To avoid errors and make flags
easier to work with, Confidence requires you to define a schema for
the flag value. The schema describe the shape of the flag value, by defining
properties and their data types.
In this tutorial, your flag controls the design of a header on a website.
The design consists of color and size, so your flag needs to set two properties: `color` and `size`.
Click the edit schema button (pencil icon) next to the **Variants** heading.
You can configure or edit your schema by opening it on the right sidebar.
Now that you defined the schema, you can create variants that have specific
values for the `color` and `size` properties.
## Create Variants
The two variants you want to create for the header redesign are black with size
14, and blue with size 16. To do that, follow these steps.
Enter `black` as the value for `color`, and `14` as the value for size.
Enter `blue` as the value for `color`, and `16` as the size.
The two variants are now created, but they're not yet reaching any user.
To test your flag, use an individual targeting rule next.
## Use the Flag
When resolving a flag into a value, you specify a default value. This default value applies if a user doesn't match or
isn't assigned by a rule. The following sections show how to integrate the Confidence SDKs and set the default value for the `header-redesign` flag to `color` green and `size` 10.
### Install Dependencies
You first need to install the necessary dependencies.
```bash JavaScript (Web) theme={null}
yarn add @openfeature/web-sdk @spotify-confidence/openfeature-web-provider
```
```bash JavaScript (Server) theme={null}
yarn add @openfeature/server-sdk @spotify-confidence/openfeature-server-provider-local
```
```xml Java theme={null}
com.spotify.confidenceopenfeature-provider-locallatest
```
```bash Go theme={null}
go get github.com/spotify/confidence-resolver/openfeature-provider/go
go get github.com/open-feature/go-sdk
```
```toml Rust theme={null}
# Add to Cargo.toml
[dependencies]
spotify-confidence-openfeature-provider-local = ""
open-feature = ""
```
```bash Python theme={null}
pip install confidence-openfeature-provider
```
```bash iOS theme={null}
// When using Swift Package Manager, add the following to Package.swift
.package(url: "git@github.com:spotify/confidence-sdk-swift.git", from: "")
.product(name: "Confidence", package: "confidence-sdk-swift"),
.product(name: "ConfidenceOpenFeature", package: "confidence-sdk-swift"),
```
```bash Android theme={null}
implementation("com.spotify.confidence:openfeature-provider-android:")
```
### Initialize Confidence
With dependencies installed, you can now create a Confidence provider for your
platform and connect it to the OpenFeature SDK.
You only need to do this once, preferably on app startup.
```javascript JavaScript (Web) theme={null}
import { OpenFeature } from '@openfeature/web-sdk';
import { createConfidenceWebProvider } from '@spotify-confidence/openfeature-web-provider';
const provider = createConfidenceWebProvider({
clientSecret: 'your-client-secret',
timeout: 3000,
});
// Set the context that is relevant for your flag, like the user ID.
OpenFeature.setContext({
user_id: 'user-test-id',
plan: 'premium'
});
try {
await OpenFeature.setProviderAndWait(provider);
} catch (error) {
console.error('Failed to initialize Confidence provider:', error);
}
```
```typescript JavaScript (Server) theme={null}
import { OpenFeature } from '@openfeature/server-sdk';
import { createConfidenceServerProvider } from '@spotify-confidence/openfeature-server-provider-local';
const provider = createConfidenceServerProvider({
flagClientSecret: 'your-client-secret',
});
await OpenFeature.setProviderAndWait(provider);
```
```java Java theme={null}
import com.spotify.confidence.OpenFeatureLocalResolveProvider;
import dev.openfeature.sdk.OpenFeatureAPI;
import dev.openfeature.sdk.Client;
import dev.openfeature.sdk.MutableContext;
// Create and register the provider
OpenFeatureLocalResolveProvider provider =
new OpenFeatureLocalResolveProvider("your-client-secret");
OpenFeatureAPI.getInstance().setProviderAndWait(provider);
```
```go Go theme={null}
import (
"context"
"github.com/open-feature/go-sdk/openfeature"
"github.com/spotify/confidence-resolver/openfeature-provider/go/confidence"
)
ctx := context.Background()
provider, err := confidence.NewProvider(ctx, confidence.ProviderConfig{
ClientSecret: "your-client-secret",
})
if err != nil {
log.Fatalf("Failed to create provider: %v", err)
}
openfeature.SetProviderAndWait(provider)
```
```rust Rust theme={null}
use open_feature::{EvaluationContext, OpenFeature};
use spotify_confidence_openfeature_provider_local::{ConfidenceProvider, ProviderOptions};
#[tokio::main]
async fn main() -> Result<(), Box> {
let options = ProviderOptions::new("your-client-secret");
let provider = ConfidenceProvider::new(options)?;
OpenFeature::singleton_mut()
.await
.set_provider(provider)
.await;
Ok(())
}
```
```python Python theme={null}
from openfeature import api
from confidence import ConfidenceProvider
provider = ConfidenceProvider(client_secret="your-client-secret")
api.set_provider_and_wait(provider)
```
```swift iOS theme={null}
import Confidence
import ConfidenceProvider
import OpenFeature
let confidence = Confidence.Builder(clientSecret: "your-client-secret", loggerLevel: .NONE)
.build()
let provider = ConfidenceFeatureProvider(
confidence: confidence,
initializationStrategy: .fetchAndActivate
)
let ctx = ImmutableContext(
targetingKey: "user-test-id",
structure: ImmutableStructure(
attributes: [
"user_id": .string("user-test-id"),
"plan": .string("premium")
]
)
)
await OpenFeatureAPI.shared.setProviderAndWait(provider: provider, initialContext: ctx)
```
```kotlin Android theme={null}
import com.spotify.confidence.ConfidenceFactory
import com.spotify.confidence.ConfidenceFeatureProvider
import com.spotify.confidence.ConfidenceRegion
import com.spotify.confidence.InitialisationStrategy
import dev.openfeature.sdk.OpenFeatureAPI
import dev.openfeature.sdk.ImmutableContext
import dev.openfeature.sdk.Value
val confidence = ConfidenceFactory.create(
context = app.applicationContext,
clientSecret = "your-client-secret",
region = ConfidenceRegion.EUROPE
)
val provider = ConfidenceFeatureProvider.create(
confidence,
initialisationStrategy = InitialisationStrategy.FetchAndActivate
)
OpenFeatureAPI.setProviderAndWait(provider)
val evaluationContext = ImmutableContext(
targetingKey = "user-test-id",
attributes = mapOf(
"user_id" to Value.String("user-test-id"),
"plan" to Value.String("premium")
)
)
OpenFeatureAPI.setEvaluationContextAndWait(evaluationContext)
```
### Access the Flag
You can access the flag and its values using dot notation for nested properties.
```javascript JavaScript (Web) theme={null}
const client = OpenFeature.getClient();
// value of header-redesign is { size: , color: }
const size = client.getNumberValue('header-redesign.size', 10);
const color = client.getStringValue('header-redesign.color', 'blue');
```
```typescript JavaScript (Server) theme={null}
const client = OpenFeature.getClient();
const context = {
targetingKey: 'user-test-id',
user_id: 'user-test-id',
plan: 'premium'
};
// value of header-redesign is { size: , color: }
const size = await client.getNumberValue('header-redesign.size', 10, context);
const color = await client.getStringValue('header-redesign.color', 'blue', context);
```
```java Java theme={null}
Client client = OpenFeatureAPI.getInstance().getClient();
// Create evaluation context
MutableContext ctx = new MutableContext("user-test-id");
ctx.add("user_id", "user-test-id");
ctx.add("plan", "premium");
// value of header-redesign is { size: , color: }
Integer size = client.getIntegerValue("header-redesign.size", 10, ctx);
String color = client.getStringValue("header-redesign.color", "blue", ctx);
```
```go Go theme={null}
client := openfeature.NewClient("my-app")
evalCtx := openfeature.NewEvaluationContext("user-test-id", map[string]interface{}{
"user_id": "user-test-id",
"plan": "premium",
})
// value of header-redesign is { size: , color: }
size, _ := client.IntValue(ctx, "header-redesign.size", 10, evalCtx)
color, _ := client.StringValue(ctx, "header-redesign.color", "green", evalCtx)
```
```rust Rust theme={null}
let client = OpenFeature::singleton().await.create_client();
let context = EvaluationContext::default()
.with_targeting_key("user-test-id")
.with_custom_field("user_id", "user-test-id")
.with_custom_field("plan", "premium");
// value of header-redesign is { size: , color: }
let size = client
.get_int_value("header-redesign.size", Some(&context), None)
.await
.unwrap_or(10);
let color = client
.get_string_value("header-redesign.color", Some(&context), None)
.await
.unwrap_or_else(|_| "green".to_string());
```
```python Python theme={null}
from openfeature.evaluation_context import EvaluationContext
client = api.get_client()
context = EvaluationContext(
targeting_key="user-test-id",
attributes={
"user_id": "user-test-id",
"plan": "premium",
}
)
# value of header-redesign is { size: , color: }
size = client.get_integer_value("header-redesign.size", default_value=10, evaluation_context=context)
color = client.get_string_value("header-redesign.color", default_value="blue", evaluation_context=context)
```
```swift iOS theme={null}
let client = OpenFeatureAPI.shared.getClient()
// value of header-redesign is { size: , color: }
let size = client.getIntegerValue(key: "header-redesign.size", defaultValue: 10)
let color = client.getStringValue(key: "header-redesign.color", defaultValue: "green")
```
```kotlin Android theme={null}
val client = OpenFeatureAPI.getClient()
// value of header-redesign is { size: , color: }
val size = client.getIntegerValue("header-redesign.size", 10)
val color = client.getStringValue("header-redesign.color", "blue")
```
```bash Curl theme={null}
curl -H "Content-type: application/json" \
--data '{
"evaluation_context": {
"user_id": "user-test-id",
"plan": "premium"
},
"flags": ["flags/header-redesign"],
"client_secret": "your-client-secret"
}' \
"https://resolver.confidence.dev/v1/flags:resolve"
```
The code snippets above set two fields in the context: the `user_id` and the `plan` this user is on,
in this case `premium`.
You can use the `plan` field in the context to create targeted rules. For example, with this
information in the evaluation context, an A/B test can include only users on the premium plan as
its target audience.
This video gives a quick overview how targeting and evaluation contexts work in 2 minutes and 10 seconds.
If you were to run the code above you would only get the default values for the
flag. Nothing tells the client that it should return any other
value. To do so, you need to create flag variants and a rule that returns a variant.
## Force a Variant For a User
To try out a variant, you can force it for a user with a specific
user ID. You do this by creating an **individual targeting rule** on the flag. Individual targeting rules let you
target specific attribute values, such as a list of user IDs. For example, you can add
the identifiers of your team members so only your team can test the new experience in its early stages.
To create an individual targeting rule, follow these steps.
Click the toggle on the rule card to enable it.
If you re-run the code that fetches the flag value you now see that the flag
resolves to a value other than the default.
### Force all Variant For 50% of Employees
You can create a conditional targeting rule with a targeting audience that includes a subset
of the users in that audience. For example, you can create a conditional targeting rule that
targets 50% of the users in the `"plan": "employee"` audience. This way, provided
that all employees have their plan set to `employee`, you can test the new experience
on a subset of your colleagues.
On the flag page
In the audience section, click **Add attribute criterion** and write `plan` as field name with type `string` and click **Add**.
Click the toggle on the rule card to enable it.
Since you now have two rules, the rules evaluate in the order of the list on the
flag page. If a user is eligible for the first rule, it returns a variant for the user.
If not, Confidence evaluates if the user is eligible for the second rule. In
this case, since only the user with ID `user-test-id` is eligible for the
individual targeting rule, Confidence evaluates eligibility for the second rule for everyone
else. The second rule, in turn, only targets users on the `employee` plan.
Change the priority of the rules by dragging the rule cards
into the desired order and clicking save.
## Resolve Tester
Use the Resolve tester to see if the rule you expect returns a variant.
In the Resolve tester, you give an evaluation context and see which rules match and which don't, together with the reasons why.
On the page of your flag, select **Test rules** at the top of the list of rules.
Click **Add evaluation context** and give the context you want to test to
resolve. If you for example have created an individual targeting rule for the user with ID
`user-test-id` to the `new-style` variant, you can test that the rule matches by
adding the `user_id` field with the value `user-test-id` to the evaluation
context. Click **Resolve** to confirm that the individual targeting rule succeeds and
returns the `new-style` variant.
Most of the SDKs also output log messages that redirect you to the resolve tester specifically for the
flag evaluation that the SDK performed.
The link has a message with the prefix:
> See resolves for \ in Confidence:
To share a specific test run with someone, copy the URL and send it as a link.
The link directs them to the Resolve tester with all context data preserved.
## Alternative: Configure Flags with AI
You can perform all the steps in this tutorial using natural language prompts with an AI assistant. Confidence provides MCP (Model Context Protocol) servers that integrate with Claude Code, Codex, Cursor, and VS Code.
For setup instructions, see the [MCP quickstart](/docs/quickstarts/use-mcp).
Once configured, here are the prompts for each step in this tutorial (note: create your client in the UI first):
| Step | Example Prompt |
| -------------------------------- | ------------------------------------------------------------------------------------------------------ |
| Create a flag | `Create a flag called "header-redesign" with a string property "color" and an integer property "size"` |
| Add variants | `Add a variant "default-style" to header-redesign with color "black" and size 14` |
| Create individual targeting rule | `Create an individual targeting rule on header-redesign for user_id "user-test-id" to see "new-style"` |
| Test resolution | `Test resolving header-redesign for user_id "user-test-id"` |
## Related Resources
Set up a metric to measure the impact of your flags
Gradually release your feature to users
Run an experiment to compare variants
Deep dive into feature flag concepts and configuration
# Configure a Metric
Source: https://confidence-auth-testing.mintlify.io/docs/quickstarts/configure-metric
This tutorial shows you how to configure a metric in Confidence.
The tutorial consists of the following steps:
1. [Create an entity](#create-an-entity) that represents the users that are part of
your experiments.
2. [Create an assignment table](#create-an-assignment-table) that tells Confidence how your
entity is assigned to experiments.
3. [Create a fact table](#create-a-fact-table) that makes some measurable facts available for metrics.
4. [Create a metric](#create-a-metric) that aggregates a measurement from the
fact table.
If you already have an entity and an
assignment table you can skip directly to [create a fact table](#create-a-fact-table).
This page targets the following audiences:
* Data Engineers or Data Scientists who want to set up Confidence Metrics for
their organization.
Before you begin:
* You need to have a [Confidence](https://app.confidence.spotify.com) account.
* *(Step 2-3)* You need to have connected Confidence Metrics to your data warehouse.
* *(Step 2-3)* You need to have data for metrics in your data warehouse.
Use this guide to set up a metric from scratch. Use the metric in the [rollout quickstart](/docs/quickstarts/launch-rollout)
to measure the effect of the change you roll out.
## Create an Entity
An entity is a representation of the users that are part of your experiments.
Entities can be anything really, but typically they are users, customers, or
visitors.
In Confidence, the entity connects other concepts such as experiments,
variants, facts, dimensions and metrics to each other. It's via the entity that
Confidence can understand how it should traverse the data in your data
warehouse. For example, when you configure an experiment to use the
`User` entity, Confidence only considers metrics that use facts that
have a relationship to the `User` entity.
Confidence comes with two default entities: `User` and `Visitor`. If you want to
experiment on and create metrics for another entity, follow these steps.
On the left sidebar, select **Admin > Entities**.
Select a data type for how your data warehouse represents the entity. For the sake of this tutorial, select `String`.
🎉 That's it! You have now created an entity that you can tie other concepts to
in Confidence. In the next section, you use the entity when creating an assignment table
so that Confidence knows how your new entity is assigned to experiments and variants.
## Create an Assignment Table
If you have already set up an assignment table, you can skip this section.
Jump straight to [Create a metric](#create-a-metric).
The most fundamental part of Confidence Metrics is the experiment assignment
data. Without it you can't compute metrics for an A/B test or any type of
experiment.
Assignment data is a log of records that contain information about how
users were assigned to experiments and variants. Confidence needs the following
data:
* A **timestamp** that indicates when the assignment happened.
* An **entity identifier**, normally a user identifier, that uniquely
identifies the user that was assigned a variant.
* An **experiment identifier** that identifies the experiment that the user was
assigned to.
* A **variant identifier** that identifies the variant that the user was
assigned to.
Any table in your data warehouse can store the assignment data. For
Confidence to understand the data, you need to tell Confidence where to find it
and how to interpret it. You define this in your [Assignment Table](/docs/metrics/assignment-tables),
which is a query that projects the data
outlined above. You can write the query in any SQL dialect that your data
warehouse supports.
For this tutorial, your assignment data should exist in a table
called `assignment_log` in your data warehouse.
Data should be continuously appended to the table as users are assigned to experiments.
The table has the following schema:
```sql theme={null}
CREATE TABLE assignment_log (
timestamp TIMESTAMP,
user_id STRING,
experiment_id STRING,
variant_id STRING
)
```
Follow these steps to set up an assignment table:
On the left sidebar, select **Admin > Assignment tables**.
Give the table a name, such as `assignment`. If your data comes from a particular feature flagging system, you can name it after that, for example `launchdarkly`.
Enter the SQL query that projects the assignment data from your data warehouse.
```sql BigQuery theme={null}
SELECT
timestamp,
user_id,
experiment_id,
variant_id
FROM assignment_log
```
```sql Redshift theme={null}
SELECT
timestamp,
user_id,
experiment_id,
variant_id
FROM assignment_log
```
```sql Snowflake theme={null}
SELECT
timestamp,
user_id,
experiment_id,
variant_id
FROM assignment_log
```
This executes the query and shows you a preview.
To make the query cheap, the query runs with a `LIMIT` clause to limit the number of rows.
To the right of the result table, you see a form where you can specify the columns in the result that correspond to the assignment data. It's this mapping that tells Confidence how to interpret the data. Fill in the form as follows:
* `timestamp` as the timestamp column.
* For entity, select `User` and enter `user_id` as the entity column.
* `experiment_id` as the experiment key column.
* `variant_id` as the variant key column.
For **Data delivery cadence** select `Data is delivered continuously`. Leave **Commit delay** at its default value. You can read more about these settings on the [data delivery cadence](/docs/metrics/delivery-cadence) page.
You have now set up an assignment table and can move on to create
fact tables and metrics for your experiments.
## Create a Fact Table
Facts are measurable data that you want to use in your metrics. Facts can be
anything that you can measure, such as revenue, number of clicks, or number of
impressions.
Facts are stored in tables in your data warehouse. For Confidence to understand
the data, you need to tell Confidence where to find it and how to interpret it.
This tutorial assumes you have your fact data in a table called
`sales` in your data warehouse. Each time a sale occurs, the system adds a new
row to the data. `amount` is the sales amount and `product` is the product sold.
Assume that data is continuously appended to the table as sales happen.
The table has the following schema:
```sql theme={null}
CREATE TABLE sales (
timestamp TIMESTAMP,
user_id STRING,
amount FLOAT,
product STRING
)
```
Follow these steps to set up a fact table:
On the left sidebar, select **Admin > Fact tables**.
Give the table a name, such as `sales`.
Enter the SQL query that projects the fact data from your data warehouse.
```sql BigQuery theme={null}
SELECT
timestamp,
user_id,
amount,
product
FROM sales
```
```sql Redshift theme={null}
SELECT
timestamp,
user_id,
amount,
product
FROM sales
```
```sql Snowflake theme={null}
SELECT
timestamp,
user_id,
amount,
product
FROM sales
```
This executes the query and shows you a preview.
To the right of the result table, you see a form where you can specify the columns in the result that correspond to the fact data. It's this mapping that tells Confidence how to interpret the data. Fill in the form as follows:
* Select `timestamp` as the timestamp column.
* For entity, select `User` and select `user_id` as the entity column.
* Add `amount` to the list of measurements.
* Add `product` to the list of dimensions.
For **Data delivery cadence** select `Data is delivered continuously`. Leave **Commit delay** at its default value. You can read more about these settings on the [data delivery cadence](/docs/metrics/delivery-cadence) page.
## Create a Metric
The final step is to create a metric that aggregates the fact data. A metric
can be any aggregation of the fact data, such as sum, average, count, etc.
All metrics aggregate measurements over some time window. Watch the following video for a 4-minute overview of the different ways to handle time in metrics.
For this tutorial, create a metric that computes the average weekly sales
amount per user.
To create a metric follow these steps:
On the left sidebar, select **Metrics**.
Enter `Average weekly sales per user` as the name of the metric and assign yourself as the owner. Skip the description for now.
Select the `User` entity, the `sales` fact table, and the `amount` measurement as the **Consumption value**.
Click **Add Attribute criteria** and select `content_type` as the attribute. Set the filter to `content_type is podcast`.
In the **When to Include Users In Metrics Results** step, select `Cumulatively during a window` and `1 week` starting `at exposure`.
Review your metric setup, and then click **Create**.
That's it! You have now created your first metric. You can now create an A/B
test using the metric. Well done.
## Related Resources
Use your metric to measure experiment results
Use your metric to monitor a gradual release
Deep dive into metric types and configuration
Learn how to verify your metrics are working correctly
# A/B Test Quickstart
Source: https://confidence-auth-testing.mintlify.io/docs/quickstarts/launch-abtest
Set up and launch an A/B test in Confidence with the help of this guide.
## Evidence For Your Ideas
An A/B test is an experiment that lets you compare user reactions to different variants of your product. You get trustworthy
evidence for or against new variants so that you can learn from your users and iterate quickly.
This guide requires a feature flag and at least one metric. In what follows, the guide
uses the flag `tutorial-feature` with a `treatment` variant, and the metric `Page views per visitor` for the
`Visitor` entity. Use other flags, variants, metrics, and entities whenever the guide refers to
these if yours have different names. If necessary, follow the
[flags quickstart](/docs/quickstarts/configure-flag) to create the `tutorial-feature` flag, and
the [metrics quickstart](/docs/quickstarts/configure-metric) to create the `Page views per visitor` metric.
Some sections link to associated videos and documentation. Make use of these
resources as they contain important information that the guide doesn't cover.
This guide helps you set up a test that doesn't change anything in your code.
Such a test lets you learn how to set up and run an A/B test in Confidence, and
goes by the name of A/A test. It can either use the same variant for treatment
and control, or use different variants that don't change the user experience.
They help you practice setting up and running A/B tests before you start testing
real changes, providing an opportunity to validate and practice your full flow.
## Step 1: Create an A/B test
Time to get started and create your A/B test. Open [Confidence](https://app.confidence.spotify.com) and
select **A/B Tests** on the left sidebar.
The overview page shows all draft, live, and ended A/B tests that you have permission to view.
Click **+ Create** in the upper right corner to create a new A/B test.
## Step 2: Name, Assignment table, Entity, and Owner
You first need to give your A/B test a name, select an assignment table, choose an entity, and assign an owner. Use a descriptive name that others understand.
The assignment table determines which entity was exposed to which treatment. The A/B test uses the entity to randomly assign treatment and to aggregate metrics.
For this exercise, use:
* **Name**: `aa-ab-test-`
* **Assignment table**: Select an appropriate assignment table for your entity. This option only appears if your account has more than one assignment table.
* **Entity**: Visitor
* **Owner**: Select yourself
Click **Create**. You're now on the A/B test configuration page.
## Step 3: Treatments
The treatments you choose for your A/B test decide what experiences you want to compare.
The treatments you want to test must exist as variants on a feature flag. Feature flags define a configuration for an
aspect of your app, website, or backend service. This step is where you select precisely what you
want to vary in your A/B test.
The first treatment group you add is the control group. The experiment compares all other treatments to the first treatment.
You can change which group is the control by dragging a group to the left-most position.
Use the current default experience as the control variant.
In this guide, you don't want to test a real feature or change for your users. Instead,
you want to use the `tutorial-feature` flag. This flag changes nothing in your code, but only
serves the purpose of testing the A/B test functionality.
Click **+ Add control** to add your first treatment group, which becomes the control. Then click **+ Add treatment** to add a second treatment group. Select:
* **Flag**: `tutorial-feature`
* **Control**: `control`
* **Treatment**: `treatment`
## Step 4: Audience
The **Audience** section is where you define what your A/B test should target.
The **Randomization** field determines what field in the evaluation
context of the flag that your rollout randomizes treatment assignment on.
Confidence pre-populates this field with the entity you selected in the creation
step. The [context schema](../flags/context-schema) maps the entity `Visitor` to
the field 'visitor\_id' (shown in parentheses). This means that the rollout
randomizes on the value passed in the feature flag's evaluation context field 'visitor\_id'.
You decide the target audience for your A/B test. For example, add
`country is Sweden` as an inclusion criterion if you want to target users in Sweden. The information
available in the evaluation context of the flag determines what you can target on. Read more on the
[Audiences page](../flags/audience#inclusion-criteria).
In this case, target everyone by leaving the inclusion criteria empty.
The evaluation context is information that you pass in when making the resolve call to Confidence to
ask what variant to serve. This means what's available in the evaluation context depends on what you
pass in. Confidence lists recently used attributes available in the evaluation context when you enter the
name of the attribute.
This video gives a quick overview of how to increase the flexibility of who you
can include and exclude in your experiments by providing more information when
you resolve feature flags in 2 minutes and 10 seconds.
### Set the Allocation
The allocation sets what proportion of your target audience is eligible for your A/B test.
Adjust the slider to 5% to allocate 5% of the traffic to the A/B test. You can also set the allocation
by entering 5% in the input field.
## Step 5: Metrics
In this step you select the metrics you want to track for your A/B test. Two types of metrics are available for A/B tests.
**Success metrics** are metrics you intend to improve with your treatment.
**Guardrail metrics** are metrics you don't expect to improve, but you want to make sure they don't deteriorate.
Use both types to decide if the treatment variant is better than control for your product.
Confidence uses the entity you selected in the creation step to decide which metrics you can select. Any metric based on the Visitor entity is available.
You need to use a fact table that includes the Visitor entity if you want to create a new metric for it.
### Add Success Metric
In the success metric section, click **Add metric** and select:
* **Metric**: `Page views per visitor` metric.
* **Preferred direction**: `Increase`. If you use a metric other than `Page views per visitor`, set this
to the direction you want your metric to move in.
* **Minimum detectable effect (MDE)**: 5%.
### Add Guardrail Metric
In the guardrail metric section, click **Add metric** and select:
* **Metric**: `Page views per visitor` metric.
* **Preferred direction**: `Increase`. If you use a metric other than `Page views per visitor`, set this
to the direction you want your metric to move in.
* **Non-inferiority margin (NIM)**: 3%.
## Step 6: Hypothesis
In this section, you should clearly describe your hypothesis so that everyone understands the purpose of the
A/B test. See ideas for good hypotheses and learn more on the [hypothesis page](/docs/experiments/design/hypothesis).
In the **Hypothesis** section at the top of the configuration page, click the text area and enter your hypothesis.
**Hypothesis:** Changing the `tutorial-feature` from the variant `control` to `treatment`
for everyone should result in a change in their behavior, as measured by the success metric `Page views per visitor`.
The data supports the hypothesis if the success metric improves by 5% and the guardrail metric `Page views per visitor`
doesn't deteriorate by more than 3%.
## Step 7: Sample Size Calculation
In this step you run the sample size calculator to find out how many users your A/B test needs.
This number represents how many users you need to
have a reasonable chance of finding significant results if the treatment truly improves the user experience as much as you hope for.
Read more about [power analysis and the required sample size](/docs/experiments/statistical-settings).
Click **Calculate** in the **Required sample size section** on the right sidebar.
The sample size calculation queries historical data in your data warehouse to estimate the required sample size. This might take a few minutes.
If the required sample size is smaller than the number of users you can receive with the current allocation, your A/B test can reach power.
To learn more about how to work with design and the statistical settings of an A/B test, see the [Alpha and Power](/docs/experiments/design/power) page in the documentation.
You have successfully configured your A/B test, great job!
## Step 8: Launch
Now it's time to launch your A/B test!
This step launches your A/B test. **Make sure that you have selected the `tutorial-feature` flag to avoid
changing a real experience.**
Click **Launch** in the Actions section on the right sidebar.
If there are other live rollouts or A/B tests that use the `tutorial-feature` flag, you may not receive
5% of the traffic. Click **Flags** on the left sidebar and go to the `tutorial-feature` flag. In the
**Rules** section, you see what rules exist on the flag. If other rules have higher priority than
yours and use all traffic, you receive no traffic. If you want to receive traffic, you need
to adjust the priority of your rule and move it up in the list. Read more about
[order of rules](/docs/how-to-guides/reorder-rules).
**Congratulations, you have launched your first A/B test!**
## Step 9: Monitoring and Results
When you launch the A/B test, Confidence calculates exposure for the A/B test at
[repeated short intervals](/docs/metrics/exposure) to make sure that the
A/B test is working as expected, and that you are seeing some traffic. Hover the `Live` status on the
right sidebar to see the current status of the checks run for the A/B test.
You can end the A/B test by clicking **End** in the Actions section on the right sidebar. If the results show a winning variant, click **Roll out** to distribute it to all users. Keep it running for longer
and check back in tomorrow to see your first results.
Remember to end your A/B test within a couple of days to not waste resources.
## Related Resources
Deep dive into A/B test configuration and best practices
Configure significance levels and testing methodology
Write effective hypotheses for your experiments
# Rollout Quickstart
Source: https://confidence-auth-testing.mintlify.io/docs/quickstarts/launch-rollout
Set up and launch a rollout in Confidence with the help of this guide.
## Gradually Release Your Changes
A rollout is a type of experiment that lets you gradually introduce a change to users instead of releasing it
immediately to everyone. A rollout enables monitoring of the effect of the change on important metrics to make sure that
the change is working correctly when releasing it. If needed, you can roll back the change instantly without having
to do a new release.
In summary, a rollout gives you:
* Monitoring of a set of metrics that you choose to track, including business metrics, performance metrics, and user-behavior metrics
* Alerts if any of your metrics move in the wrong direction or if there's a sign that there's a problem with your setup
* Recommendations for how to proceed with the rollout
This guide requires a feature flag and at least one metric. In what follows, the guide
uses the flag `tutorial-feature` with a `treatment` variant, and the metric `Page views per visitor` for the
`Visitor` entity. Use other flags, variants, metrics and entities whenever the guide refers to
these if yours have different names. If necessary, follow the
[flags quickstart](/docs/quickstarts/configure-flag) to create the `tutorial-feature` flag, and
the [metrics quickstart](/docs/quickstarts/configure-metric) to create the `Page views per visitor` metric.
Some sections link to associated videos and documentation. Make use of these
resources as they contain important information that the guide doesn't cover.
## Step 1: Create a Rollout
Time to get started and create your rollout. Open [Confidence](https://app.confidence.spotify.com) and
select **Rollout** on the left sidebar.
The overview page shows all draft, live, and ended rollouts that you have permission to view.
Click **+ Create** in the upper right corner to create a new rollout.
## Step 2: Name, Entity, and Owner
You first need to give your rollout a name and assign an owner. Use a descriptive name that others understand.
You need to decide the entity that you want to roll out to. The rollout uses the entity to randomly assign treatment and to aggregate metrics.
For this exercise, use:
* **Name**: `aa-rollout-`
* **Entity**: Visitor
* **Owner**: Select yourself
Click **Create**. You're now on the rollout configuration page.
## Step 3: Feature
The feature you choose for your rollout determines what you want to roll out. The feature you
want to release must exist as a variant on a feature flag. Feature flags define a configuration for an
aspect of your app, website, or backend service. This step is where you select precisely what you
want to control in your rollout.
In this guide, you don't want to roll out a real feature or change for your users. Instead,
you want to use the `tutorial-feature` flag. This flag changes nothing in your code, but only
serves the purpose of testing the rollout functionality.
Click **Add variant** and select:
* **Feature flag**: `tutorial-feature`
* **Variant**: `treatment`
The variant selected here is the variant you want to roll out. Before you reach 100% with your
rollout, not everyone receives the new variant. The users that don't receive the new feature instead
get their variant set from other experiments, flag rules or the defaults defined by the clients
that use the flag.
## Step 4: Audience
The **Audience** section is where you define what your rollout should target.
The `treatment randomization unit` field determines what field in the evaluation
context of the flag that your rollout randomizes treatment assignment on.
Confidence pre-populates this field with the entity you selected in the creation
step. The [context schema](../flags/context-schema) maps the entity `Visitor` to
the field `visitor_id` (shown in parenthesis). This means that the rollout
randomizes on the value passed in the feature flag's evaluation context field `visitor_id`.
You decide the target audience for your rollout. For example, add
`country is Sweden` as an inclusion criterion if you want to target users in Sweden. The information
available in the evaluation context of the flag determines what you can target on. Read more on the
[Audiences page](../flags/audience#inclusion-criteria).
In this case, target everyone and leave the inclusion criteria empty.
The evaluation context is information that you pass in when making the resolve call to Confidence to
ask what variant to serve. This means what's available in the evaluation context depends on what you
pass in. Confidence lists recently used attributes available in the evaluation context when you enter the
name of the attribute.
This video gives a quick overview of how targeting and evaluation contexts work in 2 minutes and 10 seconds.
## Step 5: Metrics
In this step you select the metrics you want to track to make sure that your rollout isn't
negatively affecting the user experience or performance in any way. Confidence calls such metrics
**guardrail metrics**.
Confidence uses the entity you selected in the creation step to decide which metrics you can select. Any metric based on the Visitor entity is available.
You need to use a fact table that includes the Visitor entity if you want to create a new metric for it.
In the metric section, click **Add metric** and select:
* **Metric**: `Page views per visitor` metric.
* **Non-desired direction**: `Decrease`. If you use a metric other than `Page views per visitor`, set this
to the direction you don't want your metric to move in.
* **Non-inferiority margin**: 0.
You have successfully configured your rollout.
## Step 6: Launch
Now it's time to launch your rollout!
This step launches your rollout. **Make sure that you have selected the `tutorial-feature` flag to avoid
changing a real experience.**
Click **Launch** in the top right corner.
## Step 7: Scale Up
When you've launched your rollout, you can scale up the reach to what you want.
Use the slider to scale it up to 5%. Click **Save** for the new reach to apply.
You're now serving the `treatment` variant of the `tutorial-feature` flag to 5% of your users.
If there are other live rollouts or A/B tests that use the `tutorial-feature` flag, you may not receive
5% of the traffic. Click **Flags** on the left sidebar and go to the `tutorial-feature` flag. In the
**Rules** section, you see what rules exist on the flag. If other rules have higher priority than
yours and use all traffic, you receive no traffic. If you want to receive traffic, you need
to adjust the priority of your rule and move it up in the list. Read more about
[order of rules](/docs/how-to-guides/reorder-rules).
**Congratulations, you have launched your first rollout!**
## Step 8: Monitoring and Results
When you launch the rollout, Confidence calculates exposure for the rollout at
[repeated short intervals](/docs/metrics/exposure) to make sure that the
rollout is working as expected, and that you are seeing some traffic. Hover the `Live` status on the
right sidebar to see the current status of the checks run for the rollout.
You can end the rollout by clicking **End** in the upper right corner. Keep it running for longer
and check back in tomorrow to see your first results.
Remember to end your rollout within a couple of days to avoid wasting resources.
## Bonus Step: Automate the Ramp-up of Your Rollout
Confidence lets you automate the ramp-up of your rollout. The rollout schema
allows you to define the reach steps that the rollout should take and at what
time points the increases in reach should happen. Read more about it in the
[rollout documentation](/docs/experiments/workflows/rollouts).
## Related Resources
Deep dive into rollout configuration and automation
Compare variants with a controlled experiment
Set up metrics to monitor your rollout
# Use AI with Confidence
Source: https://confidence-auth-testing.mintlify.io/docs/quickstarts/use-mcp
Learn how to use Confidence MCP servers with AI assistants like Claude Code, Codex, Cursor, or VS Code to manage flags and analyze experiments.
Confidence provides MCP (Model Context Protocol) servers that enable AI assistants to manage your feature flags and analyze your experiments. Instead of navigating the UI or writing API calls, you can describe what you want in natural language and let your AI assistant handle it.
## What You Can Do with MCP
With Confidence MCP servers, you can:
* Get help with setting up your integration
* Search documentation
* Create and configure feature flags
* Add variants and define schemas
* Set up targeting rules
* Test flag resolution
* Analyze flag usage in your codebase
* Explore and list experiments (A/B tests and rollouts)
* Analyze experiment results
## Before You Begin
To complete this quickstart:
* You need a [Confidence](https://app.confidence.spotify.com) account
* You need an AI assistant that supports MCP: [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Codex](https://developers.openai.com/codex/), [Cursor](https://www.cursor.com/), or VS Code with GitHub Copilot
## Set Up the Documentation MCP Server
The Documentation MCP server gives your AI assistant access to Confidence documentation through semantic search. No authentication required.
```bash theme={null}
claude mcp add --transport http confidence-docs https://mcp.confidence.dev/mcp/docs
```
```bash theme={null}
codex mcp add confidence-docs --url https://mcp.confidence.dev/mcp/docs
```
Use `Command + Shift + P` (`Ctrl + Shift + P` on Windows) and search for `Open MCP settings`.
Select **New MCP Server** and add the following to the `mcpServers` JSON:
```json theme={null}
"confidence-docs": "https://mcp.confidence.dev/mcp/docs"
```
Create a `.vscode` directory in your workspace root if it doesn't exist.
Create a file named `mcp.json` in the `.vscode` directory.
Add the `confidence-docs` server to your configuration:
```json theme={null}
{
"servers": {
"confidence-docs": { "url": "https://mcp.confidence.dev/mcp/docs" }
}
}
```
Click `Start` next to the MCP server name in the JSON file to start it.
## Set Up the Flags MCP Server
The Flags MCP server enables your AI assistant to create and manage feature flags. This server requires authentication.
```bash theme={null}
claude mcp add --transport http confidence-flags https://mcp.confidence.dev/mcp/flags
```
After adding, open Claude Code, run `/mcp` and select `confidence-flags` to authenticate.
```bash theme={null}
codex mcp add confidence-flags --url https://mcp.confidence.dev/mcp/flags
codex mcp login confidence-flags
```
Use `Command + Shift + P` (`Ctrl + Shift + P` on Windows) and search for `Open MCP settings`.
Select **New MCP Server** and add the following to the `mcpServers` JSON:
```json theme={null}
"confidence-flags": "https://mcp.confidence.dev/mcp/flags"
```
Click `Connect` to authenticate.
Create a `.vscode` directory in your workspace root if it doesn't exist.
Create a file named `mcp.json` in the `.vscode` directory.
Add the `confidence-flags` server to your configuration:
```json theme={null}
{
"servers": {
"confidence-flags": { "url": "https://mcp.confidence.dev/mcp/flags" }
}
}
```
Click `Start` next to the MCP server name in the JSON file and authenticate when prompted.
## Set Up the Experiments MCP Server
The Experiments MCP server enables your AI assistant to explore and analyze A/B tests and rollouts. This server requires authentication.
```bash theme={null}
claude mcp add --transport http confidence-experiments https://mcp.confidence.dev/mcp/experiments
```
After adding, open Claude Code, run `/mcp` and select `confidence-experiments` to authenticate.
```bash theme={null}
codex mcp add confidence-experiments --url https://mcp.confidence.dev/mcp/experiments
codex mcp login confidence-experiments
```
Use `Command + Shift + P` (`Ctrl + Shift + P` on Windows) and search for `Open MCP settings`.
Select **New MCP Server** and add the following to the `mcpServers` JSON:
```json theme={null}
"confidence-experiments": "https://mcp.confidence.dev/mcp/experiments"
```
Click `Connect` to authenticate.
Create a `.vscode` directory in your workspace root if it doesn't exist.
Create a file named `mcp.json` in the `.vscode` directory.
Add the `confidence-experiments` server to your configuration:
```json theme={null}
{
"servers": {
"confidence-experiments": { "url": "https://mcp.confidence.dev/mcp/experiments" }
}
}
```
Click `Start` next to the MCP server name in the JSON file and authenticate when prompted.
## Create a Feature Flag with AI
This tutorial walks through creating the same header redesign flag as the [Configure a Flag](/docs/quickstarts/configure-flag) quickstart, but using natural language prompts instead of the UI.
### Verify Your Client Exists
The MCP server requires an existing client. List your available clients to find one to use:
> List available clients in Confidence
If you don't have a client yet, [create one in the Confidence UI](/docs/how-to-guides/manage-clients) before continuing.
### Create a Flag with Schema
Create a flag and define its schema in a single prompt:
> Create a flag called "header-redesign" with a string property "color" and an integer property "size"
Or you can ask it to base the schema of an existing struct in your codebase:
> Create a flag in Confidence with the same schema as the header-design struct
### Add Variants
Add the variants you want to test:
> Add a variant "default-style" to the header-redesign flag with color "black" and size 14
>
> Add a variant "new-style" to the header-redesign flag with color "blue" and size 16
### Create an Individual Targeting Rule
Set up an individual targeting rule to test a specific variant:
> Create an individual targeting rule on header-redesign for `user_id` `user-test-id` to see the "new-style" variant
### Test Flag Resolution
Verify your configuration works as expected:
> Test resolving header-redesign for a user with `user_id` `user-test-id`
The AI assistant tells you which variant resolves and why.
## Example Prompts for Common Tasks
Use these prompts for everyday flag management:
### Flag Operations
* "List all feature flags in Confidence"
* "Get details about the checkout-flow flag"
* "Create a boolean flag called enable-dark-mode"
* "Add a variant to the signup-experiment flag"
### Targeting and Rules
* "Create an individual targeting rule on checkout-flow for `user_id` `test-user` to see variant 'new-checkout'"
* "Test what variant resolves for checkout-flow with `user_id` `test-user`"
### Documentation and Integration
* "Show how to integrate Confidence with Python"
* "Search documentation for information about A/B testing"
* "Show the React SDK integration guide"
### Codebase Analysis
* "Analyze what flags to clean up in this codebase"
* "Find where the header-redesign flag appears in the code"
### Experiment Analysis
* "List all running experiments"
* "Search for experiments related to checkout"
* "Show details about the homepage-redesign A/B test"
* "What's the status of the metrics in the onboarding rollout?"
## Related Resources
Step-by-step UI tutorial for creating feature flags
Complete reference for all MCP tools and capabilities
Migrate from PostHog, Eppo, Statsig, or Optimizely with AI-powered migration kits
Deep dive into feature flag concepts and configuration
Integrate Confidence SDKs into your application
# Apply Event
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/apply-event
Confidence has the concept of applying a flag, this means marking the flag as used by a user.
Confidence automatically generate Apply Events, which you can use out-of-the-box to create [Assignment Tables](/docs/how-to-guides/create-assignment-table).
## On-Access Apply
Client-Side SDKs automatically emit `apply` events to the Confidence backend once applications read a flag or a flag's property. This allows Confidence to track who was exposed to what variant and when.
The `apply` event is only generated for flags that are successfully evaluated (that is, default values returned due to errors don't generate `apply` events).
The `apply` event reports which flag and variant the application read, but not which property the application has read from such variant's value.
To avoid generating redundant data, as long as the flags' data returned from the backend for a user remains unchanged, only the first reading of flag's property generates an `apply` event.
The Mobile Providers (iOS and Android) stores `apply` events on disk until it can send them correctly, thus ensuring the apply data reaches the backend even if generated when there is no network available (assuming the device re-connects to the network before the user deletes the application).
## On-Resolve Apply
Server SDKs and Edge Resolvers generate `apply` data at resolve time. The SDKs then batch this data and send it to the Confidence backend in the background, on a cadence.
## Related Resources
Overview of available SDKs
Configure assignment tracking
Set up assignment tables
Understand experiment exposure
# Android (Kotlin)
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/client/android
Confidence Android SDK for feature flag evaluation in Kotlin and Java applications.
The Confidence Android SDK provides feature flag evaluation for Android applications. Flags resolve once according to the evaluation context, and values read from a local cache for fast access.
## Features
* **Local caching**: Flag values cached locally for instant access
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Automatic apply events**: Tracks flag usage
* **Managed context**: Automatic visitor ID and app lifecycle context
* **Offline support**: Cached values available without network
## Installation
```kotlin theme={null}
dependencies {
implementation("com.spotify.confidence:openfeature-provider-android:")
}
```
```groovy theme={null}
dependencies {
implementation 'com.spotify.confidence:openfeature-provider-android:'
}
```
## Quickstart
```kotlin theme={null}
import com.spotify.confidence.ConfidenceFactory
import com.spotify.confidence.ConfidenceFeatureProvider
import com.spotify.confidence.ConfidenceRegion
import com.spotify.confidence.InitialisationStrategy
import dev.openfeature.sdk.OpenFeatureAPI
// Create the Confidence provider
val provider = ConfidenceFeatureProvider.create(
ConfidenceFactory.create(
context = applicationContext,
clientSecret = "your-client-secret",
region = ConfidenceRegion.EUROPE
),
initialisationStrategy = InitialisationStrategy.FetchAndActivate
)
// Register with OpenFeature
OpenFeatureAPI.setProviderAndWait(provider)
val client = OpenFeatureAPI.getClient()
// Evaluate a flag
val message = client.getStringValue("flag-name.message", "default message")
println("Flag value: $message")
```
## Resources
Source code, examples, and full documentation
OpenFeature documentation
Configure evaluation context
Understand flag assignment tracking
# .NET
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/client/dotnet
Confidence .NET SDK for client-side feature flag evaluation.
Confidence provides a fully supported .NET SDK for client-side feature flag evaluation.
For installation instructions, usage examples, and source code, visit the [Confidence .NET SDK repository](https://github.com/spotify/confidence-sdk-dotnet) on GitHub.
# Flutter
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/client/flutter
Confidence Flutter SDK for feature flag evaluation in cross-platform applications.
The Confidence Flutter SDK provides feature flag evaluation for Flutter applications on iOS, Android, web, and desktop platforms. Flags resolve once according to the evaluation context, and values read from a local cache for fast access.
## Features
* **Cross-platform**: Single SDK for iOS, Android, web, and desktop
* **Local caching**: Flag values cached locally for instant access
* **Automatic apply events**: Tracks flag usage
* **Managed context**: Automatic visitor ID and platform context
## Installation
Install via the Flutter CLI:
```bash theme={null}
flutter pub add confidence_flutter_sdk
```
## Quickstart
```dart theme={null}
import 'package:confidence_flutter_sdk/confidence_flutter_sdk.dart';
void main() async {
final confidence = ConfidenceFlutterSdk();
// Setup with API key
await confidence.setup("your-client-secret");
// Add context (optional)
await confidence.putContext("targeting_key", "user-123");
await confidence.putContext("country", "US");
// Fetch and activate flags
await confidence.fetchAndActivate();
// Get flag value
String flagValue = await confidence.getString("my-feature-flag.message", "default");
print('Flag value: $flagValue');
// Track custom event
confidence.track("event-name", {});
}
```
## Resources
Source code, examples, and full documentation
Flutter package documentation
Configure evaluation context
Understand flag assignment tracking
# iOS (Swift)
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/client/ios
Confidence iOS SDK for feature flag evaluation in Swift applications.
The Confidence iOS SDK provides feature flag evaluation for iOS, iPadOS, macOS, tvOS, and watchOS applications. Flags resolve once according to the evaluation context, and values read from a local cache for fast access.
## Features
* **Local caching**: Flag values cached locally for instant access
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Automatic apply events**: Tracks flag usage
* **Managed context**: Automatic visitor ID and app lifecycle context
* **Offline support**: Cached values available without network
## Installation
Add the package dependency to your `Package.swift`:
```swift theme={null}
dependencies: [
.package(url: "git@github.com:spotify/confidence-sdk-swift.git", from: "")
]
```
Then add the products to your target:
```swift theme={null}
.product(name: "Confidence", package: "confidence-sdk-swift"),
.product(name: "ConfidenceOpenFeature", package: "confidence-sdk-swift"),
```
## Quickstart
The SDK uses OpenFeature for flag evaluation.
```swift theme={null}
import Confidence
import ConfidenceProvider
import OpenFeature
// Initialize Confidence
let confidence = Confidence.Builder(clientSecret: "your-client-secret", loggerLevel: .NONE)
.build()
// Create OpenFeature provider with initial context
let provider = ConfidenceFeatureProvider(confidence: confidence)
let ctx = ImmutableContext(
targetingKey: "user-123",
structure: ImmutableStructure()
)
// Register with OpenFeature
await OpenFeatureAPI.shared.setProviderAndWait(provider: provider, initialContext: ctx)
// Get client and evaluate a flag
let client = OpenFeatureAPI.shared.getClient()
let value = client.getBooleanValue(key: "my-flag.my-boolean", defaultValue: false)
print("Flag value: \(value)")
```
## Resources
Source code, examples, and full documentation
OpenFeature documentation
Configure evaluation context
Understand flag assignment tracking
# JavaScript (Web)
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/client/javascript
Confidence JavaScript SDK for feature flag evaluation in web browsers.
**This SDK is being phased out.** For new integrations, we recommend using the [JavaScript Server SDK](/docs/sdks/server/javascript) with local resolve, which provides better performance and reliability. Only use this SDK when client-side context modifications are required that the local resolve SDK cannot support.
The Confidence JavaScript SDK provides feature flag evaluation for web browser applications. Flags resolve once according to the evaluation context, and values read from a local cache for fast access.
## Features
* **Local caching**: Flag values cached locally for instant access
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Automatic apply events**: Tracks flag usage
* **Managed context**: Optional visitor ID and page context
* **Event tracking**: Built-in analytics event tracking
## Installation
```bash npm theme={null}
npm install @spotify-confidence/sdk
```
```bash yarn theme={null}
yarn add @spotify-confidence/sdk
```
```bash pnpm theme={null}
pnpm add @spotify-confidence/sdk
```
## Quickstart
```javascript theme={null}
import { Confidence } from '@spotify-confidence/sdk';
// Initialize Confidence
const confidence = Confidence.create({
clientSecret: 'your-client-secret',
});
// Set the evaluation context
confidence.setContext({
targeting_key: 'user-123',
country: 'US',
});
// Subscribe to flag updates and evaluate
await confidence.activate();
// Evaluate a flag
const value = confidence.getFlag('my-feature-flag', false);
console.log('Flag value:', value);
```
### With OpenFeature
```javascript theme={null}
import { OpenFeature } from '@openfeature/web-sdk';
import { createConfidenceWebProvider } from '@spotify-confidence/openfeature-web-provider';
// Create and register the provider
const provider = createConfidenceWebProvider({
clientSecret: 'your-client-secret',
});
await OpenFeature.setProviderAndWait(provider);
const client = OpenFeature.getClient();
// Evaluate a flag
const value = client.getBooleanValue('my-feature-flag', false);
```
## React Integration
**The standalone React SDK is being phased out.** For new React integrations, we recommend using the [JavaScript Server SDK](/docs/sdks/server/javascript) with local resolve and its React hooks, which provides better performance and reliability.
The Confidence React SDK provides React hooks and components for feature flag evaluation. Built on top of the JavaScript SDK, it offers a React-native developer experience.
### Installation
```bash npm theme={null}
npm install @spotify-confidence/sdk @spotify-confidence/react
```
```bash yarn theme={null}
yarn add @spotify-confidence/sdk @spotify-confidence/react
```
```bash pnpm theme={null}
pnpm add @spotify-confidence/sdk @spotify-confidence/react
```
### Usage with Confidence SDK
```tsx theme={null}
import { Confidence } from '@spotify-confidence/sdk';
import { ConfidenceProvider, useFlag } from '@spotify-confidence/react';
// Initialize Confidence
const confidence = Confidence.create({
clientSecret: 'your-client-secret',
});
// Set initial context
confidence.setContext({
targeting_key: 'user-123',
country: 'US',
});
// Wrap your app with the provider
function App() {
return (
);
}
// Use flags in components
function MyComponent() {
const showNewFeature = useFlag('my-feature-flag', false);
return (
{showNewFeature ? : }
);
}
```
### Usage with OpenFeature
```tsx theme={null}
import { OpenFeatureProvider, useFlag } from '@openfeature/react-sdk';
import { createConfidenceWebProvider } from '@spotify-confidence/openfeature-web-provider';
const provider = createConfidenceWebProvider({
clientSecret: 'your-client-secret',
});
function App() {
return (
);
}
function MyComponent() {
const { value: showNewFeature } = useFlag('my-feature-flag', false);
return (
{showNewFeature ? : }
);
}
```
## Resources
Source code, examples, and full documentation
OpenFeature documentation
OpenFeature React documentation
Configure evaluation context
# Unity
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/client/unity
Confidence Unity SDK for client-side feature flag evaluation in Unity games.
The Unity client SDK is currently in development. [Contact us](https://confidence.spotify.com/contact) if you're interested in using Confidence with Unity for client-side evaluation.
# Evaluation Context
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/context
The SDK uses the context data for targeting and randomization in flag evaluations (see [Define Rules](/docs/flags/define-rules)).
The Confidence SDKs use OpenFeature's [Evaluation Context](https://openfeature.dev/docs/reference/concepts/evaluation-context) to pass context data for flag evaluations. The context contains information about the user, session, or environment that Confidence uses for targeting rules and randomization.
Here is an example with the Go SDK:
```go theme={null}
evalCtx := openfeature.NewEvaluationContext("user-123", map[string]interface{}{
"country": "US",
"membershipLevel": "premium",
})
value, err := client.BooleanValue(ctx, "my-feature-flag.enabled", false, evalCtx)
```
## Managed Contexts
The Confidence SDKs can generate contextual data automatically, depending on the SDK.
### Visitor ID
Supported SDKs: JS (Client), Android, iOS, Flutter
The `visitor_id` is a unique identifier for each installation of a mobile application or browser instance (cookie). This context field is automatically added to the context by the mobile SDKs.
You can override its value by setting a custom context with the same key: `visitor_id`.
When using a combination of Client SDKs and Server SDKs, you should
append the `visitor_id` in relevant endpoint calls to your backend. This way,
the Server SDK can access it. Client SDKs expose APIs to read the
`visitor_id` for this purpose.
Refer to the dedicated section for more information on how and when to use the visitor ID as your
entity: [Entities](/docs/metrics/entities).
### Page
Supported SDKs: JS (Client)
Data about the current webpage where the user is at. The various fields are all wrapped in
a `page` struct:
* `path`
* `referrer`
* `search`
* `title`
* `url`
This page data is opt-in: refer to the [SDK README](https://github.com/spotify/confidence-sdk-js) for more information on how to enable this
context data in your application.
### App Lifecycle
Supported SDKs: Android, iOS
These context entries are automatically generated:
* `app_version`
* `app_build`
These App Lifecycle fields are opt-in: refer to each SDK README ([iOS](https://github.com/spotify/confidence-sdk-swift), [Android](https://github.com/spotify/confidence-sdk-android)) for more information on how to
enable this context data in your application.
## Related Resources
Overview of available SDKs
Configure entity types
Configure flag targeting
# Cloudflare Workers
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/edge/cloudflare
Confidence Cloudflare resolver for edge-based feature flag evaluation.
The Confidence Cloudflare resolver enables feature flag evaluation at the edge using Cloudflare Workers. Built on the [Confidence Resolver](https://github.com/spotify/confidence-resolver), a Rust-based resolver, it evaluates flags as close to your users as possible. You can then use the Confidence SDKs to resolve from the Cloudflare resolver, either via service binding or regular calls from clients.
## Features
* **Edge evaluation**: Flag rules evaluate at Cloudflare's edge locations worldwide
* **Ultra-low latency**: Evaluation happens close to users, minimizing latency
* **Rust-based resolver**: High-performance flag evaluation powered by the Confidence Resolver
* **Deployer-driven sync**: Run the deployer to fetch the latest flag rules from Confidence and re-deploy the Worker
## Service binding vs HTTP calls
When integrating with the Cloudflare resolver, you have two options for how your services communicate with it:
**Service binding (recommended)**: Cloudflare's [service bindings](https://developers.cloudflare.com/workers/runtime-apis/bindings/service-bindings/) allow Workers to call other Workers directly within Cloudflare's network. This internal routing bypasses the public internet, resulting in ultra-low latency. Use service bindings when your application runs on Cloudflare Workers.
**HTTP calls**: Standard HTTP requests to the resolver endpoint. This involves normal network routing with typically higher latency. Use this approach when calling from external services or client applications.
For the best performance in Cloudflare-based architectures, configure service bindings between your application Worker and the Confidence resolver Worker.
## Deployment
The Cloudflare resolver is deployed using a Docker-based deployer that handles building and publishing the Worker to your Cloudflare account.
### Prerequisites
* Docker installed
* Cloudflare API token with the following permissions:
* **Account > Workers Scripts > Edit**
* **Account > Workers Queues > Edit** (needed for the first deploy)
* Confidence client secret (must be type **BACKEND**)
### Deploy to Cloudflare
On the first deploy, create the required Cloudflare Queue:
```bash theme={null}
CLOUDFLARE_API_TOKEN='your-cloudflare-api-token' npx wrangler queues create flag-logs-queue
```
Then run the [deployer image](https://github.com/spotify/confidence-resolver/pkgs/container/confidence-cloudflare-deployer) with your credentials:
```bash theme={null}
docker run -it \
-e CLOUDFLARE_API_TOKEN='your-cloudflare-api-token' \
-e CONFIDENCE_CLIENT_SECRET='your-confidence-client-secret' \
ghcr.io/spotify/confidence-cloudflare-deployer:latest
```
The deployer automatically detects:
* **Cloudflare account ID** from your API token
* **Resolver state** from Confidence CDN
* **Existing deployment** to avoid unnecessary re-deploys
The deployer does not poll for changes. Each run fetches the current state from Confidence, deploys the Worker if the state has changed, and then exits. To keep the Worker up to date, run the deployer on a schedule (for example, via a cron job) or trigger it when flag rules or targeting changes are made in Confidence.
### Optional configuration
| Variable | Description |
| ------------------------------------ | ---------------------------------------------------------- |
| `CLOUDFLARE_ACCOUNT_ID` | Required only if API token has access to multiple accounts |
| `CONFIDENCE_RESOLVER_STATE_URL` | Custom resolver state URL (overrides CDN) |
| `CONFIDENCE_RESOLVER_ALLOWED_ORIGIN` | Configure allowed origins for CORS |
| `FORCE_DEPLOY` | Force re-deploy regardless of state changes |
| `NO_DEPLOY` | Build only, skip deployment |
## Using the resolver with service bindings
This section shows how to call the Confidence resolver from your own Cloudflare Worker using a service binding.
### Set up the project
```bash theme={null}
npm create cloudflare@latest my-worker
cd my-worker
npm install @spotify-confidence/sdk
```
### Configure the service binding
Add a service binding to your `wrangler.json` to connect your Worker to the resolver:
```json theme={null}
{
"$schema": "node_modules/wrangler/config-schema.json",
"name": "my-worker",
"main": "src/index.ts",
"compatibility_date": "2025-02-04",
"services": [
{
"binding": "ConfidenceBinding",
"service": "confidence-cloudflare-resolver"
}
]
}
```
### Resolve flags in your Worker
Use the [`@spotify-confidence/sdk`](https://github.com/spotify/confidence-sdk-js) package and route resolve requests through the service binding with `fetchImplementation`:
```typescript theme={null}
import { Confidence } from '@spotify-confidence/sdk';
interface Env {
CONFIDENCE_CLIENT_SECRET: string;
ConfidenceBinding: {
fetch: (request: Request) => Promise;
};
}
export default {
async fetch(request, env, ctx): Promise {
const confidence = Confidence.create({
clientSecret: env.CONFIDENCE_CLIENT_SECRET,
environment: 'backend',
fetchImplementation: (req: Request) => env.ConfidenceBinding.fetch(req),
timeout: 1000,
});
const flag = await confidence
.withContext({ targeting_key: 'user-123' })
.evaluateFlag('my-flag', {});
return new Response(JSON.stringify({ flag }), {
headers: { 'Content-Type': 'application/json' },
});
},
} satisfies ExportedHandler;
```
* **`fetchImplementation`** routes resolve requests through the service binding instead of the public internet.
* **`environment: 'backend'`** is required for server-side usage.
* **`withContext()`** passes your evaluation context for flag targeting.
The Cloudflare resolver also works with the [`@spotify-confidence/openfeature-server-provider`](https://github.com/spotify/confidence-sdk-js/tree/main/packages/openfeature-server-provider), if you prefer using the OpenFeature API.
### Deploy
```bash theme={null}
npx wrangler deploy
```
## Limitations
* **Sticky assignments**: Not currently supported with the Cloudflare resolver. Flags with sticky assignment rules will return "flag not found".
## Resources
Source code and deployment instructions
Confidence SDK for JavaScript/TypeScript
Cloudflare Workers documentation
Configure evaluation context
Understand flag assignment tracking
# Fastly Compute
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/edge/fastly
Confidence Fastly resolver for edge-based feature flag evaluation.
The Fastly edge resolver is currently in development. [Contact us](https://confidence.spotify.com/contact) if you're interested in using Confidence with Fastly Compute for edge-based flag evaluation.
# Confidence SDKs
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/introduction
Confidence provides SDKs for multiple languages and platforms, with the goal of managing and simplifying flag resolving.
All the Confidence SDKs support the [OpenFeature](https://openfeature.dev)
standard, a project from [Cloud Native Computing Foundation](https://www.cncf.io/) (CNCF) that aims to standardize feature
flagging. With the OpenFeature integration your application interacts with the
OpenFeature API and a `Provider` provides the connection to Confidence.
This documentation refers to the overall setup as "Confidence SDK" for
simplicity.
## Server SDKs
Confidence offers **Server SDKs**
built on an open source [Confidence Resolver](https://github.com/spotify/confidence-resolver)—a Rust-based
flag resolver that can run natively or as WebAssembly. These SDKs evaluate flag rules to flag values entirely on your infrastructure. The resolver syncs flag rules
and logging with the Confidence backend periodically, in the background. This enables:
* **Ultra-low latency**: Flag evaluations happen locally in microseconds
* **High reliability**: No network dependency at evaluation time
Go SDK
Java SDK
Node.js SDK
Next.js with App Router
Rust SDK
Python SDK
Ruby SDK
.NET SDK (server)
PHP SDK
## Client SDKs
Client SDKs are a specialized form of local resolve designed for mobile and web applications.
They resolve all flags once according to the evaluation context, and flag values are then
read from a local cache. This approach works well when the evaluation context doesn't change
often—for example, a mobile client that sets the context on login or app start and maintains
it throughout the session.
Swift SDK
Kotlin SDK
JavaScript SDK
Flutter SDK
Unity SDK
.NET SDK (client)
## Edge Resolve
The [WebAssembly based](https://github.com/spotify/confidence-resolver) resolver is also packaged to run on edge computing platforms.
This enables flag evaluation as close to your users as possible, minimizing latency for
geographically distributed applications.
Edge resolver for Cloudflare Workers
Edge resolver for Fastly Compute
## Related Resources
Configure evaluation context
Track flag assignments
Overview of feature flags
# MCP Servers
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/mcp-servers
Confidence MCP servers give AI assistants access to documentation, feature flag management, and experiment analysis tools.
Confidence has three MCP servers:
* **Documentation MCP**: Search and retrieve Confidence documentation
* **Flags MCP**: Create and manage feature flags
* **Experiments MCP**: Explore and analyze experiments and their results
## Documentation MCP Server
The Documentation MCP provides AI assistants with access to Confidence documentation through semantic search, regular expression search, and SDK integration guides.
### Integration Setup
**No authentication required** for the Documentation MCP server. The server uses streamable HTTP as the transport and the URL is `https://mcp.confidence.dev/mcp/docs`.
```bash theme={null}
claude mcp add --transport http confidence-docs https://mcp.confidence.dev/mcp/docs
```
```bash theme={null}
codex mcp add confidence-docs --url https://mcp.confidence.dev/mcp/docs
```
Use `Command + Shift + P` (`Ctrl + Shift + P` on Windows) and search for `Open MCP settings`.
Select **New MCP Server** and add the following to the `mcpServers` JSON:
```json theme={null}
"confidence-docs": "https://mcp.confidence.dev/mcp/docs"
```
Create a `.vscode` directory in your workspace root if it doesn't exist.
Create a file named `mcp.json` in the `.vscode` directory.
Add the `confidence-docs` server to your configuration:
```json theme={null}
{
"servers": {
"confidence-docs": { "url": "https://mcp.confidence.dev/mcp/docs" }
}
}
```
Click `Start` next to the MCP server name in the JSON file to start it.
### Available Tools
#### searchDocumentation
Searches the Confidence documentation using semantic search to learn about experiments, A/B tests, rollouts, feature flags, metrics, surfaces, insights, and other experiment-related topics. Returns the 10 most relevant chunks using Maximum Marginal Relevance (MMR) to balance relevance and diversity.
#### getFullSource
Retrieves the full text content from a specific Confidence documentation source URL.
#### grepDocumentation
Searches the Confidence documentation content using regular expressions. Returns a list of documentation pages that contain matches for the given regular expression pattern.
#### getCodeSnippetAndSdkIntegrationTips
Gets complete integration guides with code examples and README documentation for integrating Confidence feature flags. Returns full integration examples including OpenFeature provider setup, best practices, and configuration for the specified SDK.
### Example Usage
Once integrated, ask your AI assistant to:
* "Search Confidence documentation for information about A/B testing"
* "Show how to integrate Confidence with Python"
* "Summarize the A/B test quickstart"
## Flag Management MCP Server
The Flag Management MCP provides AI assistants with tools to manage Confidence feature flags, including creation, modification, targeting rules, and testing.
### Integration Setup
The Confidence Flags MCP **requires authentication**. Allow access from your IDE or developer tool where you use the MCP server. The server uses streamable HTTP as the transport and the URL is `https://mcp.confidence.dev/mcp/flags`.
```bash theme={null}
claude mcp add --transport http confidence-flags https://mcp.confidence.dev/mcp/flags
```
After adding, open Claude Code, run `/mcp` and select `confidence-flags` to authenticate.
```bash theme={null}
codex mcp add confidence-flags --url https://mcp.confidence.dev/mcp/flags
codex mcp login confidence-flags
```
Use `Command + Shift + P` (`Ctrl + Shift + P` on Windows) and search for `Open MCP settings`.
Select **New MCP Server** and add the following to the `mcpServers` JSON:
```json theme={null}
"confidence-flags": "https://mcp.confidence.dev/mcp/flags"
```
Click `Connect` to authenticate.
Create a `.vscode` directory in your workspace root if it doesn't exist.
Create a file named `mcp.json` in the `.vscode` directory.
Add the `confidence-flags` server to your configuration:
```json theme={null}
{
"servers": {
"confidence-flags": { "url": "https://mcp.confidence.dev/mcp/flags" }
}
}
```
Click `Start` next to the MCP server name in the JSON file and authenticate when prompted.
### Available Tools
#### listClients
Lists all available Confidence clients for flag operations.
#### listFlags
Lists all active feature flags with summary information including names, schemas, and variants.
#### getFlag
Retrieves detailed information about a specific feature flag, including its complete schema, variants, and targeting rules.
#### createFlag
Creates a new feature flag with schema and variant definitions.
#### addFlagVariant
Adds a new variant to an existing feature flag.
#### updateFlagSchema
Updates the schema definition of an existing feature flag.
#### createOverrideRule
Creates targeting rules to override flag behavior for specific entities.
#### testResolveFlag
Tests flag resolution for specific clients and entities and explains why the system selects a particular variant.
#### analyzeFlagUsage
Finds unused Confidence feature flags or flags fully rolled out, helping you clean up feature flag code.
### Example Usage
Once integrated, ask your AI assistant to:
* "List all feature flags in Confidence"
* "Create a new flag called 'new-checkout-flow' with a boolean schema"
* "Add a variant to the 'new-checkout-flow' flag"
* "Create an individual targeting rule for user 'test-user' to see the 'variant-b' of 'experiment-flag'"
* "Test flag resolution for 'new-checkout-flow' with user '[john@example.com](mailto:john@example.com)'"
* "Analyze what flags you can clean up in this codebase"
## Experiments MCP Server
The Experiments MCP provides AI assistants with tools to explore and analyze Confidence experiments, including A/B tests and rollouts.
### Integration Setup
The Confidence Experiments MCP **requires authentication**. Allow access from your IDE or developer tool where you use the MCP server. The server uses streamable HTTP as the transport and the URL is `https://mcp.confidence.dev/mcp/experiments`.
```bash theme={null}
claude mcp add --transport http confidence-experiments https://mcp.confidence.dev/mcp/experiments
```
After adding, open Claude Code, run `/mcp` and select `confidence-experiments` to authenticate.
```bash theme={null}
codex mcp add confidence-experiments --url https://mcp.confidence.dev/mcp/experiments
codex mcp login confidence-experiments
```
Use `Command + Shift + P` (`Ctrl + Shift + P` on Windows) and search for `Open MCP settings`.
Select **New MCP Server** and add the following to the `mcpServers` JSON:
```json theme={null}
"confidence-experiments": "https://mcp.confidence.dev/mcp/experiments"
```
Click `Connect` to authenticate.
Create a `.vscode` directory in your workspace root if it doesn't exist.
Create a file named `mcp.json` in the `.vscode` directory.
Add the `confidence-experiments` server to your configuration:
```json theme={null}
{
"servers": {
"confidence-experiments": { "url": "https://mcp.confidence.dev/mcp/experiments" }
}
}
```
Click `Start` next to the MCP server name in the JSON file and authenticate when prompted.
### Available Tools
#### list\_experiments
Lists Confidence experiments (A/B tests and rollouts). Returns experiment names, display names, states, owners, and creation times. Supports two modes:
* **List mode** (default): Use the `filter` parameter with `Lucene` query syntax (for example, `state:live` for running experiments) and `orderBy` to sort results.
* **Search mode**: Provide a `query` parameter for full-text search. In search mode, results order by relevance and the `filter` and `orderBy` parameters have no effect.
You can infer the experiment type from the resource name (for example, `workflows/abtest/...` or `workflows/rollout/...`).
#### get\_experiment
Retrieves detailed information about a specific experiment. By default, returns a compact summary with key fields:
* Name, state, owner, and creation time
* Metrics with types, minimum detectable effect (MDE), and non-inferiority margin (NIM)
* Decision outcome (for ended A/B tests)
* Available analysis result names
Set `summary=false` for full details including configuration, checks, stats, and state history.
#### get\_results
Retrieves statistical analysis results for an experiment. By default, returns a summary with:
* Relative effect estimates and confidence intervals
* Significance status and status messages
* Sample sizes per treatment
* Shipping recommendations per treatment
Accepts either an experiment instance name (for example, `workflows/abtest/instances/{id}`) or a specific analysis result name. When given an instance name, the tool automatically resolves the primary analysis result.
Set `summary=false` for the full detailed output.
#### get\_resource
Looks up details of Confidence resources referenced in experiment data. Supports:
* `metrics/*` - Metric definitions
* `entities/*` - Entity definitions
* `surfaces/*` - Surface configurations
* `segments/*` - Segment definitions
* `factTables/*` - Fact table definitions
* `flags/*` - Feature flag configurations
* `identities/*` - User identities
* `assignmentTables/*` - Assignment table definitions
* `clients/*` - Client configurations
### Example Usage
Once integrated, ask your AI assistant to:
* "List all running experiments"
* "Search for experiments related to checkout"
* "Show details about the homepage-redesign experiment"
* "Get the results for the checkout-flow A/B test"
* "What's the status of the metrics in the onboarding rollout?"
# .NET
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/dotnet
Confidence .NET SDK for server-side feature flag evaluation.
Confidence provides a fully supported .NET SDK for server-side feature flag evaluation.
For installation instructions, usage examples, and source code, visit the [Confidence .NET SDK repository](https://github.com/spotify/confidence-sdk-dotnet) on GitHub.
# Go
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/go
Confidence Go SDK for server-side feature flag evaluation.
The Confidence Go SDK provides ultra-low latency feature flag evaluation for Go applications using the [Confidence Resolver](https://github.com/spotify/confidence-resolver)—a Rust-based resolver that runs natively or as WebAssembly.
## Features
* **Local evaluation**: Flag rules evaluate on your infrastructure in microseconds
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Background sync**: Flag rules and logging sync with Confidence in the background
* **High reliability**: No network dependency at evaluation time
## Installation
```bash theme={null}
go get github.com/spotify/confidence-resolver/openfeature-provider/go
go mod tidy
```
## Quickstart
```go theme={null}
package main
import (
"context"
"log"
"github.com/open-feature/go-sdk/openfeature"
"github.com/spotify/confidence-resolver/openfeature-provider/go/confidence"
)
func main() {
ctx := context.Background()
// Initialize the Confidence provider
provider, err := confidence.NewProvider(ctx, confidence.ProviderConfig{
ClientSecret: "your-client-secret",
})
if err != nil {
log.Fatalf("Failed to create provider: %v", err)
}
// Register with OpenFeature
openfeature.SetProviderAndWait(provider)
client := openfeature.NewClient("my-app")
// Evaluate a flag
evalCtx := openfeature.NewEvaluationContext("user-123", map[string]interface{}{
"country": "US",
})
value, err := client.BooleanValue(ctx, "my-feature-flag.enabled", false, evalCtx)
if err != nil {
log.Printf("Flag evaluation failed: %v", err)
}
log.Printf("Flag value: %v", value)
}
```
## Resources
Source code and examples
OpenFeature documentation
Configure evaluation context
Understand flag assignment tracking
# Java
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/java
Confidence Java SDK for server-side feature flag evaluation.
The Confidence Java SDK provides ultra-low latency feature flag evaluation for Java applications using the [Confidence Resolver](https://github.com/spotify/confidence-resolver)—a Rust-based resolver that runs natively.
## Features
* **Local evaluation**: Flag rules evaluate on your infrastructure in microseconds
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Background sync**: Flag rules and logging sync with Confidence in the background
* **High reliability**: No network dependency at evaluation time
## Installation
```xml theme={null}
com.spotify.confidenceopenfeature-provider-locallatest
```
```groovy theme={null}
implementation 'com.spotify.confidence:openfeature-provider-local:'
```
## Quickstart
```java theme={null}
import com.spotify.confidence.OpenFeatureLocalResolveProvider;
import dev.openfeature.sdk.OpenFeatureAPI;
import dev.openfeature.sdk.Client;
import dev.openfeature.sdk.MutableContext;
public class App {
public static void main(String[] args) {
// Initialize the Confidence provider
OpenFeatureLocalResolveProvider provider =
new OpenFeatureLocalResolveProvider("your-client-secret");
// Register with OpenFeature
OpenFeatureAPI.getInstance().setProviderAndWait(provider);
Client client = OpenFeatureAPI.getInstance().getClient();
// Evaluate a flag
MutableContext ctx = new MutableContext("user-123");
ctx.add("country", "US");
Boolean value = client.getBooleanValue("my-feature-flag.enabled", false, ctx);
System.out.println("Flag value: " + value);
}
}
```
## HTTP Proxy Service
The `FlagResolverService` lets you proxy flag resolution requests from client SDKs through your Java backend. Instead of client SDKs connecting directly to Confidence servers, they connect to your service, which resolves flags locally.
The proxy service enables:
* **Backend-controlled credentials**: Client SDKs don't need their own client secrets
* **Context enrichment**: Add server-side context (user ID from auth, request metadata) before resolution
* **Low-latency resolution**: Client SDKs make requests to your backend instead of Confidence servers, which can sometimes improve latency
### Setup
```java theme={null}
import com.spotify.confidence.OpenFeatureLocalResolveProvider;
import com.spotify.confidence.FlagResolverService;
// Create and initialize the provider
OpenFeatureLocalResolveProvider provider =
new OpenFeatureLocalResolveProvider("your-client-secret");
OpenFeatureAPI.getInstance().setProviderAndWait(provider);
// Create the HTTP service
FlagResolverService flagResolver = new FlagResolverService(provider);
```
### Context Decoration
Add server-side context to requests before resolution:
```java theme={null}
FlagResolverService flagResolver = new FlagResolverService(provider,
ContextDecorator.sync((ctx, req) -> {
// Replace with your own header key
List userIds = req.headers().get("X-User-Id");
if (userIds != null && !userIds.isEmpty()) {
return ctx.merge(new ImmutableContext(userIds.get(0)));
}
return ctx;
}));
```
### Framework Integration
The service exposes `handleResolve` and `handleApply` methods that accept a `ConfidenceHttpRequest` and return a `ConfidenceHttpResponse`. Adapt these to any HTTP framework:
```java theme={null}
public class FlagServlet extends HttpServlet {
private final FlagResolverService flagResolverService;
public FlagServlet(OpenFeatureLocalResolveProvider provider) {
this.flagResolverService = new FlagResolverService(provider,
ContextDecorator.sync((context, request) -> {
List userIds = request.headers().get("X-User-Id");
if (userIds != null && !userIds.isEmpty()) {
return context.merge(new ImmutableContext(userIds.get(0)));
}
return context;
}));
}
@Override
protected void service(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
ConfidenceHttpResponse response;
if (req.getPathInfo().endsWith("v1/flags:resolve")) {
response = flagResolverService.handleResolve(toConfidenceRequest(req))
.toCompletableFuture().join();
} else if (req.getPathInfo().endsWith("v1/flags:apply")) {
response = flagResolverService.handleApply(toConfidenceRequest(req))
.toCompletableFuture().join();
} else {
resp.setStatus(404);
return;
}
resp.setStatus(response.statusCode());
response.headers().forEach(resp::setHeader);
resp.getOutputStream().write(response.body());
}
private ConfidenceHttpRequest toConfidenceRequest(HttpServletRequest req) {
final byte[] bodyBytes;
try {
bodyBytes = req.getInputStream().readAllBytes();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
return new ConfidenceHttpRequest() {
@Override
public String method() { return req.getMethod(); }
@Override
public byte[] body() { return bodyBytes; }
@Override
public Map> headers() {
Map> headers = new HashMap<>();
Collections.list(req.getHeaderNames()).forEach(name ->
headers.put(name, Collections.list(req.getHeaders(name))));
return headers;
}
};
}
}
```
### Client SDK configuration
Configure client SDKs to resolve flags through your backend:
```javascript theme={null}
const confidence = Confidence.create({
clientSecret: 'not-used-but-required',
resolveBaseUrl: 'https://your-backend.com/confidence-flags',
applyBaseUrl: 'https://your-backend.com/confidence-flags',
});
```
Only `application/json` content type is supported. Requests with other content types receive a 415 response.
## Resources
Source code and examples
OpenFeature documentation
Configure evaluation context
Understand flag assignment tracking
# JavaScript (Node.js)
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/javascript
Confidence JavaScript SDK for server-side feature flag evaluation in Node.js.
The Confidence JavaScript SDK provides ultra-low latency feature flag evaluation for Node.js applications using the [Confidence Resolver](https://github.com/spotify/confidence-resolver)—a Rust-based resolver that runs as WebAssembly.
## Features
* **Local evaluation**: Flag rules evaluate on your infrastructure in microseconds
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Background sync**: Flag rules and logging sync with Confidence in the background
* **High reliability**: No network dependency at evaluation time
## Installation
```bash npm theme={null}
npm install @spotify-confidence/openfeature-server-provider-local
```
```bash yarn theme={null}
yarn add @spotify-confidence/openfeature-server-provider-local
```
```bash pnpm theme={null}
pnpm add @spotify-confidence/openfeature-server-provider-local
```
**Requirements**: Node.js 18+ with WebAssembly support.
## Quickstart
```typescript theme={null}
import { OpenFeature } from '@openfeature/server-sdk';
import { createConfidenceServerProvider } from '@spotify-confidence/openfeature-server-provider-local';
// Initialize the Confidence provider
const provider = createConfidenceServerProvider({
flagClientSecret: process.env.CONFIDENCE_FLAG_CLIENT_SECRET!,
});
// Register with OpenFeature
await OpenFeature.setProviderAndWait(provider);
const client = OpenFeature.getClient();
// Evaluate a flag with context
const context = {
targetingKey: 'user-123',
country: 'US',
plan: 'premium',
};
const value = await client.getBooleanValue('my-feature-flag.enabled', false, context);
console.log('Flag value:', value);
```
## Resources
Source code and examples
OpenFeature documentation
Configure evaluation context
Understand flag assignment tracking
# Next.js
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/nextjs
Confidence SDK for feature flag evaluation in Next.js applications with App Router support.
The Confidence Next.js SDK provides feature flag evaluation for Next.js applications using the [Confidence Resolver](https://github.com/spotify/confidence-resolver)—a Rust-based resolver that runs as WebAssembly. It supports both React Server Components and Client Components with the App Router.
## Features
* **Local evaluation**: Flag rules evaluate on your infrastructure in microseconds
* **App Router support**: Works with React Server Components and Client Components
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Background sync**: Flag rules and logging sync with Confidence in the background
* **High reliability**: No network dependency at evaluation time
## Installation
```bash npm theme={null}
npm install @spotify-confidence/openfeature-server-provider-local
```
```bash yarn theme={null}
yarn add @spotify-confidence/openfeature-server-provider-local
```
```bash pnpm theme={null}
pnpm add @spotify-confidence/openfeature-server-provider-local
```
**Requirements**: Node.js 18+ with WebAssembly support.
## Setup
### 1. Configure the provider
Create a file to initialize the Confidence provider:
```typescript theme={null}
// lib/confidence.ts
import { OpenFeature } from '@openfeature/server-sdk';
import { createConfidenceServerProvider } from '@spotify-confidence/openfeature-server-provider-local';
const provider = createConfidenceServerProvider({
flagClientSecret: process.env.CONFIDENCE_FLAG_CLIENT_SECRET!,
});
// Initialize provider at startup
await OpenFeature.setProviderAndWait(provider);
```
### 2. Add the provider to your layout
Wrap your application with the `ConfidenceProvider`:
```tsx theme={null}
// app/layout.tsx
import { ConfidenceProvider } from '@spotify-confidence/openfeature-server-provider-local/react-server';
import './lib/confidence';
export default async function RootLayout({ children }: { children: React.ReactNode }) {
const context = {
targetingKey: 'user-123',
country: 'US',
};
return (
{children}
);
}
```
## Usage
### Server Components
Use the `getFlag` function in React Server Components:
```tsx theme={null}
// app/page.tsx
import { getFlag } from '@spotify-confidence/openfeature-server-provider-local/react-server';
export default async function Page() {
const showNewLayout = await getFlag('page-layout.showNewLayout', false, {
targetingKey: 'user-123'
});
return showNewLayout ? : ;
}
```
### Client Components
Use the `useFlag` hook in Client Components:
```tsx theme={null}
// components/FeatureButton.tsx
'use client';
import { useFlag } from '@spotify-confidence/openfeature-server-provider-local/react-client';
export function FeatureButton() {
const enabled = useFlag('new-feature.enabled', false);
if (!enabled) return null;
return ;
}
```
## Resources
Source code and examples
OpenFeature documentation
Configure evaluation context
Understand flag assignment tracking
# PHP
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/php
Confidence PHP SDK for server-side feature flag evaluation.
The PHP SDK is currently in development. [Contact us](https://confidence.spotify.com/contact) if you're interested in using Confidence with PHP.
# Python
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/python
Confidence Python SDK for server-side feature flag evaluation.
The Confidence Python SDK provides ultra-low latency feature flag evaluation for Python applications using the [Confidence Resolver](https://github.com/spotify/confidence-resolver)—a Rust-based resolver that runs as WebAssembly.
## Features
* **Local evaluation**: Flag rules evaluate on your infrastructure in microseconds
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Background sync**: Flag rules and logging sync with Confidence in the background
* **High reliability**: No network dependency at evaluation time
## Installation
```bash theme={null}
pip install confidence-openfeature-provider
```
**Requirements**: Python 3.10+ and OpenFeature SDK 0.8.0+.
## Quickstart
```python theme={null}
from openfeature import api
from openfeature.evaluation_context import EvaluationContext
from confidence import ConfidenceProvider
# Initialize the Confidence provider
provider = ConfidenceProvider(client_secret="your-client-secret")
# Register with OpenFeature
api.set_provider_and_wait(provider)
client = api.get_client()
# Evaluate a flag with context
context = EvaluationContext(
targeting_key="user-123",
attributes={
"country": "US",
"plan": "premium",
}
)
enabled = client.get_boolean_value("my-feature-flag.enabled", default_value=False, evaluation_context=context)
print(f"Flag value: {enabled}")
# Shutdown the provider when your application exits
api.shutdown()
```
## Resources
Source code and examples
OpenFeature documentation
Configure evaluation context
Understand flag assignment tracking
# Ruby
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/ruby
Confidence Ruby SDK for server-side feature flag evaluation.
The Ruby SDK is currently in development. [Contact us](https://confidence.spotify.com/contact) if you're interested in using Confidence with Ruby.
# Rust
Source: https://confidence-auth-testing.mintlify.io/docs/sdks/server/rust
Confidence Rust SDK for server-side feature flag evaluation.
The Confidence Rust SDK provides ultra-low latency feature flag evaluation for Rust applications using the [Confidence Resolver](https://github.com/spotify/confidence-resolver)—a native Rust resolver with async/await support built on Tokio.
## Features
* **Local evaluation**: Flag rules evaluate on your infrastructure in microseconds
* **OpenFeature compatible**: Standard feature flag API through OpenFeature provider
* **Background sync**: Flag rules and logging sync with Confidence in the background
* **High reliability**: No network dependency at evaluation time
## Installation
Add these dependencies to your `Cargo.toml`:
```toml theme={null}
[dependencies]
spotify-confidence-openfeature-provider-local = ""
open-feature = ""
```
## Quickstart
```rust theme={null}
use open_feature::{EvaluationContext, OpenFeature};
use spotify_confidence_openfeature_provider_local::{ConfidenceProvider, ProviderOptions};
#[tokio::main]
async fn main() -> Result<(), Box> {
// Create provider options with your client secret
let options = ProviderOptions::new("your-client-secret");
// Create the Confidence provider
let provider = ConfidenceProvider::new(options)?;
// Set the provider on the OpenFeature singleton
OpenFeature::singleton_mut()
.await
.set_provider(provider)
.await;
// Create an OpenFeature client
let client = OpenFeature::singleton().await.create_client();
// Create evaluation context with user attributes for targeting
let context = EvaluationContext::default()
.with_targeting_key("user-123")
.with_custom_field("country", "US")
.with_custom_field("plan", "premium");
// Evaluate a boolean flag
let enabled = client
.get_bool_value("my-feature-flag.enabled", Some(&context), None)
.await
.unwrap_or(false);
println!("Flag value: {}", enabled);
Ok(())
}
```
## Resources
Source code and examples
OpenFeature documentation
Configure evaluation context
Understand flag assignment tracking
# Surfaces
Source: https://confidence-auth-testing.mintlify.io/docs/surfaces/introduction
Use surfaces to organize your experiments with a focus on your product.
When multiple teams experiment in the same app or on the same website they often need to work
together and be aware of each other. Surfaces is a concept in Confidence that facilitates this
collaboration.
Watch this video to get an introduction to surfaces and learn how to organize, coordinate, and configure experiments with surfaces.
Think of a surface as a logical representation of some part of your app or website, under which you
can organize experiments. A surface becomes a natural place where folks in your company can go to
see which experiments are running and what's coming up for a particular part of the
product. For example, customer support can quickly go in and see what kind of experiments are
running if a user reports a problem on a particular page.
## Surface Settings
The surface owner and admins can configure the
[surface settings](./surface-settings) which include creation of exclusivity groups and holdbacks,
notifications, required metrics, automatic actions, and reviews.
## Related Resources
Configure exclusivity groups, holdbacks, and reviews
Use surfaces to coordinate experiments
Make experiments mutually exclusive
Configure surface notifications
# Organize Experiments with Surfaces
Source: https://confidence-auth-testing.mintlify.io/docs/surfaces/organize-experiments
Tie experiments to surfaces to coordinate scheduling and prevent conflicts between overlapping changes.
Tie an experiment to a surface to let others working on that surface see your experiment. This
allows you to plan experiments better. If you experiment on the same part of an app or a page, or if
there's another reason your experiments mustn't collide, you can coordinate them using exclusivity
tags. You can also decide to run them at distinct points in time.
Interaction effects between experiments are rare, see
[this report](https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/a-b-interactions-a-call-to-relax/)
by Microsoft. For this reason, many companies, including Spotify, only coordinate experiments that
directly influence each other. For example, if two teams work on the design of a page and the work
by one of the teams is only functional if the other team's work doesn't change the page, they should
coordinate their experiments.
## The Global Surface
Every account has a global surface. All experiments are automatically part of the
global surface and it's impossible to run experiment outside of this surface.
Use this surface to configure settings that should apply to all your
organization's experiments.
## Create Surfaces
Give your surfaces names that:
* are self-explanatory, so that people from different parts of the organization can understand what
the surface is and what it isn't
* don't use organization-specific words, as the structure of the organization often changes more
often than the product structure
Describe your surfaces with:
* a short sentence about what the surface is
* examples of typical experiments that should run on this surface
* examples of experiments that you suspect your colleagues might think should run on this surface, but that in fact shouldn't
For an e-commerce website, examples of surfaces could be:
* Home page
* Checkout
* Products
* Search
* Customer support
Each surface is its own area in the product. Multiple teams can work on the same surface.
Experiments can belong to multiple surfaces at the same time. All experiments belong to the global surface.
### Surface Size
When you start to exceed 30-40 live experiments in a surface at any given time, you should consider
splitting up the surface into smaller surfaces to lower the load of the experimenter. The purpose of
surfaces is to make it easier for experimenters to keep track of experiments that are relevant for
their experiment. If there are too many experiments on a surface, this is overwhelming.
## Use Surfaces to Coordinate Experiments
Create exclusivity groups and holdbacks on surfaces to coordinate experiments.
Go to the **Surface** tab and click *Settings* in the top right corner.
Watch this video to get a quick overview of how to use exclusivity groups to coordinate experiments in Confidence.
Learn how to create exclusivity groups and holdbacks in the [documentation](/docs/experiments/exclusive-experiments).
## Related Resources
Overview of surfaces in Confidence
Configure surface settings
Make experiments mutually exclusive
Run A/B tests on surfaces
# Surface Settings
Source: https://confidence-auth-testing.mintlify.io/docs/surfaces/surface-settings
Understand the settings you can configure on a surface.
On the surface page, you can configure settings that apply to all experiments
on the surface.
Surface settings apply for experiments that run on the surface where the
settings live. Only settings on the global surface apply to all
experiments.
## Reviews
Add suggested or required reviewers to all experiments on a surface. Required
reviews block the launch of experiments until at least one of the required
reviewers has approved the experiment setup. Suggested reviewers automatically
show up as optional reviewers on the experiment design page.
Required reviews add friction to the experiment launch process. Use it
only on surfaces where mistakes come with high costs.
Review requests appear as a todo-list item on the reviewer's Confidence home
page. To get Slack notifications, the reviewer needs to [integrate their Slack
account with
Confidence](../notifications/introduction#integrate-your-slack-account-with-confidence).
## Notifications
Send notifications for activities on a surface to Slack or email.
## Exclusivity Groups
Use exclusivity groups in experiments to make them mutually exclusive to each other.
You can configure exclusivity groups to be:
* **Optional**: The exclusivity group is not automatically selected by experiments on the surface, but experimenters can choose to add it.
* **Suggested**: The exclusivity group is pre-selected by experiments on the surface. Experimenters can choose to remove it.
To select an exclusivity group in an experiment, you need to select to
run the experiment on the surface on which the exclusivity group lives.
Read more about mutually exclusive experiments in the [exclusivity documentation](../experiments/exclusive-experiments).
An exclusivity group is not allocating anything in itself. An exclusivity group is a
resource that other experiments and rules can use to decide how to overlap or
not overlap with each other.
### Name Exclusivity Groups
Although exclusivity groups are naturally grouped by the surfaces they live on, you should give them names that are descriptive of the
coordination you use them for. For example, if you create an exclusivity group
to coordinate experiments for the search result ranker ML models, you can name
the exclusivity group `search-ranker`.
## Holdbacks
Holdbacks are a random set of users that you can reuse over time. Use holdbacks to:
* Hold users back from a set of experiments for a certain period of time
* Experiment on the same subset of users across several experiments over a period of time
You can configure holdbacks to be:
* **Optional**: The holdback is not automatically selected by experiments on the surface, but experimenters can choose to add it.
* **Suggested**: The holdback is pre-selected by experiments on the surface. Experimenters can choose to remove it.
* **Required**: The holdback is pre-selected by experiments on the surface. Experimenters *cannot* choose to remove it.
If you create several holdbacks on the same surface, they are mutually exclusive
to each other. Holdbacks on different surfaces are randomly overlapping. In one
experiment, you can use one or several holdbacks from one or several surfaces.
Holdbacks don't allocate any users themselves and Confidence
collects no exposure or assignment data for units in the holdback. A holdback
is a reference to a set of users that you can target or avoid in your
experiments.
### New Users in Holdbacks
A holdback is a random subset which makes up some proportion of all units. When
new users come in, they spread proportionally into and outside of the
holdback. For example, if you have a holdback with a 15% allocation, and 100
new users visit your app, around 15 of them belong to the holdback.
## Metrics
Add metrics to a surface to make them easier to find for
experimenters experimenting on this surface, or enforce that all experiments
on the surface check this metric for regressions by making it `required`.
Required
metrics [don't have non-inferiority margins](../experiments/metrics#without-non-inferiority-margin), but test for
deterioration. These metrics don't impact power analyses and sample size
calculations. If an experiment is part of multiple surfaces, the required
metrics are automatically de-duplicated. You can also explicitly add a required
metric to an experiment. In this case, the configuration from the experimenter
takes precedence, because Confidence checks all metrics that are part of an
experiment for deterioration. Confidence alerts the
user if any metric moves significantly in the non-desired direction.
Read more about notifications for alerts in the [notifications documentation](../notifications/introduction).
Suggested metrics are shown at the top of the metrics list when experimenters select metrics for an experiment.
## Linked Users and Groups
Link Confidence users or groups to a surface to ensure they run their experiments on it. When a linked user creates an experiment, the surface is automatically included based on the enforcement level.
You can configure linked users and groups to be:
* **Suggested**: The surface is pre-selected when the linked user creates an experiment. The user can choose to remove it.
* **Required**: The surface is pre-selected when the linked user creates an experiment. The user *cannot* remove it.
Use required linking for teams that must always run experiments on a
specific surface, for example to enforce review policies, required
holdbacks, or metric guardrails.
## Actions
Trigger actions when experiments on a surface meet certain conditions. You can configure 'if this then that' rules.
The following triggers are available:
* Experiment ends
* Metric moves significantly in the non-desired direction
With the following actions:
* Create and run an exploratory analysis
* End the experiment
For the actions based on metrics moving significantly in the non-desired
direction, the metric must be a required metric on the surface.
You can create multiple actions for the same trigger.
**Example:** If you want to end an experiment and create an exploratory analysis when a metric moves significantly in the non-desired direction, you create the following actions:
* If metric `` moves significantly in the non-desired direction, then end the experiment
* If metric `` moves significantly in the non-desired direction, then create and run an exploratory analysis with dimensions `` and ``
## Related Resources
Learn about surfaces in Confidence
Coordinate experiments with surfaces
Configure mutually exclusive experiments
Manage experiment reviews
# Get Started with BigQuery
Source: https://confidence-auth-testing.mintlify.io/docs/warehouse-setup/bigquery
Configure read and write connections to BigQuery.
## Introduction
If you followed the setup guide on the **Admin** page, your
setup is already complete. You can skip directly to [what's next](#whats-next).
You can run this setup with an AI assistant instead. Run this skill and your assistant guides you through the steps:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins --skill setup-warehouse-bigquery
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants.
This tutorial helps you configure Confidence to:
1. Run queries in BigQuery to compute exposure and metrics.
2. Store assignment data in BigQuery.
Step 2 is optional if you already have assignment data in BigQuery. For
example, if you are using a feature flagging solution other than Confidence
Flags.
This document targets the following audiences:
* Administrators who want to set up Confidence for their organization
In this tutorial, you create a service account in GCP that Confidence impersonates.
The service account used by Confidence is `account-@spotify-confidence.iam.gserviceaccount.com`.
You can name your service account anything you want.
For clarity, the tutorial assumes you create a service account called `confidence`.
You can read more about service account impersonation in the Google Cloud documentation.
## Before You Begin
* You need to have a Confidence account.
* You need to have a Google Cloud Platform account.
* You need to have permissions to create service accounts.
* You need to have the following APIs enabled in your Google Cloud Platform
project:
* BigQuery API
* Cloud Resource Manager API
## Terraform Scripts
Terraform scripts are available to create the service account, set up permissions, and create the required datasets.
If you are using Terraform, then after you have applied the Terraform configuration, you can skip directly to [Step 1c](#step-1c-configure-a-metrics-data-warehouse).
## Step 1: Prepare GCP and Create Data Warehouse Connection
In this step, you:
* create a service account in GCP that Confidence impersonates
* create a dataset for storing the tables that Confidence generates
* configure a metrics data warehouse connection
### Step 1a: Create Service Account
For Confidence to access BigQuery, you need to create a service account that
has permissions to access BigQuery. You also need to allow Confidence to
impersonate the service account. Follow these steps:
Go to the Google Cloud Platform Console and select the project you want to use for Confidence.
Go to IAM & Admin > Service Accounts.
Enter a name for the service account, for example, `confidence`. Make note of the service account email address. You can click copy-to-clipboard icon to copy it. Click **CREATE AND CONTINUE**.
Grant the following roles to the account:
* **BigQuery Job User**
* **BigQuery Data Viewer**
Click **CONTINUE**.
#### Grant Confidence Access to Impersonate the Service Account
Confidence needs to be able to impersonate the service account you just created
to be able to access BigQuery. You do this by granting one of Confidence's own
service accounts the "Workload Identity User" role, called the principal account:
```text theme={null}
account-@spotify-confidence.iam.gserviceaccount.com
```
To grant the principal account access to impersonate the service account, follow
these steps:
Click on the service account you just created to go to the service account details page.
Enter `account-@spotify-confidence.iam.gserviceaccount.com` as principal.
### Step 1b: Create BigQuery Datasets to Store Metrics and Exposure
Confidence needs a dataset in BigQuery where it can store intermediate metrics
and exposure calculations. For Confidence to access the dataset,
you need to grant the principal service account access to the dataset.
```text theme={null}
account-@spotify-confidence.iam.gserviceaccount.com
```
To create the dataset, follow these steps:
Go to the Google Cloud Platform Console and select the project you want to use for Confidence.
In BigQuery, click **⋮ > CREATE DATASET** next to the name of the project you want to create it in.
Enter a name for the dataset. For example, `confidence_metrics`.
Select the location where you want to store the dataset.
The dataset location must be the same as the region of your Confidence account. The recommendation is to select a multi-region location for your dataset, like `EU` or `US`.
Enter the email address of the `confidence` service account you created and assign the role **BigQuery Data Owner**.
### Step 1c: Configure a Metrics Data Warehouse
With the completion of steps 1a and 1b, you have now created a service account and
a dataset in BigQuery. You can now configure Confidence to use BigQuery to
compute metrics. To do so, follow these steps:
On the bottom of the left sidebar, select **Admin > Connections > Metrics Data Warehouse**.
Enter the GCP Project ID, your metrics dataset (`confidence_metrics`) and the email address of the `confidence` service account you created in step 1a.
When you click save, Confidence tries to connect to BigQuery using the
service account you created in step 1a. If you have misconfigured anything,
you should see an error message.
**🎉 Well done! Now you've configured Confidence to compute metrics using BigQuery.**
If you plan to use Confidence Flags, you also need to configure Confidence to
store assignment data in BigQuery. To do so, follow the next steps outlined
below.
## Step 2: Configure Confidence to Write and Read Assignment Data
In this step, you:
* create a dataset in BigQuery that Confidence can write assignment data to
* configure a flag applied connector so that Confidence is able to write assignment data
* configure an assignment table so that Confidence is able to read assignment data
### Step 2a: Create BigQuery Datasets to Store Assignment Data
If you want to use Confidence Flags to run experiments, you need to complete
a few more steps to configure Confidence to store assignment data in BigQuery
and an assignment table that reads from the BigQuery table.
Confidence needs a dataset in BigQuery where it can store assignment data. For
Confidence to access the dataset, you need to grant the principal
service account access to the dataset.
```text theme={null}
account-@spotify-confidence.iam.gserviceaccount.com
```
To create the dataset, follow these steps:
Go to the Google Cloud Platform Console and select the project you want to use for Confidence.
Go to BigQuery and click **⋮ > CREATE DATASET** next to the name of the project you want to create it in.
Enter a name for the dataset, for example, `confidence_flag_applied`.
Select the location where you want to store the dataset.
The dataset location must be the same as the region of your Confidence account. The recommendation is to select a multi-region location for your dataset, like `EU` or `US`.
Enter the email address of the `confidence` service account you created and add the role **BigQuery Data Owner**.
### Step 2b: Configure a Flag Applied Connector
For Confidence to be able to store assignment data in BigQuery, you need to set
up a connector between Confidence and BigQuery.
Assignment data is information on which users were assigned to which variants
in the experiments you run. Assignment data goes into exposure calculations.
Metrics use exposure to calculate results in your tests.
This connector is a **"Flag Applied"** connector.
The connector is the part responsible for writing assignment data to BigQuery that Confidence Metrics
can later read.
On the bottom of the left sidebar, select **Admin > Connections > Flag applied connections**.
Enter the Project ID, the dataset (for example, `confidence_flag_applied`) you created in step 2a and the email address of the `confidence` service account you created in step 1a.
Enter the name of the table (for example, `flag_applied`) to write assignment data to. Confidence creates the table with the right schema for you.
When you click save, Confidence tries to connect to BigQuery using the
service account you created in step 1a. If you have misconfigured anything,
you should see an error message.
### Step 2c: Configure an Assignment Table
For Confidence to use the stored assignment table, you need to set
up an assignment table that reads from the BigQuery table. You first need to create
an entity, which represents the thing you're experimenting on, like your users.
To do so, follow these steps:
On the bottom of the left sidebar, select **Admin > Connections > Flag applied connections** and then the BigQuery connection you created.
Enter a name for the assignment table, such as `flag_applied`. This name should typically match the name you used in step 2b. Confidence can then read assignments from the destination table of your flag assignments.
Create a new entity or select an existing entity. Entities are the things you're experimenting on, like your users. Enter `User` and specify the data type of the identifier that identifies the entity. For example, if you have a UUID that identifies your users, your primary key type is a **String**.
**🎉 Well done! You are all set up and ready to go.**
## Step 3: Grant View Access to Metrics Data Sources
Confidence uses fact tables to describe the data that you create metrics from.
To ensure that Confidence can access data from the data sources you use for fact tables, you
need to grant the role **BigQuery Data Viewer** to the service account named `confidence` that you have created.
Grant this role for the datasets and tables you want Confidence to have access to.
Go to the Google Cloud Platform Console and select the project that the data you want to grant Confidence access to is in.
Go to BigQuery and select the dataset. Optionally select the table.
If you selected a dataset, click **SHARING > Manage Permissions**. If you selected a table, click **SHARE**.
Enter the email address of the `confidence` service account you created and assign the role **BigQuery Data Viewer**.
## What's Next?
The next step is to [create a fact table](/docs/metrics/fact-tables), and a
[metric](/docs/metrics/introduction). For an overview, see the [metric introduction](/docs/metrics/introduction)
page, and the [metrics quickstart](/docs/quickstarts/configure-metric).
## Related Resources
Overview of metrics in Confidence
Configure fact tables for metrics
Set up assignment tracking
Create your first metric
# Get Started with Databricks
Source: https://confidence-auth-testing.mintlify.io/docs/warehouse-setup/databricks
Configure read and write connections to Databricks.
You can run this setup with an AI assistant instead. Run this skill and your assistant guides you through the steps:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins --skill setup-warehouse-databricks
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants.
This tutorial helps you configure Confidence to:
1. Run queries in Databricks to compute exposure and metrics.
2. Store assignment data as Parquet files in S3, and then load them into Databricks.
Step 2 is optional if you already have assignment data in Databricks. For
example, if you are using a feature flagging solution other than Confidence
Flags.
This document targets the following audiences:
* Administrators who want to set up Confidence for their organization
## Before You Begin
* You need to have a [Confidence](https://spotify.com) account.
* You need to have an AWS account.
* You need to have permissions to create S3 buckets, IAM users and roles, and manage the Databricks cluster.
## Step 1: Create an S3 Bucket
To load assignment data, Confidence first copies Parquet files to an S3 bucket, and then triggers load jobs to
copy these into Databricks.
* Go to the S3 console, click **Create bucket**.
* Give it a name, and put it in the same AWS region as you have your Databricks instance in.
## Step 2: Create the Confidence IAM Role
Now you need to create an IAM role that Confidence can assume with the correct permissions.
Two options for authentication are available. Either Confidence can use a regular AWS access key and secret to authenticate as an
IAM User and then assume the role, or it can use [AssumeRoleWithWebIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html)
to authenticate without having to store any credentials, by using a Google service account as the trusted entity.
`AssumeRoleWithWebIdentity` is usually preferable, but sometimes it might interfere with other settings such as custom identity providers.
In those cases, you may need the credentials-based approach.
Do either step 2a or 2b depending on what approach you choose.
### Step 2a: Set up the Trust Policy for AssumeRoleWithWebIdentity
* Go to the IAM console, click **Roles** and **Create role**
* Select "Custom trust policy" as the trusted entity type.
* In the text field, paste the following JSON snippet, replacing `` with the unique service account ID you are using to authenticate from the Confidence side. You can find the ID in the `Your Service Account ID` box that is part of the configure flag applied connector form for Databricks.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "accounts.google.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"accounts.google.com:sub": ""
}
}
}
]
}
```
* Click **Next**, and don't select any of the predefined permissions. Confidence adds its own inline policy that is more restrictive than the built-in policies.
* Input a name for the role, for example, `confidence-role`, and then click **Create role**.
### Step 2b: Set up the Trust Policy with an IAM User
* Go to the IAM console, click **Users** and **Create user**
* Give the user a name and create it.
* Go to the user details and generate an access key and secret for the user. Keep the access key and secret for later when you configure the warehouse in Confidence.
* Go to the IAM console, click **Roles** and **Create role**
* Select "Custom trust policy" as the trusted entity type.
* In the text field, paste the following JSON snippet, replacing `` with the ARN of the user you created in step 2 above (there is a button to copy the ARN on the user page).
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": ""
},
"Action": "sts:AssumeRole"
}
]
}
```
### Step 2c: Set Up the IAM Role Policy
* Find the role you created earlier and click it, then click the **Add permission** dropdown list and then **Create inline policy**
* Switch the policy editor to JSON, and then paste the following snippet, replacing `` placeholders with the name of the bucket you created.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:ListBucket",
"Effect": "Allow",
"Resource": "arn:aws:s3:::"
},
{
"Action": [
"s3:PutObjectAcl",
"s3:PutObject",
"s3:GetObjectAcl",
"s3:GetObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::/*"
}
]
}
```
* Give the policy a name, click **Next** and **Create policy** to attach it to the role.
## Step 3: Create Schemas for Confidence Data
Confidence needs to have a schema to write the results of exposure and metric calculations. These could either be separate schemas or the same, for simplicity
you just create one schema for everything here.
* Open a SQL notebook and run the following SQL to create the schema:
```sql theme={null}
CREATE SCHEMA confidence;
```
## Step 4: Create Service Principal
* Go to the Databricks Identity and access settings and then **Service principals**.
* Add a new service principal, name it what you like.
* Generate an OAuth Client ID and secret for the service principal following the instructions from the [Databricks
docs](https://docs.databricks.com/en/dev-tools/service-principals.html#step-4-generate-a-databricks-personal-access-token-for-the-databricks-service-principal).
Then set up the permissions for the service principal to have write access to the schema you created in the earlier step, and read access to any tables that contain
metric data you want to use for experimentation.
## Step 5a: Configure a Metrics Data Warehouse
1. Go to the Confidence App.
2. On the bottom of the left sidebar, select **Admin > Connections > Metrics Data Warehouse**.
3. Select **Databricks** and configure the required settings.
4. Click **Save**.
## Step 5b: Configure a Flag Applied Connector
For Confidence to be able to store assignment data in Databricks, you need to set
up a connector between Confidence and Databricks.
Assignment data is information on which users were assigned to which variants
in the experiments you run. Assignment data goes into exposure calculations.
Metrics use exposure to calculate results in your tests.
This connector is a **"Flag Applied"** connector.
The connector is the part responsible for writing assignments to Databricks that Confidence Metrics
can later read.
1. Go to the Confidence App.
2. On the bottom of the left sidebar, select **Admin > Connections > Flag Applied**.
3. Click **Create**
4. Select **Databricks** as destination.
5. Enter the details from the earlier setup steps.
6. Click **Save**.
When you click save or have entered the required details, Confidence tries to connect to Databricks and load some sample data.
If you have misconfigured anything, you see an error message.
## Step 5c: Configure an Assignment Table
For Confidence to use the stored assignment table, you need to set
up an assignment table that reads from the Databricks table. You first need to create
an entity, which represents the thing you're experimenting on, like your users.
To do so, follow these steps:
On the bottom of the left sidebar, select **Admin > Connections > Flag Applied** and select the Databricks connection you created.
Create a new entity or select an existing entity. Entities are the things you're experimenting on, like your users. Enter `User` and specify the data type of the identifier that identifies the entity. For example, if you have a UUID that identifies your users, your primary key type is a **String**.
Enter a name for the assignment table, such as `flag_applied`. This name should typically match the name you used in step 5b. Confidence can then read assignments from the destination table of your flag assignments.
You are all set up and ready to go.
## What's Next?
The next step is to [create a fact table](/docs/metrics/fact-tables), and a
[metric](/docs/metrics/introduction). For an overview, see the [metric introduction](/docs/metrics/introduction)
page, and the [metrics quickstart](/docs/quickstarts/configure-metric).
## Related Resources
Overview of metrics in Confidence
Configure fact tables for metrics
Set up assignment tracking
Create your first metric
# Get Started with Redshift
Source: https://confidence-auth-testing.mintlify.io/docs/warehouse-setup/redshift
Configure read and write connections to Redshift.
You can run this setup with an AI assistant instead. Run this skill and your assistant guides you through the steps:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins --skill setup-warehouse-redshift
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants.
This tutorial helps you configure Confidence to:
1. Run queries in Redshift to compute exposure and metrics.
2. Store assignment data as Parquet files in S3, and then load them into Redshift.
Step two (2) is optional if you already have assignment data in Redshift. For
example, if you are using a feature flagging solution other than Confidence
Flags.
This document targets the following audiences:
* Administrators who want to set up Confidence for their organization
## Before You Begin
* You need to have a [Confidence](https://spotify.com) account.
* You need to have an AWS account.
* You need to have permissions to create S3 buckets, IAM users and roles, and manage Redshift instances.
## Terraform Scripts
Terraform scripts are available to create the S3 bucket, IAM user and role described in the steps below.
If you are using Terraform, then after you have applied the Terraform Configuration, you can skip directly to step [Step 3](#step-3-associate-the-iam-role-with-the-redshift-cluster).
## Step 1: Create an S3 Bucket
To load assignment data, Confidence first copies Parquet files to an S3 bucket, and then triggers load jobs to
copy these into Redshift.
* Go to the S3 console, click **Create bucket**.
* Give it a name, and put it in the same AWS region as you have your Redshift instance in.
## Step 2: Create the Confidence IAM role
Now you need to create an IAM role that Confidence can assume with the correct permissions.
Two options for authentication are available. Either Confidence can use a regular AWS access key and secret to authenticate as an
IAM User and then assume the role, or it can use [AssumeRoleWithWebIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html)
to authenticate without having to store any credentials, by using a Google service account as the trusted entity.
`AssumeRoleWithWebIdentity` is usually preferable, but sometimes it might interfere with other settings such as custom identity providers.
In those cases, you may need the credentials-based approach.
Do either step 2a or 2b depending on what approach you choose.
### Step 2a: Setup the Trust policy for AssumeRoleWithWebIdentity
* Go to the IAM console, click **Roles** and **Create role**
* Select "Custom trust policy" as the trusted entity type.
* In the text field, paste the following JSON snippet, replacing `` with the unique service account ID you are using to authenticate from the Confidence side. You can find the ID in the `Your Service Account ID` box that is part of the configure data warehouse form for Redshift.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"redshift.amazonaws.com",
"redshift-serverless.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"Federated": "accounts.google.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"accounts.google.com:sub": ""
}
}
}
]
}
```
* Click next, and don't select any of the predefined permissions. Confidence adds its own inline policy that is more restrictive than the built-in policies.
* Input a name for the role, for example, `confidence-role`, and then click **Create role**.
### Step 2b: Setup the Trust Policy with an IAM User
* Go to the IAM console, click **Users** and **Create user**
* Give the user a name and create it.
* Go to the user details and generate an access key and secret for the user. Keep the access key and secret for later when you configure the warehouse in Confidence.
* Go to the IAM console, click **Roles** and **Create role**
* Select "Custom trust policy" as the "Trusted entity" type.
* In the text field, paste the following JSON snippet, replacing `` with the ARN of the user you created in step 2 above (there is a button to copy the ARN on the user page).
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": [
"redshift-serverless.amazonaws.com",
"redshift.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": ""
},
"Action": "sts:AssumeRole"
}
]
}
```
### Step 2c: Setup the IAM Role Policy
* Find the role you created earlier and select it, then click the **Add permission** dropdown list and then **Create inline policy**
* Switch the policy editor to JSON, and then paste the following snippet, replacing the placeholders with the name of the S3 bucket you created, your AWS region, AWS account ID, and the Redshift cluster name.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:ListBucket",
"Effect": "Allow",
"Resource": "arn:aws:s3:::"
},
{
"Action": [
"s3:PutObjectAcl",
"s3:PutObject",
"s3:GetObjectAcl",
"s3:GetObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::/*"
},
{
"Action": [
"redshift-data:ListStatements",
"redshift-data:GetStatementResult",
"redshift-data:DescribeStatement",
"redshift-data:CancelStatement"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": [
"redshift:GetClusterCredentialsWithIAM",
"redshift:GetClusterCredentials",
"redshift-data:ListTables",
"redshift-data:ListSchemas",
"redshift-data:ListDatabases",
"redshift-data:ExecuteStatement",
"redshift-data:DescribeTable",
"redshift-data:BatchExecuteStatement"
],
"Effect": "Allow",
"Resource": [
"arn:aws:redshift:::dbname:/*",
"arn:aws:redshift:::cluster:"
]
}
]
}
```
* Give the policy a name, click **Next** and **Create policy** to attach it to the role.
## Step 3: Associate the IAM Role with the Redshift cluster
Now you need to give the Confidence IAM role permissions to run load jobs.
* Go to the Redshift cluster you want to use in the AWS console.
* Go to the **Properties** tab, scroll down to the **Cluster permissions**, click **Manage IAM roles** and then **Associate IAM role**.
* In the dialog that comes up, the role you created in the earlier step should show up. Select the role and click **Associate IAM roles**.
## Step 4: Create the Redshift Database and Schema
* Open the query editor for the Redshift cluster you are using.
* Create a Confidence database to keep the data separate from the rest of your data. Then switch the query editor to use that database.
```sql theme={null}
CREATE database confidence;
```
* Create a schema to contain the tables. Then create a user corresponding to the IAM role and grant it access to write to this schema. You can copy the SQL with the information pre-filled by clicking the `Copy SQL` button.
```sql theme={null}
CREATE SCHEMA confidence;
CREATE USER "IAMR:" PASSWORD DISABLE;
GRANT ALL ON SCHEMA confidence TO "IAMR:";
GRANT ALL ON ALL TABLES IN SCHEMA confidence TO "IAMR:";
ALTER DEFAULT PRIVILEGES FOR USER "IAMR:" IN SCHEMA confidence GRANT ALL ON TABLES TO "IAMR:";
```
The role name here is not the ARN, just the name (`role-name` rather than
`arn:aws:iam::191394936087:role/role-name`).
The CREATE USER command may fail with "user already exists" if someone has
logged in with that user, if so ignore the error and continue.
## Step 5a: Configure a Metrics Data Warehouse
1. Go to the Confidence App.
2. On the bottom of the left sidebar, select **Admin > Connections > Metrics Data Warehouse**.
3. Select **Redshift**.
4. Enter the details from the earlier setup steps.
5. Click **Save**.
## Step 5b: Configure a Flag Applied Connector
For Confidence to be able to store assignment data in Redshift, you need to set
up a connector between Confidence and Redshift.
Assignment data is information on which users were assigned to which variants
in the experiments you run. Assignment data goes into exposure calculations.
Metrics use exposure to calculate results in your tests.
This connector is a **"Flag Applied"** connector.
The connector is the part responsible for writing assignments to Redshift that Confidence Metrics
can later read.
To set it up:
1. Go to the Confidence App.
2. On the bottom of the left sidebar, select **Admin > Connections > Flag Applied**.
3. Click **Create**
4. Select **Redshift** as destination.
5. Enter the details from the earlier setup steps.
6. Click **Save**.
When you click save or have entered the required details, Confidence tries to connect to Redshift and load some sample data.
If you have mis-configured anything, you see an error message.
## Step 5c: Configure an Assignment Table
For Confidence to use the stored assignment table, you need to set
up an assignment table that reads from the Redshift table. You first need to create
an entity, which represents the thing you're experimenting on, like your users.
To do so, follow these steps:
On the bottom of the left sidebar, select **Admin > Connections > Flag Applied** and select the Redshift connection you created.
Create a new entity or select an existing entity. Entities are the things you're experimenting on, like your users. Enter `User` and specify the data type of the identifier that identifies the entity. For example, if you have a UUID that identifies your users, your primary key type is a **String**.
Enter a name for the assignment table, such as `flag_applied`. This name should typically match the name you used in step 5b. Confidence can then read assignments from the destination table of your flag assignments.
**🎉 Well done! You are all set up and ready to go.**
## What's Next?
The next step is to [create a fact table](/docs/metrics/fact-tables), and a
[metric](/docs/metrics/introduction). For an overview, see the [metric introduction](/docs/metrics/introduction)
page, and the [metrics quickstart](/docs/quickstarts/configure-metric).
## Related Resources
Overview of metrics in Confidence
Configure fact tables for metrics
Set up assignment tracking
Create your first metric
# Get Started with Snowflake
Source: https://confidence-auth-testing.mintlify.io/docs/warehouse-setup/snowflake
Configure read and write connections to Snowflake.
You can run this setup with an AI assistant instead. Run this skill and your assistant guides you through the steps:
```bash theme={null}
npx skills add spotify/confidence-ai-plugins --skill setup-warehouse-snowflake
```
The installer works with Claude Code, Cursor, Codex, Gemini CLI, and other AI assistants.
This tutorial helps you configure Confidence to talk to Snowflake to store
assignment data and calculate metrics. The steps to take are the following:
1. Create a Snowflake user and role that Confidence can use to read and write data to your warehouse.
2. Grant the role permissions to use a Snowflake warehouse.
3. Set up a schema and/or a database to store assignment data coming out of Confidence Flags.
4. Set up a schema and/or a database to store exposure calculation data.
5. Generate a key pair in Confidence, and assign the public key to your Snowflake user.
6. Configure a Data Warehouse connection so that Confidence can compute metrics
using Snowflake.
7. Configure a Flag Applied connector so that Confidence can store
assignment data in Snowflake.
If you already followed the setup guide on the **Admin** page, your
setup is already complete. You can skip directly to [what's next](#whats-next).
## Before You Begin
* This tutorial assumes you have a Confidence account.
* This tutorial assumes you have a Snowflake account ready, and have the permissions needed to
create users, roles, databases and schemas for that account. You should also have a Snowflake warehouse
available to run Confidence queries.
## Step 1. Create a Snowflake User and/or Role
As a first step, you create a Snowflake user, and grant a role to that user. You can either use a pre-existing role
that you already have, or create a new role just for the Confidence integration.
```sql theme={null}
CREATE ROLE CONFIDENCE_ROLE;
CREATE USER CONFIDENCE DEFAULT_ROLE = CONFIDENCE_ROLE;
GRANT ROLE CONFIDENCE_ROLE TO USER CONFIDENCE;
```
## Step 2. Give the Role Permissions to Run Queries in a Warehouse
Confidence needs to run queries to calculate metrics, so for this you need to grant the role you created permissions
to use the warehouse. You can either [create a new warehouse](https://docs.snowflake.com/en/sql-reference/sql/create-warehouse) only for Confidence, or use an existing warehouse.
```sql theme={null}
GRANT USAGE ON WAREHOUSE TO ROLE CONFIDENCE_ROLE;
```
## Step 3. Create Database and Schemas for Assignment Data
Confidence needs two schemas in Snowflake: one to store assignment data and one to store internal data for the exposure calculations.
In the following example, you create a separate database to hold Confidence data. To create the schemas, follow these steps (or change according to your organization guidelines):
```sql theme={null}
CREATE DATABASE CONFIDENCE;
GRANT USAGE ON DATABASE CONFIDENCE TO ROLE CONFIDENCE_ROLE;
CREATE SCHEMA CONFIDENCE.FLAG_APPLIED;
GRANT USAGE ON SCHEMA CONFIDENCE.FLAG_APPLIED TO ROLE CONFIDENCE_ROLE;
GRANT CREATE TABLE ON SCHEMA CONFIDENCE.FLAG_APPLIED TO ROLE CONFIDENCE_ROLE;
GRANT ALL ON ALL TABLES IN SCHEMA CONFIDENCE.FLAG_APPLIED TO ROLE CONFIDENCE_ROLE;
GRANT ALL ON FUTURE TABLES IN SCHEMA CONFIDENCE.FLAG_APPLIED TO ROLE CONFIDENCE_ROLE;
```
The role also needs **read-only** access to any tables you want to use to define metrics (the tables you use to define fact table queries).
## Step 4. Create Schemas for Exposure Calculations
Repeat the preceding steps to create a second schema. For example, `EXPOSURE`.
## Step 5. Generate a Key in Confidence, and Assign the Public Key to Your Snowflake User
1. Go to the Confidence App.
2. On the bottom of the left sidebar, select **Admin > Connections > Metrics Data Warehouse**.
3. Select **Snowflake**.
4. Click the **Crypto key** combo box, and generate a new key that you can call whatever you want. Then click **Copy public key**
to the right of the combo box to copy the public key to your clipboard.
5. Run the following SQL command in Snowflake to let Confidence authenticate as the user you created using the public key:
```sql theme={null}
ALTER USER SET rsa_public_key='';
```
You are now ready to configure the rest of the settings on the warehouse connection.
## Step 6. Configure a Metrics Data Warehouse
For Confidence to be able to compute metrics using Snowflake, you need to set up
the connection between Confidence and Snowflake. To do so, follow these steps:
1. Go to the Confidence App.
2. On the bottom of the left sidebar, select **Admin > Connections > Metrics Data Warehouse**.
3. Select **Snowflake**.
4. Select the Crypto key you created in the earlier step, and complete the rest of the configuration form. After configuring
the account identifier, user, and the crypto key, Confidence reads the available roles, databases and schemas from Snowflake and
populates the remaining options.
5. Click **SAVE**.
If you have misconfigured anything, you see an error message. Otherwise,
you should see a success message.
## Step 7. Configure a Flag Applied Connector
For Confidence to be able to store assignment data in Snowflake, you need to set
up the connection between Confidence and Snowflake. To do so, follow these steps:
1. Go to the Confidence App.
2. On the bottom of the left sidebar, select **Admin > Connections > Flag Applied**.
3. Click **Create**.
4. Select **Snowflake** as destination.
5. Select the Crypto key you created in the earlier step, and complete the rest of the configuration form. After configuring
the account identifier, user, and the crypto key, Confidence reads the available roles, databases and schemas from Snowflake and
populates the remaining options.
6. Click **Save**.
## Step 8. Configure an Assignment Table
For Confidence to be able to use the stored assignment table, you need to set
up an assignment that reads from the Snowflake table. You first need to create
an entity, which represents the thing you're experimenting on, like your users.
To do so, follow these steps:
On the left sidebar, select **Admin > Entities**.
Enter `User` and specify the data type of the identifier that identifies the entity. For example, if you have a UUID that identifies your users, your primary key type is a **String**.
Now that you have an entity, you can create an assignment table that reads from
the Snowflake table. To do so, follow these steps:
On the bottom of the left sidebar, select **Admin > Connections > Flag Applied** and select the Snowflake connection you created.
Enter a name for the assignment table, such as `flag_applied`.
## Step 9. Celebrate
That's it! You have now configured Confidence to store assignment data and compute metrics using
Snowflake. You can now start defining metrics and compute results.
## What's Next?
The next step is to [create a fact table](/docs/metrics/fact-tables), and a
[metric](/docs/metrics/introduction). For an overview, see the [metric introduction](/docs/metrics/introduction)
page, and the [metrics quickstart](/docs/quickstarts/configure-metric).
## Related Resources
Overview of metrics in Confidence
Configure fact tables for metrics
Set up assignment tracking
Create your first metric