A/B Testing

What is A/B Testing?

A/B testing lets you route a percentage of live traffic to a test version while the rest continues to the current live version. Both versions produce execution logs, so you can compare their performance with real data before fully switching.

Prerequisites

Before starting an A/B test, you need:

A policy group with a currently deployed (live) version
A test version that has been published (ACTIVE status)
Both versions must belong to the same policy group

Traffic Identity Key

A/B routing splits traffic by a single value your application sends with each execution request: context.trafficKey. The engine hashes that value together with the policy group ID and assigns the request to a bucket, so the same key value always lands on the same version for that group. Send it in context, not in facts. Routing reads the context object only. A trafficKey placed in facts is ignored.

{
  "facts": {
    "paymentAmount": 150000,
    "customerTier": "VIP"
  },
  "context": {
    "trafficKey": "customer-8f3a21"
  }
}

If trafficKey is missing or empty, no split happens and the test version receives 0% of traffic. The test still shows as running in the console and execution logs keep accumulating, so nothing about the response reveals the problem. The server logs a warning on every such request. Choosing a value:

Stable per experiment unit. A customer ID, a merchant ID, a session ID, whatever you want the experiment measured on
Opaque. The whole context object is stored in the execution ledger, so send a hash or a surrogate key rather than an email address or a phone number
Never a value that changes every request. A request ID or a timestamp scatters one user across both versions and destroys the comparison

Bucketing is per group. The group ID is mixed into the hash, so the same trafficKey value gets an independent assignment in every policy group and one experiment never skews another.

Where the key goes

Endpoint	Where routing reads the key
`POST /execution/groups/{groupId}`	`context.trafficKey`
`POST /execution/composite`	`context.trafficKey`. Each group buckets independently
`POST /execution/groups/{groupId}/batch`	`sharedContext.trafficKey`. The whole batch runs on one version, and a per-request `context.trafficKey` does not change that
`POST /execution/groups/{groupId}/versions/{versionId}`	Not read. This endpoint bypasses routing by design

GET /execution/groups/{groupId}/requirements returns an exampleRequest whose context does not include trafficKey. Copying that template as-is leaves an A/B test at 0%.

Starting an A/B Test

Console

Navigate to the policy group → click Start A/B Test → select the test version → set the traffic percentage.

CLI

lexq groups ab-test start \
  --group-id <groupId> \
  --version-id <testVersionId> \
  --traffic-rate 10

The --traffic-rate is the percentage (1–99) of traffic routed to the test version. The remaining traffic stays on the live version.

Monitoring Results

While the A/B test is running, both versions generate execution logs. Compare them using:

lexq history stats --group-id <groupId>

Check success rates, match rates, latency, and output variable distributions across both versions.

Adjusting Traffic

Gradually increase the test version’s traffic as you gain confidence:

lexq groups ab-test adjust --group-id <groupId> --traffic-rate 30
lexq groups ab-test adjust --group-id <groupId> --traffic-rate 50

Stopping an A/B Test

lexq groups ab-test stop --group-id <groupId>
lexq groups ab-test stop --group-id <groupId> --force

Stopping reverts all traffic to the live version. The test version remains ACTIVE but no longer receives traffic.

Typical Workflow

Create a new DRAFT version with rule changes
Dry-run to validate the changes
Publish the new version (DRAFT → ACTIVE)
Confirm your execution requests send context.trafficKey
Start A/B test at 10% traffic
Monitor execution logs and compare metrics
Gradually increase: 10% → 30% → 50%
If satisfied, deploy the test version as the new live version
Stop the A/B test

A/B tests affect production traffic. Start with a low traffic percentage (5–10%) and monitor closely before scaling up.

Getting Started

Core Concepts

Features

What is A/B Testing?

Prerequisites

Traffic Identity Key

Where the key goes

Starting an A/B Test

Console

CLI

Monitoring Results

Adjusting Traffic

Stopping an A/B Test

Typical Workflow

Next Steps

Impact Simulation

Deployments

​What is A/B Testing?

​Prerequisites

​Traffic Identity Key

​Where the key goes

​Starting an A/B Test

​Console

​CLI

​Monitoring Results

​Adjusting Traffic

​Stopping an A/B Test

​Typical Workflow

​Next Steps

Impact Simulation

Deployments

What is A/B Testing?

Prerequisites

Traffic Identity Key

Where the key goes

Starting an A/B Test

Console

CLI

Monitoring Results

Adjusting Traffic

Stopping an A/B Test

Typical Workflow

Next Steps