Test Concurrency & Traffic Distribution

Shogun supports multiple types of experiments.

Since these tests can run simultaneously, Shogun must decide which test a shopper (visitor) participates in when multiple tests apply to the same visit.

This is controlled by the traffic distribution algorithm: Greedy or Even.

Single Test Participation Rule

A shopper can only be part of one test at a time.

If multiple tests could apply (e.g., a Theme Test and a Page Test on the same page), Shogun must determine which one to assign.

🧭 Algorithm 1: Greedy

Greedy means that the shopper is assigned to the first eligible test they encounter.

How it works

When a shopper visits the store, Shogun checks which tests are currently active on the page. If only one test is running — for example, a Theme Test — the shopper is automatically assigned to that test.

If multiple tests are active, such as a Theme Test and a Home Page Test, Shogun follows a “first seen” rule. The first test the shopper encounters captures them. Once that assignment happens, the shopper won’t be added to any other test later, even if they qualify for it.

This approach mimics how users naturally move through a store — whichever experiment they hit first is the one they end up participating in.

Example

Scenario: A Theme Test (running site-wide) and a Home Page Test (specific to the homepage).

When a shopper lands on the homepage:

1. Both tests are active.
2. Shogun identifies that the shopper is eligible for both.
3. The shopper is randomly assigned to one of the two tests at a 50/50 ratio.
4. After assignment, they will not enter the other test later.

Notes

• When multiple tests apply, random assignment ignores each test’s internal variation split.
• For example, a Page Test might have a 70/30 split and a Theme Test might have a 50/50 split, but between the two tests, the split remains 50/50.
• If a shopper first lands on a page not under test (for example, a blog page) while a Theme Test is running, they’ll be placed into the Theme Test since it’s the first applicable test.
• A shopper can only participate in one test at a time. Once assigned, they will not be reassigned elsewhere.

⚖️ Algorithm 2: Even

Even means that shoppers are assigned to tests in equal proportions, regardless of “first seen” logic.

How it works

In Even mode, each shopper is deterministically assigned to a test using their session ID. This ensures that all tests are equally likely to receive traffic, regardless of where the shopper enters the funnel.

Once the shopper’s test is determined, their assigned variant within that test is selected according to the configured traffic split (for example, 70/30). This approach guarantees a consistent, balanced distribution across all active tests, preventing top-of-funnel pages from dominating.

Example

Scenario: A Theme Test (site-wide), a Home Page Test, and a Product Page Test are all running.

When a shopper lands on the homepage:

1. Shogun checks the shopper’s pre-assignment.
2. Even though the homepage test appears first, the shopper will not automatically enter it.
3. If the shopper was pre-allocated to the Product Page Test, they will skip the homepage test.
4. When they later visit a product page, they will enter their assigned Product Page Test as expected.

Notes

• Even distribution ensures all tests receive balanced, predictable traffic across the funnel.
• Pre-assignment occurs before the shopper engages with any page.
• Once assigned to a test, the shopper will only see variants from that test and will not enter others.
• Internal variant splits (for example, a 70/30 split within a test) are still respected after assignment.

🧠 Key Takeaways

• Shogun always enforces one active test per shopper.
• Greedy prioritizes the first encountered test, mimicking real-world browsing order.
• Even prioritizes balanced traffic distribution across all running tests.
• When a Theme Test overlaps with Page/Template Tests, Shogun randomizes the assignment between them to avoid double exposure.

Cache Management

Shogun stores a visitor’s test variant assignment in local storage, which ensures they see the same version of a test each time they return.

Key details:

• For theme and template tests, dynamic scripts ensure the correct variant is served regardless of Shopify’s default caching.
• Shogun’s metafield and optimizer configuration on Shopify is refreshed approximately every 12 hours to ensure shops are running the latest experiment setup and optimizer code.

Audience Targeting in Concurrent Tests

Each test evaluates audience conditions independently, meaning a visitor might qualify for one test but not another.

Targeting rules include:

• Device type (e.g., mobile, desktop)
• Visitor status (new vs. returning)
• Geographic location
• Login status
• URL pattern matching
• Referrer sources (e.g., campaigns, social)