Understanding Sessions, Variant Persistence, and Reporting in A/B Tests

A/B test results depend on how sessions are defined, how visitors are assigned to variants, and how data is processed behind the scenes. This article explains how these pieces work together, so you can better understand how visitors are counted, why they continue to see the same experience, and how results are calculated.

1. What is a Session?

A session represents a single visit to your store.

• A session starts when a visitor lands on your site
• It ends after 30 minutes of inactivity
• Returning after that creates a new session

👉 We track this using a session cookie.

Important:

• The same visitor can have multiple sessions
• A new session does not mean a new user

2. Variant Persistence

Once a visitor is assigned to a variant:

• It is stored in local storage
• It persists indefinitely (unless cleared)

👉 This means:

• Users continue to see the same variant across sessions
• Sessions do not affect variant assignment

3. Test Types and Retention Behavior

Theme Tests

Theme tests rely on Shopify’s theme system to determine which version of the store a visitor sees.

• Shopify maintains its own mechanism for keeping a visitor on a theme
• This can result in long-lived persistence (often up to ~1 year)

👉 This behavior comes from Shopify, not Shogun

Why this matters:

• Visitors may stay on the same theme for a long time
• This persistence exists independently of session timing

All Other Tests

Other tests modify the live theme directly.

These use:

• Session cookie → defines sessions (30 min inactivity)
• Local storage → persists variant assignment

Important:

• There is no long-term Shopify-controlled persistence here
• Variant consistency comes from local storage

4. Why Cookies Still Matter

Even though variant assignment persists, cookies are still essential.

Cookies are used to:

• Track sessions
• Group user activity within a visit
• Support reporting accuracy

Without session cookies:

• Visits could not be grouped into sessions
• Reporting would become inconsistent

➕ 5. How Reporting Updates (Advanced)

This section explains how results are processed behind the scenes. It is separate from sessions, cookies, and variant behavior.

Important:

This should not be confused with session length or variant persistence.

How data is calculated

We process test results continuously to keep reporting as up to date as possible.

• Data is recalculated every minute
• Metrics are stored at an hourly level
• Each update recalculates:
  • The current hour
  • The previous hour

Why we recalculate recent data

We revisit recent data because:

• A session is only considered complete after 30 minutes of inactivity
• Orders may occur after a session starts or across hour boundaries

Reprocessing ensures:

• Sessions are counted correctly
• Orders are attributed accurately

What the Lookback Period Means (in reporting)

When recalculating data, we don’t just look at the current hour.

We also look slightly before and after the time window to capture related activity.

👉 This is the lookback period (1.5 hours)

It allows us to:

• Connect sessions with later conversions
• Capture events that span across time boundaries

Important:

• Reporting is continuously updated, not delayed until a session ends
• The lookback period here relates to data processing, not user sessions

🔑 Key Takeaways

• A session ends after 30 minutes of inactivity
• Users stay in the same variant due to local storage
• Theme tests rely on Shopify’s persistence behavior
• Other tests rely on session tracking + local storage
• Reporting is continuously updated in the background