Starting Your First Test

Starting Your First Test

Launching your first A/B test is designed to be simple, even if you’ve never run one before. Within the app, you’ll choose a page or template you want to experiment with, which might be a product page, a collection page, or even your homepage. When you create a test, Shogun will automatically generate a copy of that page or template as Variant B, so you can start making changes without affecting the original.

🧩 Page Test

In a Page Test, you’ll start by selecting an existing page from your store to act as your Original.

From there, you can decide how to create your Variant B, your options being:

• Copy original: Shogun duplicates the page, allowing you to edit the variant safely without changing the live page.
• Select a template: Instead of copying, choose a template that matches the page type (for example, a product template for a product page). This lets you test an existing alternative layout quickly.

Once your variant (or variants in the case of multivariant testing) is set, define your goal metric (e.g., add-to-cart clicks, conversion rate, or average order value), choose your traffic split, and, optionally, set audience targeting before publishing.

Visitors will always see the same page version for the full duration of the test, ensuring accurate results.

🧱 Template Test

In a Template Test, you’re comparing Shopify templates of the same type, such as two product templates or two collection templates. You can compare multiple in the case of multivariant testing.

First, choose a template to use as your Original, then decide how to create the Variant, your options being:

• Copy original: Shogun creates an editable duplicate of the template assigned as the origional
• Select a template: Instead of copying, choose a template that matches the page type (for example, a product template for a product page). This lets you test an existing alternative layout quickly.

Shogun handles traffic assignment and ensures each visitor remains assigned to their template variant throughout the test.

Example: Compare a product template emphasizing reviews against one focused on imagery to see which increases conversions.

🔀 Split URL Test

A Split URL Test compares two existing pages in your store.

Unlike Page or Template Tests, you don’t create or edit a variant within Shogun. Instead, 50% of traffic to the Original page will be redirected to the Variant page, and Shogun tracks how each performs.

You can even test pages of different types, for example, a landing page against a product or collection page to see which drives better results for your chosen goal.

Split URL Tests are ideal for testing large content or layout differences when both pages already exist.

Redirect Behavior (Specific to URL Test)

In Split URL Testing, how visitors are redirected from the Original page to the Variant affects both test accuracy and user experience.

Shogun offers two options:

• Redirect Once (default): Visitors are redirected to the Variant only on their first visit. After that, they can access both URLs directly, while still being tracked for the test.
• Redirect Every Time: Once a visitor is assigned to a Variant, they will always be redirected to that page for the duration of the test. They cannot access the other URL while the test is live. This ensures consistent tracking, but may confuse customers who try to visit the other page.

🎨 Theme Test

A Theme Test compares your currently published theme (the Original) against another theme in your library (the Variant).

Theme Tests are ideal for validating complete redesigns, layout overhauls, or major visual updates before pushing changes live.

Important: Only one Theme Test can run at a time per store. Make sure both themes are fully configured and ready before launching.

⚖️ Traffic Split

Traffic Split determines how visitors are distributed between your variants during a test.

By default, traffic is divided evenly, 50% to the Original and 50% to the Variant, giving both versions an equal chance to collect data. However, you can adjust this ratio to control how much traffic each version receives.

For example:

• 80/20 Split: Send most visitors to the Original while testing the Variant on a smaller portion of traffic, great for cautious rollouts.
• 60/40 or 70/30 Splits: Useful when one variant is already performing well and you want to limit exposure to the other.

No matter how you configure your split, Shogun ensures that once a visitor is assigned to a version, they continue seeing that version for the entire duration of the test. This helps maintain consistency and prevents visitors from bouncing between experiences.

For new tests, a 50/50 split typically gives you faster, more statistically reliable results.

🎯 Goal Setting

Each test includes a goal that defines success for your experiment, such as increasing add-to-cart clicks, boosting sales, or improving average order value.

Shogun uses your selected goal to track performance during the test and recommend a winner once enough data has been collected.

Learn more: Read the full guide on setting and tracking test goals →

📈 Audiences

With Audiences, you can refine which visitors see each variant by segmenting based on device type, location, traffic source, or behavior.

This lets you understand how specific groups respond to changes, providing more nuanced insights than overall test results alone.

Learn more: Read the full guide on audience setup and segmentation →

Additional Resources: