Updated October 2026: Rewrote the Split gem tutorial from its current README (Redis-backed, no generator or migration, ab_finished instead of a non-existent helper) and added sections on statistical significance and feature-flag based experimentation.

A/B testing, also known as split testing, is a method used to compare two or more versions of a web page or application to determine which one performs better in achieving a predefined goal. This technique is widely used in web development, marketing, and user experience optimization to make data-driven decisions and improve overall performance. In this article, we’ll delve into the concept of A/B testing and explore how it can be implemented using the Split gem in Ruby on Rails.

I. What is A/B Testing?

A/B testing involves creating two or more variations of a webpage, email, or application feature and randomly assigning users to each variation. By comparing the performance metrics of each variation, such as conversion rates, click-through rates, or engagement levels, organizations can identify which version resonates better with users and drives the desired outcomes. A/B testing allows for iterative improvements by continuously refining designs, content, or functionalities based on empirical data rather than assumptions.

II. Why Use A/B Testing?

  1. Data-Driven Decisions: A/B testing provides empirical evidence to inform decisions, reducing reliance on intuition or guesswork.
  2. Optimization: It enables continuous optimization of web assets to maximize conversions, engagement, or other key metrics.
  3. User Insights: A/B testing helps uncover user preferences, behaviors, and pain points, leading to improved user experiences.
  4. Risk Mitigation: By testing changes on a subset of users, organizations can mitigate the risk of deploying ineffective or harmful modifications to their entire user base.

III. Implementing A/B Testing in Rails with the Split Gem

In Ruby on Rails, the Split gem provides a convenient way to run A/B tests inside your application. It is still maintained (latest release 4.0.5, August 2025) and is tested with Ruby >= 2.5 and Rails >= 5.2.

Prerequisite: Redis. Split stores experiments, alternatives and results in Redis (4.0 or greater), not in your database — so there is no generator to run and no migration.

1. Installation

Add the Split gem to your Gemfile. The require option also loads the optional dashboard:

gem "split", require: "split/dashboard"

Then, install the gem by running:

bundle install

In Rails, the gem is loaded automatically when the app boots.

2. Configuration

By default Split connects to the Redis server in the REDIS_URL environment variable, falling back to redis://localhost:6379. You can also set it explicitly, and define your experiments, alternatives and goals in an initializer:

# config/initializers/split.rb
Split.redis = ENV.fetch("REDIS_URL", "redis://localhost:6379")

Split.configure do |config|
  config.experiments = {
    button_color: {
      alternatives: ["blue", "red"],
      goals: ["sign_up"]
    }
  }
end

By default, the alternative assigned to each user is stored in the session. Set config.persistence = :cookie to store it in a cookie instead.

3. Usage

Split’s Rails engine includes Split::Helper into controllers and views automatically, so ab_test and ab_finished are available everywhere without an include.

ab_test returns one of the alternatives and keeps returning the same one to a returning user. Call it with just the experiment name once the experiment is configured, either with a block in a view:

<% ab_test(:button_color) do |color| %>
  <% if color == "red" %>
    <%= link_to "Sign Up", new_user_path, class: "btn btn-danger" %>
  <% else %>
    <%= link_to "Sign Up", new_user_path, class: "btn btn-primary" %>
  <% end %>
<% end %>

or as a return value in a controller:

@button_color = ab_test(:button_color)

4. Tracking conversions

When the user reaches the goal, call ab_finished. For an experiment with goals, pass a hash of experiment name to goal:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def create
    @user = User.new(user_params)
    if @user.save
      ab_finished(button_color: "sign_up")
      redirect_to root_path, notice: "User successfully created!"
    else
      render :new, status: 422 # so Turbo renders the form errors
    end
  end
end

For an experiment without goals, it is simply ab_finished(:button_color). By default, finishing an experiment resets the user so they can enter it again later; pass reset: false (e.g. ab_finished(:button_color, reset: false)) to keep showing them the same alternative.

5. The dashboard

Mount the dashboard to see participants, conversions and statistical confidence per alternative — and protect it, since it can also pick winners and reset experiments:

# config/routes.rb
Split::Dashboard.use Rack::Auth::Basic do |username, password|
  # Compare digests so the check doesn't leak length; use & (not &&) so it doesn't short circuit.
  ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(username), ::Digest::SHA256.hexdigest(ENV["SPLIT_USERNAME"])) &
    ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(password), ::Digest::SHA256.hexdigest(ENV["SPLIT_PASSWORD"]))
end

Rails.application.routes.draw do
  mount Split::Dashboard, at: "split"
end

IV. Reading the Results Correctly

The hard part of A/B testing is not the code, it is not fooling yourself with noise:

  • Decide the sample size before you start. Pick the minimum effect you care about and compute how many users each variation needs. Small samples produce “winners” that are just chance. Split’s dashboard uses a z-test, which needs more than 30 participants and at least 5 conversions per alternative to be meaningful at all — real experiments usually need far more.
  • Look for statistical significance, not a bigger number. A variation with a higher conversion rate is not a winner until the difference is statistically significant at the confidence level you chose up front (commonly 95%).
  • Don’t peek and stop early. Checking the results every day and stopping the moment one variation looks significant greatly inflates false positives. Run the test until it reaches the planned sample size (and ideally full weekly cycles), then decide.
  • Test one change per experiment and keep the goal metric fixed, so you know what caused the difference.

V. Feature Flags as the Modern Alternative

Many teams now run experiments on top of a feature-flag system instead of a dedicated A/B testing gem: the same flag that rolls a feature out gradually can split users into variations and record exposures and conversions. In the Rails world, Flipper is a popular feature-flag library, and products such as GrowthBook and PostHog combine feature flags with experiment analysis.

VI. Conclusion

A/B testing is a powerful technique for optimizing web assets and driving user engagement and conversions. With the Split gem and a Redis server, a Rails app can run experiments with a few lines of code: configure the experiment, call ab_test to assign users, and ab_finished to record the goal. The code is the easy part — plan your sample size, wait for statistical significance and avoid peeking, and the results will be a reliable basis for data-driven decisions.