Skip to content

Conversation

@matt-bernhardt
Copy link
Member

@matt-bernhardt matt-bernhardt commented Aug 28, 2023

This PR is meant to demonstrate how an A/B testing framework might function within our Rails applications. It is only a demonstration.

To see the experiment
When users visit the application, they'll be randomly selected into one of two groups. Each will be shown one version of the experiment. As of 1/7/25, the two versions look like this:

  1. The baseline (imagine this is the normal state of things). To see this version explicitly, append ?ab_test[ui_colors]=standard to the URL.
Screenshot 2026-01-07 at 3 53 41 PM
  1. The alternative (this is the change being tested). To see this version explicitly, append ?ab_test[ui_colors]=alt to the URL.
Screenshot 2026-01-07 at 3 54 00 PM

To see the results
The results dashboard can be seen at the /split path. The Split gem will collect data, and should report results when they become available.

Developer

  • All new ENV is documented in README
  • All new ENV has been added to Heroku Pipeline, Staging and Prod
  • ANDI or Wave has been run in accordance to
    our guide and
    all issues introduced by these changes have been resolved or opened as new
    issues (link to those issues in the Pull Request details above)
  • Stakeholder approval has been confirmed (or is not needed)

Code Reviewer

  • The commit message is clear and follows our guidelines
    (not just this pull request message)
  • There are appropriate tests covering any new functionality
  • The documentation has been updated or is unnecessary
  • The changes have been verified
  • New dependencies are appropriate or there were no changes

Requires database migrations?

NO

Includes new or updated dependencies?

YES

@mitlib mitlib temporarily deployed to timdex-ui-pi-ab-test-bahpibact August 28, 2023 19:59 Inactive
@matt-bernhardt matt-bernhardt changed the title A/B test demonstration [DEMO] A/B test demonstration Aug 28, 2023
@matt-bernhardt matt-bernhardt marked this pull request as draft August 28, 2023 20:00
@mitlib mitlib temporarily deployed to timdex-ui-pi-ab-test-buqsdckwb September 1, 2023 21:34 Inactive
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 3 times, most recently from cc4fc0f to 5a7c840 Compare January 10, 2024 20:35
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 3 times, most recently from 7170de9 to 63c21fa Compare February 29, 2024 19:19
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 3 times, most recently from 0b28cac to 947d918 Compare March 7, 2024 19:44
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 2 times, most recently from 39584c5 to 336d289 Compare June 3, 2024 20:31
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 2 times, most recently from 9236f6a to b42adf4 Compare November 13, 2025 17:08
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 7 times, most recently from 2aabeff to fa1fd94 Compare November 24, 2025 15:21
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 2 times, most recently from 8fa812d to b5f200a Compare December 3, 2025 19:45
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 6 times, most recently from 84f921b to 989f13d Compare December 16, 2025 21:22
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 4 times, most recently from 64f8cc6 to 59926ef Compare December 22, 2025 16:10
@matt-bernhardt matt-bernhardt force-pushed the ab-test branch 3 times, most recently from 14c39d5 to 39203fa Compare January 7, 2026 19:18
Add reference to readme for how to run Redis

The split gem includes an instruction about how to run Redis (via homebrew), but I don't want to replicate those instructures - so this is mostly just a reminder to future-me about how to run it.

Ultimately this should be part of the devcontainer, or something similar?

Fixup
This sets up a simple UI experiment, defining an alternative set of styles for the search form.

The search form starts in the _form.html.erb partial, with most users getting a "standard" class - the experimental group would get the "alt" class, which is rendered differently.

The experiment ends when a user reaches a search results page (i.e, they submitted the form and thus got results).

Curious folks can force themselves into one or the other pool by using a querystring such as

?ab_test[ui_colors]=alt
or
?ab_test[ui_colors]=standard

Refactor UI experiment based on recent changes

Fixup search controller
This tries implementing a second (concurrent) experiment, which would rely on knowing how often a user clicks an outbound link.

That means we need an interstitial redirect in order to conclude the experiment, which happens in the new outbound path.

PLEASE NOTE: This isn't working yet, because the redirect is getting caught by Turbo, it seems, and violating CORS policies. As a result, we're just logging the outbound URLs to the console.
if params[:url].present?
ab_finished(:result_format)

redirect_to params[:url], allow_other_host: true

Check warning

Code scanning / CodeQL

URL redirection from remote source Medium

Untrusted URL redirection depends on a
user-provided value
.

Copilot Autofix

AI 1 day ago

In general, to fix untrusted URL redirection you must avoid redirecting directly to arbitrary user input. Instead, you either (a) map a small, controlled set of keys to full redirect URLs on the server, or (b) validate the supplied URL so that only acceptable targets (for example specific hosts or schemes) are allowed, and reject or replace everything else with a safe default.

For this specific RecordController#out method, the least invasive fix that preserves existing behavior (redirecting to external links for A/B testing) is to validate the params[:url] using Ruby’s URI parsing and an allow‑list of acceptable schemes (typically just http and https). This still allows external redirects but prevents obviously malicious or malformed inputs (e.g., javascript: URLs). Since the controller already opts in to allow_other_host: true, we should not restrict by host without additional business requirements, but we can at least ensure the value parses as a URI and has an allowed scheme, or is a relative path. If validation fails or parsing raises an exception, we redirect to a safe fallback ('/'), just as the current code does in the “no url” case.

Concretely, in app/controllers/record_controller.rb:

  • Add a small private helper method, e.g., safe_redirect_url, that takes the raw params[:url], returns a String if it’s a valid and allowed target, and returns nil otherwise. It should:
    • require 'uri' at the top of the file to use URI.parse.
    • Accept nil or blank values and treat them as invalid (return nil).
    • Attempt to parse the URL with URI.parse.
    • Allow redirects if:
      • the parsed URI has no scheme and no host (a relative URL), or
      • the scheme is http or https (case-insensitive).
    • Rescue URI::InvalidURIError and treat as invalid.
  • Update out to use this helper: compute target_url = safe_redirect_url(params[:url]). If target_url is present, call ab_finished(:result_format) and redirect_to target_url, allow_other_host: true; otherwise, redirect to '/' without calling ab_finished.

This approach keeps the observable behavior the same for valid, typical outbound links but adds robust validation, addressing the CodeQL finding.

Suggested changeset 1
app/controllers/record_controller.rb

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/app/controllers/record_controller.rb b/app/controllers/record_controller.rb
--- a/app/controllers/record_controller.rb
+++ b/app/controllers/record_controller.rb
@@ -1,3 +1,5 @@
+require 'uri'
+
 class RecordController < ApplicationController
   before_action :validate_id!, only: %i[view]
 
@@ -7,10 +9,11 @@
   #
   # @param url String a URL where the URL will be redirected
   def out
-    if params[:url].present?
-      ab_finished(:result_format)
+    target_url = safe_redirect_url(params[:url])
 
-      redirect_to params[:url], allow_other_host: true
+    if target_url.present?
+      ab_finished(:result_format)
+      redirect_to target_url, allow_other_host: true
     else
       redirect_to '/'
     end
@@ -33,6 +35,24 @@
 
   private
 
+  def safe_redirect_url(raw_url)
+    return nil if raw_url.blank?
+
+    begin
+      uri = URI.parse(raw_url)
+
+      # Allow relative URLs (no scheme/host)
+      return raw_url if uri.scheme.nil? && uri.host.nil?
+
+      # Allow only HTTP(S) URLs
+      return raw_url if %w[http https].include?(uri.scheme&.downcase)
+    rescue URI::InvalidURIError
+      # fall through to return nil
+    end
+
+    nil
+  end
+
   def validate_id!
     return if params[:id]&.strip.present?
 
EOF
@@ -1,3 +1,5 @@
require 'uri'

class RecordController < ApplicationController
before_action :validate_id!, only: %i[view]

@@ -7,10 +9,11 @@
#
# @param url String a URL where the URL will be redirected
def out
if params[:url].present?
ab_finished(:result_format)
target_url = safe_redirect_url(params[:url])

redirect_to params[:url], allow_other_host: true
if target_url.present?
ab_finished(:result_format)
redirect_to target_url, allow_other_host: true
else
redirect_to '/'
end
@@ -33,6 +35,24 @@

private

def safe_redirect_url(raw_url)
return nil if raw_url.blank?

begin
uri = URI.parse(raw_url)

# Allow relative URLs (no scheme/host)
return raw_url if uri.scheme.nil? && uri.host.nil?

# Allow only HTTP(S) URLs
return raw_url if %w[http https].include?(uri.scheme&.downcase)
rescue URI::InvalidURIError
# fall through to return nil
end

nil
end

def validate_id!
return if params[:id]&.strip.present?

Copilot is powered by AI and may make mistakes. Always verify output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants