Tracking New User Registrations by Source & Search Terms

In order to better analyze my registrations, I wanted to be able to associate them with their respective traffic sources and any search terms used to find my dating service. While searching for an existing gem that takes care of this, the closest thing I could find was a plugin named search_sniffer. Unfortunately, this plugin hasn’t been updated in over 3 years, and I’d much prefer a gem in order to take advantage of the benefits of bundler.

After finding nothing on my own and not getting any help from StackOverflow, I resigned to the fact that I’m going to have to implement my own solution. The solution I came up with is relatively easy, though I would ultimately like to refactor it so it doesn’t seem quite as spread out between the controller and model layers.

First, I created a migration to add new columns to my user model to capture the data I want.

class AddTrafficSourceToUsers < ActiveRecord::Migration
  def self.up
    add_column :users, :referrer, :text
    add_column :users, :traffic_source, :string
    add_column :users, :traffic_keywords, :string
  end
 
  def self.down
    remove_column :users, :referrer
    remove_column :users, :traffic_source
    remove_column :users, :traffic_keywords
  end
end

Next, I updated my ApplicationController to capture the referrer on a user’s first visit. (Non sequitur: the grammar Nazi in me can’t stand that “referer” is misspelled in the HTTP spec.)

class ApplicationController < ActionController::Base
  before_filter :capture_referrer
 
protected
 
  def capture_referrer
    session[:referrer] = request.env['HTTP_REFERER'] if !session[:referrer]
  end
end

Since it’s bad practice in the MVC methodology for the model to talk directly to the session (since models can be used outside the scope of a web request), I had to then manually set the referrer value on the model via the signup controller action.

class UsersController < ApplicationController
  def create
    @user = User.new(params[:user])
    @user.referrer = session[:referrer]
 
    if @user.save
      redirect_to home_path
    else
      render :new
    end
  end
end

By this point, you’re probably wondering why I’ve only set the referrer attribute on the model when we also need to worry about the traffic_source and traffic_keywords. Since the latter two attributes are based upon the value of the referrer attribute, that means we can set them in a model callback at the point when the record is created.

class User < ActiveModel::Base
  # NOTE: Ensure that referrer, traffic_source, and traffic_keywords
  # are NOT included in the call to attr_accessible since we don't
  # want the user to pass faux values.
  attr_accessible :username, :email, :password
 
  before_create :set_traffic_source
 
private
 
  def set_traffic_source
    if self.referrer
      uri = URI.parse(self.referrer)
      self.traffic_source ||= uri.host.downcase.gsub(/^www\./, '')
      self.traffic_keywords ||= search_terms(uri)
    end
  end
 
  def search_terms(uri)
    return nil if uri.nil?
    return nil if uri.query.nil?
 
    host = uri.host.downcase.gsub(/^www\./, '')
    params = CGI.parse(uri.query)
 
    result = host.match(/yahoo\.com$/) ? params['p'] : (params['q'] || params['query'])
    result ? result.join : nil
  end
end

There’s definitely a bit of explaining I need to do here. First, when setting traffic_source, I’m only interested in the domain (sans leading “www.”). The gsub call is used to strip out any leading “www.” so that, for example, “www.google.com” and “google.com” both end up being saved as “google.com”. Similarly, I converted all domain names to lowercase to ensure all capitalization variations are treated the same since they ultimately are.

Second, I made an assumption that any search string is going to be passed in under a query parameter named either “q” or “query”. When testing the major search engines along with a few of the lesser used ones, I found this to be true for all of them with the exception of Yahoo, which uses “p” for some reason. This is why you’ll see in my code that there is a specific check for the domain name ending in “yahoo.com” that is used to determine which query string parameter to check before defaulting to “q” or “query”.

There you have it! Now I’m able to perform simple queries on where my users are coming from and how they’re finding me. This alone isn’t much more useful than what a free service like Google Analytics offers, but obviously with greater data-mining, truly insightful information can be learned about each of your users and their behaviors’ depending upon where they’re from and how they found you.

3 thoughts on “Tracking New User Registrations by Source & Search Terms

Leave a Reply

Your email address will not be published. Required fields are marked *