Futurechimp

We recently did some e-commerce work for an online startup called Hubbub. We’re pretty proud of it here, so it seems worth it to talk a little about the challenges involved in building it.

Most e-commerce jobs are relatively simple from a business process point of view – one website has one shop inside it, and in most cases, when you buy something it instantly gets sent for fulfillment (either physically or digitally). For example, if you buy a pair of shoes from Humanic (which we worked on last year), the shoes move into the delivery system immediately, and are usually sent out the next day. Fulfillment, i.e. the delivery of your shoes, is handled by a dedicated service provider external to the website itself.

Hubbub is completely different. Instead of having one shop and a relatively simple delivery system, the site is composed of multiple shops (a baker, an Italian deli, a fish shop, etc) which are in a delivery area – so the Hubbub site needs to keep track of which products have been ordered from which shops. New shops and new delivery areas can be added at any time. If Hubbub decides to expand into Kensington, Chelsea, and Clapham tomorrow, it can add those three neighbourhoods as new delivery areas, and add the shops from each neighbourhood into the right area. Specialist shops (like Christchurch Fish, which can serve all of London) can be shared across multiple delivery areas. The site then keeps track of which shops have had which items ordered from them.

Items can be scheduled for delivery on a specific hour anytime in the 30 days from the date of purchase. The site has its own fulfilment system built into it, automating the scheduling of store pick-ups and item deliveries for each delivery area on an hourly basis. Delivery drivers have their pickup and delivery schedules emailed to them every day in the early afternoon, allowing them to plan out their routes for the day. All in all, the site code is much more than a simple webshop, it’s a fully integrated customized e-commerce software platform which enables Hubbub to function. It’s arguable that the Hubbub business would be impossible without the code to back it up.

Most sites take payment when a customer orders an item. On Hubbub, this isn’t possible because the price of the item may not be known when it’s ordered. In the case of a steak, a chicken, or a bag of fruit, for instance, the site may give an indicative price (steak is £3.99/lb), but the actual price of the steak which will be delivered isn’t known until the butcher actually cuts it up and weighs it on the day of delivery. This is a huge issue from a business process point of view. In order to make this relatively complex payment flow work, we wrote our own payment gateway integration code so that we can do 3-D Secure authentication with the SecureTrading Xpay gateway.

The end result is a site which looks deceptively simple. It unifies multiple webshops (which correspond to multiple real-world businesses) into a unified front-end, offering each shop its own login area for product management and order tracking. The Hubbub site administrators also have their own secure login area which allows them to track the progress of the business.

The shop code is based on a heavily-modified version of Spree retrofitted with our own internal role-based access control systems and an extensive test suite based on shoulda, machinist and cucumber.

If we made one mistake during development, it was to write too much new code before we’d adequately covered the Spree codebase with tests we liked. The default Spree comes with an rspec test suite which was incomplete (when we forked the code from Spree six months ago) – we like Shoulda and Test::Unit better, so we rewrote the suite. We should have gone all the way with the code coverage before writing any new code, though – it was amazing how much our development effort sped up once we got code coverage above ~70%. For new projects, we’ve instituted a 95% code coverage minimum on our CruiseControl server (with an actual internal target of 100%).

It’s a pretty good feeling – over the past few years, we’ve gotten way more agile. Codewise at least.

Time for another brain-externalizing post to remind myself how things work when setting up new systems. The basic process for setting up a Rails environment with exciting testing goodness is:

sudo gem install ZenTest autotest-rails redgreen

Then in your home directory, make a .autotest file containing:

require 'redgreen/autotest'

I’d missed it, but Rails now incorporates the functionality previously provided by Thoughtbot’s quietbacktrace gem. It’s pretty easy to use, too. By default, it whacks all of the stacktrace garbage that used to happen in a Rails 2.2 app. It’s also possible to easily add new filters in your test_helper.rb file, e.g.

  # Add more helper methods to be used by all tests here...
  #
  Rails.backtrace_cleaner.add_silencer { |line| line =~ /lib\/shoulda/ }

Artificially inducing a test failure then gives a beautiful little stack trace, like so:

  1) Failure:
test: the JobsController should get new. (JobsControllerTest) [/test/functional/jobs_controller_test.rb:19]:
Expected response to be a <:successsadfsdaf>, but was <200>

Here’s a quick bit of capistrano to copy your production data into your staging environment if you’re running a multi-stage setup:

desc "Loads the production database and assets into the staging environment" 
task :load_production_data, :roles => :app do
  require 'yaml'
  # get the database.yml file from the server
  production_yml = "config/database.production#{Time.now.strftime '%Y-%m-%d_%H:%M:%S'}.yml" 
  staging_yml = "config/database.staging#{Time.now.strftime '%Y-%m-%d_%H:%M:%S'}.yml" 

  begin
    get "/path/to/your/production/current/config/database.yml", production_yml
    get "#{current_path}/config/database.yml", staging_yml

    # load the values
    p_config = YAML::load_file(production_yml)
    username = p_config['production']['username']
    password = p_config['production']['password']
    database = p_config['production']['database']
    s_config = YAML::load_file(staging_yml)
    s_username = s_config['staging']['username']
    s_password = s_config['staging']['password']
    s_database = s_config['staging']['database']
  rescue
    raise
  ensure
    `rm -f #{production_yml};rm -f #{staging_yml}`
  end

  # copy the SQL data
  filename = "/tmp/dump.#{Time.now.strftime '%Y-%m-%d_%H:%M:%S'}.sql.gz" 
  run "mysqldump -u #{username} --password='#{password}' #{database} | gzip > #{filename}" 
  run "gunzip -c #{filename} | mysql -u #{s_username} --password='#{s_password}' #{s_database} && rm -f gunzip #{filename}" 

  # Copy media files under system  
  run "cp -Rf /path/to/your/production/current/public/system/* #{current_path}/public/system/" 
end

Seems to work alright for us – remember that you may need to re-apply database migrations using cap deploy:migrate in case your staging database is ahead of your production database.

As part of a general quest to slim down my programming environment (driven largely by the purchase of a Samsung NC-10 netbook and a love of programming on trains), I’ve switched to gedit as my main editor. It’s been hugely enjoyable to code in a lightweight environment – and the gedit-mate git repo in combination with apt-get install gedit-plugins makes it easy to set up a decent Ruby environment.

The only small problem I’ve had with the gedit-mate installer was that syntax highlighting wasn’t working for me out of the box. To fix it, I’ve forked the github repo and pushed a commit that installs the ruby_on_rails.lang file so that everything works properly on Ubuntu 9.04.

A very quick note about testing using Cucumber and Webrat when using subdomains, in case it helps anyone (or perhaps my future self when I no longer remember this little detail in six months time).

Today I was working on a feature which had as one of its requirements that the user go to a special admin subdomain in order to log in to the admin system. This was blowing up all over the place, but we’ve now got it working alright. The first thing to realize is that Cucumber/Webrat need to be told about your subdomain. We first tried dumping the following into Cucumber’s env file:

def use_admin_subdomain
  host! "#{::ADMIN_SUBDOMAIN}.test.host" 
end

Our reasoning was that Rails normally runs its functional tests with a @request.host value of “test.host”, so if we wanted to shift things around, we figured we could just stick the admin subdomain on the front of that host and everything would be peachy. We put use_admin_subdomain at the top of our admin_steps, like so:

Given /^I am logged in as an admin user$/ do
  use_admin_subdomain
  Given %q{An admin user with username "admin@example.com" and password "password"}
  And %q{I log in with username "admin@example.com" and password "password"}
end

The result:

Then I should see "Thing was successfully created." # features/step_definitions/webrat_steps.rb:118
     expected the following element's content to include "Thing was successfully created.":
     You are being redirected. (Spec::Expectations::ExpectationNotMetError)
     features/admin_manage_thing.feature:16:in `Then I should see "Thing was successfully created."'

Failing Scenarios:
cucumber features/admin_manage_thing.feature:9 # Scenario: Create Thing

We scratched our heads and outputted the failing page into the browser. Weirdly, it just had a message in it saying “You are being redirected” with a link to the redirect. After doing a bit of research, we figured out that this happens because Cucumber/Webrat run tests in the www.example.com domain, so we were effectively trying to redirect in our tests from admin.test.host to www.example.com and Rails didn’t like doing that very much.

Changing our use_admin_subdomain method to do this instead worked great:

def use_admin_subdomain
  host! "#{::ADMIN_SUBDOMAIN}.example.com" 
end

This works because Rails is happy to do subdomain redirects and doesn’t display that weird “you are being redirected” page. And now, back to the code!