My ugly mug

Hi, I'm Thomas Cannon!

I work on Freckle with Amy Hoy and Thomas Fuchs, and I teach people how to add an extra layer of polish to their work. Previously I worked for BMW and Clemson University, and have a bunch of smaller apps I tinker with on the side. Follow me on Twitter!

Porting your test suite to MiniTest, part 1: what you gain from it

About a month ago we ported Freckle’s test suite from test-spec to MiniTest. Doing so not only made our test suite faster, but it also paved the road for upgrading Rails in the future. Since MiniTest is now the default testing library for Rails, we now have one less thing to worry about when upgrading Rails versions. Plus, we now have Nyancat running our tests.

While porting over a test suite is not a technically challenging problem, it is certainly daunting. You’re changing thousands of lines of code, and sometimes you have to rethink tests entirely. And to make matters worse, your tests don’t just go from “green” to “red”. Instead, they go from “green” to “OH GOD, WHY?! NOTHING WORKS, EVERYTHING IS ON FIRE!”, which can be pretty demoralizing. But thankfully with some patience, a few regular expressions, and by divying up the work; the process becomes relatively painless.

In this first part, I’ll be talking about why you should go through the trouble to porting your test suite. There aren’t just performance benefits with porting to MiniTest, you’ll also make your tests clear, readable, and concise. I’ll also talk about some of design decisions we made when porting our tests and the reasoning behind them.

The benefits of “spec-assert”

Freckle’s test suite was written in “spec-shoulda” style using test-spec, meaning that the tests were written like:

describe User, "attributes" do
  before :all do
    @user = users(:bob)
  end

  it "should have a name" do
    @user.name.should == "Bob Kerbin"
  end
end

When we ported over to MiniTest, we changed the suite to be written in “spec-assert” style:

describe User, "attributes" do
  before do
    @user = users(:bob)
  end

  it "should have a first name" do
    assert_equal "Bob Kerbin", @user.name
  end
end

The “spec-assert” style provides a number of benefits over “spec-shoulda”:

It’s easier to scan assertions and see what’s being tested

One problem I always ran into when writing “spec-shoulda” tests is separating the object being tested and the test itself. Since “shouldas” are written by chaining testing methods onto the object you’re testing, your tests can easily become muddled and difficult to understand. Assertions create a clear separation between the object your testing, the expected result, and the type of test you’re performing.

Compare the following examples:

subscription.user.should.be.valid
assert subscription.user.valid?

user.account.should == accounts(:test_account)
assert_equal accounts(:test_account), user.account

account.users.admin.enabled.should.include users(:bob)
assert_includes account.users.admin.enabled, users(:bob)

“Shoulda” testing also makes it difficult to determine an object’s methods. Let’s look at the test on whether the subscription’s user is valid:

subscription.user.should.be.valid
assert subscription.user.valid?

When should.be.valid is used, you have to mentally recompile the test to understand that it checks valid? == true. And in some cases, the valid method might not be defined in the class. when a simple assert is used, you immediately know that the test is going to check that the result of subscription.user.valid? is true.

Using assertions means less overhead

This one’s pretty straightforward: assertions are the foundation of MiniTest (which is now Ruby’s standard testing library). When you use them directly you get the best possible performance. Translating a “shoulda” to an assert adds overhead (Let’s assume 1ms on average). Assuming you have 3 “shoulda” assertions per test, a test suite with only 1000 tests would have 3000ms (3s) of overhead. Since the test suite for a complex Rails app is likely to have thousands of tests, you could speed up the entire suite by minutes when switching to assertions.

Changing testing styles

The biggest decisions you’ll make when porting your test suite will be deciding how to restructure tests. Besides syntax differences, the behavior of the test suite might change drastically. We reviewed our test suite and set the following rules as part of the port:

Use status code labels instead of the numeric code in functional tests

Some of our functional tests checked for the exact response code numerically

response.should.be 200
response.should.be 400
response.should.be 403
response.should.be 401

While there’s nothing wrong with these tests, it adds some unnecessary complexity, particularly when dealing with other response codes (quick: what does 403 stand for? Imagine doing that a dozen times when reading tests). with assert_response, you can pass in a symbol with a human-readable version of the response code:

assert_response :ok
assert_response :bad_request
assert_response :forbidden
assert_response :unauthorized

It’s just one more thing to make the test suite readable.

Use as specific assertions as possible

There are a lot of really great assertions that ActiveSupport and MiniTest provide you for free. And the more specific your assertions are, the easier your test suite is to understand.

It’s technically possible to just use assert in your entire test suite, but that quickly becomes unreadable and difficult to maintain:

# specific asserts
assert_nil subscription.user
assert_not_nil subscription.user
assert_includes account.users, users(:bob)
assert_not_include account.users, users(:jeb)
assert_kind_of User, subscription.user

# generic asserts
assert subscription.user.nil?
assert !subscription.user.nil?
assert account.users.include? users(:bob)
assert !account.users.include? users(:bob)
assert subscription.user.class == User

Again: the end goal of using assertions is to quickly identify what you’re testing, what your expected result is, and what test you’re running. the more specific your assertions are, the better.

Don’t use refutations

This was a design decison we made in order to make the test suite easier to scan and reduce confusion. The biggest stumbling block with refutations is that there isn’t a 1:1 mapping to the assertations provided. Not every assert* has a matching refute*. Often, the inverse test is assert_not*. Therefore, we decided to ignore refutations altogether. It’s just as easy to use assert !x. And we now have the added benefit of only having to look for assert* in our test suite, instead of both assert* and refute*.

Next up: porting your tests!

The next step is to dive in and start rewriting your test suite (gulp). Don’t worry though! I’ve gone over some of the nitty-gritty details in this post and how to make it as painless as possible. You’ll get some workflow tips, fixes for some of the “gotchas” you’ll run into, and over 20 regexes to make porting tests a breeze.

Ready to start porting your tests? Read the next part here