Porting your test suite to MiniTest, part 1: what you gain from it
About a month ago we ported Freckle’s test suite from
test-spec to MiniTest. Doing so not only made our test suite faster, but it also paved the road for upgrading Rails in the future. Since MiniTest is now the default testing library for Rails, we now have one less thing to worry about when upgrading Rails versions. Plus, we now have Nyancat running our tests.
While porting over a test suite is not a technically challenging problem, it is certainly daunting. You’re changing thousands of lines of code, and sometimes you have to rethink tests entirely. And to make matters worse, your tests don’t just go from “green” to “red”. Instead, they go from “green” to “OH GOD, WHY?! NOTHING WORKS, EVERYTHING IS ON FIRE!”, which can be pretty demoralizing. But thankfully with some patience, a few regular expressions, and by divying up the work; the process becomes relatively painless.
In this first part, I’ll be talking about why you should go through the trouble to porting your test suite. There aren’t just performance benefits with porting to MiniTest, you’ll also make your tests clear, readable, and concise. I’ll also talk about some of design decisions we made when porting our tests and the reasoning behind them.
The benefits of “spec-assert”
Freckle’s test suite was written in “spec-shoulda” style using
test-spec, meaning that the tests were written like:
describe User, "attributes" do before :all do @user = users(:bob) end it "should have a name" do @user.name.should == "Bob Kerbin" end end
When we ported over to MiniTest, we changed the suite to be written in “spec-assert” style:
describe User, "attributes" do before do @user = users(:bob) end it "should have a first name" do assert_equal "Bob Kerbin", @user.name end end
The “spec-assert” style provides a number of benefits over “spec-shoulda”:
It’s easier to scan assertions and see what’s being tested
One problem I always ran into when writing “spec-shoulda” tests is separating the object being tested and the test itself. Since “shouldas” are written by chaining testing methods onto the object you’re testing, your tests can easily become muddled and difficult to understand. Assertions create a clear separation between the object your testing, the expected result, and the type of test you’re performing.
Compare the following examples:
subscription.user.should.be.valid assert subscription.user.valid? user.account.should == accounts(:test_account) assert_equal accounts(:test_account), user.account account.users.admin.enabled.should.include users(:bob) assert_includes account.users.admin.enabled, users(:bob)
“Shoulda” testing also makes it difficult to determine an object’s methods. Let’s look at the test on whether the subscription’s user is valid:
subscription.user.should.be.valid assert subscription.user.valid?
should.be.valid is used, you have to mentally recompile the test to understand that it checks
valid? == true. And in some cases, the
valid method might not be defined in the class. when a simple
assert is used, you immediately know that the test is going to check that the result of
Using assertions means less overhead
This one’s pretty straightforward: assertions are the foundation of MiniTest (which is now Ruby’s standard testing library). When you use them directly you get the best possible performance. Translating a “shoulda” to an assert adds overhead (Let’s assume 1ms on average). Assuming you have 3 “shoulda” assertions per test, a test suite with only 1000 tests would have 3000ms (3s) of overhead. Since the test suite for a complex Rails app is likely to have thousands of tests, you could speed up the entire suite by minutes when switching to assertions.
Changing testing styles
The biggest decisions you’ll make when porting your test suite will be deciding how to restructure tests. Besides syntax differences, the behavior of the test suite might change drastically. We reviewed our test suite and set the following rules as part of the port:
Use status code labels instead of the numeric code in functional tests
Some of our functional tests checked for the exact response code numerically
response.should.be 200 response.should.be 400 response.should.be 403 response.should.be 401
While there’s nothing wrong with these tests, it adds some unnecessary complexity, particularly when dealing with other response codes (quick: what does 403 stand for? Imagine doing that a dozen times when reading tests). with
assert_response, you can pass in a symbol with a human-readable version of the response code:
assert_response :ok assert_response :bad_request assert_response :forbidden assert_response :unauthorized
It’s just one more thing to make the test suite readable.
Use as specific assertions as possible
There are a lot of really great assertions that ActiveSupport and MiniTest provide you for free. And the more specific your assertions are, the easier your test suite is to understand.
It’s technically possible to just use
assert in your entire test suite, but that quickly becomes unreadable and difficult to maintain:
# specific asserts assert_nil subscription.user assert_not_nil subscription.user assert_includes account.users, users(:bob) assert_not_include account.users, users(:jeb) assert_kind_of User, subscription.user # generic asserts assert subscription.user.nil? assert !subscription.user.nil? assert account.users.include? users(:bob) assert !account.users.include? users(:bob) assert subscription.user.class == User
Again: the end goal of using assertions is to quickly identify what you’re testing, what your expected result is, and what test you’re running. the more specific your assertions are, the better.
Don’t use refutations
This was a design decison we made in order to make the test suite easier to scan and reduce confusion. The biggest stumbling block with refutations is that there isn’t a 1:1 mapping to the assertations provided. Not every
assert* has a matching
refute*. Often, the inverse test is
assert_not*. Therefore, we decided to ignore refutations altogether. It’s just as easy to use
assert !x. And we now have the added benefit of only having to look for
assert* in our test suite, instead of both
Next up: porting your tests!
The next step is to dive in and start rewriting your test suite (gulp). Don’t worry though! I’ve gone over some of the nitty-gritty details in this post and how to make it as painless as possible. You’ll get some workflow tips, fixes for some of the “gotchas” you’ll run into, and over 20 regexes to make porting tests a breeze.
Ready to start porting your tests? Read the next part here