Stabilize Brittle Specs with Set

Overview

This is a beginner-to-intermediate level tutorial.

Time-to-complete is probably around 5 minutes.

The Problem

You want to test equality on an array or an array-like object with unique elements, but the order of the result you are testing is non-deterministic. You’re doing some sort of integration spec where you don’t want to stub out the service call - perhaps something like a feature spec that queries your Redis instance.

A simplified example:

describe Service do
  it 'has the results I expect' do
    results = Service.get('fubar:members')

    expect(results).to eq(['foo:1', 'foo:7', 'foo:42'])
  end
end

This works fine, except for when the Service occasionally returns the results in a different order.

If the results from Service.get happen to return ['foo:7', 'foo:1', 'foo:42'] then your spec would fail.

To get around this, you’ll see test cases that have this shape:

describe Service do
  it 'has the results i expect' do
    results = Service.get('fubar:members')

    expect(results).to include('foo:1')
    expect(results).to include('foo:7')
    expect(results).to include('foo:42')
  end
end

This approach works, but if you’re like me, you try to stick to one assertion per spec

Solution

If you’re not concerned with order and you’re only interested in testing for membership, then Ruby’s Set will do the trick.

You can initialize a set with an array using Set.new(your_array).

And as you can see, while the ordering of elements does determine equality in an Array, it does not for a Set.

# order matters to an array
%w(a b c) == %w(c b a)
# => false

# but not to a set
require 'set'
Set.new(%w(a b c)) == Set.new(%w(c b a))
# => true

We can rewrite our sample spec like this:

describe Service do
  it 'has the results I expect' do
    raw_results = Service.get('fubar:members')

    results     = Set.new(raw_results)
    expectation = Set.new(['foo:1', 'foo:7', 'foo:42'])

    expect(results).to eq(expectation)
  end
end

Given our problem (unique elements with non-deterministic ordering), this version of our spec solves our problem.

Closing

A great point can be made of not hitting any sort of live service from your tests to begin with.

If your test data is coming from a fixture (stub, vcr, factory girl, fakeredis, etc etc), then the order is determined to begin with, and these sorts of non-deterministic spec problems are a non-issue.

Another point is that, admittedly, this solution only really works for when you’re wrapping results sets that you expect to have unique elements. Calling Set.new([1,1,1]) is somewhat akin to calling [1, 1, 1].uniq, so you’ll want to make sure that a Set really is a good fit for your data.

Still, it’s a great tool to have in your toolkit. You’ll sometimes see these issues pop up in certain types of integration specs or when your lib code is internally using Sets.

Obviously, they are also helpul in regular application code when the order of your data doesn’t matter, and you only want to ingest unique elements.

In these cases and quite a few others, you’ll be glad to have Set at your fingertips.

Update (12/28/2015)

As some people pointed out, RSpec comes with match_array, which would negate the need for using Set.

I probably made a mistake saying fix brittle specs, as my intention was to be a reminder of what you can do with the RSL rather than being a tutorial on RSpec.

Still, match_array is pretty handy when you are writing specs. I’ve used it at least once since writing this post.

In the case when you want a framework agnostic way to achieve the same results, or if you’re writing your own matcher, then Set would be useful.

– Jim