Hunting flaky tests 1: List ordering

See table of contents here for more flaky test resolution examples.

At Doctolib we use PostgreSQL, but the behaviour would be the same across all databases. As explained in the PostgreSQL documentation: “If sorting is not chosen, the rows will be returned in an unspecified order”.

• INSERT INTO mytable VALUES (a, b)
• SELECT * FROM mytable=> [b, a]

So, as a result of this behaviour the test seen here will be flaky:

users = [create(:user), create(:user)]
assert_equal users, Users.all.to_a

The solution is to always sort your database arrays before using them, either in sql or ruby:

assert_equal users[0], User.order(:id)[0]
assert_equal users[0], User.all.sort_by(&:id)[0]

Please note that unlike ActiveRecord::all, ActiveRecord::first and ActiveRecord::last are safe: they implicitly order by id.

As you can see in the real world example below, we test the json response of an api endpoint. Since we did not designate an order in the get call, the endpoint will respond the same as the database: