Run scenarios in fully random order

[threedaymonk]

This documents the desired behaviour of the random order option across multiple features and corrects the ordering of filters within runtime.rb in order to run scenarios in a genuinely random order.

Previously, using --order=random resulted in the order of scenarios being randomised within a feature, but features themselves always proceeded in alphabetical (or ASCIIbetical) order based on their file path, i.e.:

Feature: 1
  Scenario: 1B
  Scenario: 1A

Feature: 2
  Scenario: 2B
  Scenario: 2A

Although Randomizer does indeed shuffle the scenarios, it was followed in the filter chain by LocationsFilter which groups scenarios by file path. Moving Randomizer later in the chain avoids this.

Now, each scenario is run in random order regardless of the name or path of the feature:

Feature: 1
  Scenario: 1A

Feature: 2
  Scenario: 2B
  Scenario: 2A

Feature: 1
  Scenario: 1B

Fixes #969.

[threedaymonk]

Well, this is difficult: the feature depends on the seeded random number generator behaving predictably, but the results on Ruby 1.9 and JRuby are different from Ruby 2.x.

[threedaymonk]

The problem isn't Random itself, but the implementation of Array#shuffle in Ruby. Random.new(5) always returns the same sequence for rand, but shuffling is different:

Ruby 2.x:

[1,2,3].shuffle(random: Random.new(4))
=> [2, 1, 3]
[1,2,3].shuffle(random: Random.new(5))
=> [1, 2, 3]

Ruby 1.9.3:

[1,2,3].shuffle(random: Random.new(4))
=> [1, 2, 3]
[1,2,3].shuffle(random: Random.new(5))
=> [3, 2, 1]

And JRuby appears to be different again.

[mattwynne]

Nice, tidy work @threedaymonk, thanks!

As part of a new policy and as recognition of the quality of your contribution, I've invited you as a committer to this repo. If you'd like to help out some more and don't know where to start, feel free to grab me on gitter.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.