Testing Strategy

RapidFTR has been built following Behaviour-Driven Development practices using Cucumber and RSpec, and contributors should continue to follow these practices. To briefly sum up our development process and testing strategy:

Stories (all viewable in Mingle) marked ready to be picked up by developers should already have some acceptance criteria attached to them. These can be used the developer as guidance for what needs to be done for the story to be complete. These acceptance criteria will make a good basis for defining a number of acceptance tests, using Cucumber. (If you don’t have Mingle access yet, then see the bottom of this page for an example of a story’s acceptance criteria.)

A Cucumber test should be created before code for a new feature is added to the system. (If new to Cucumber, have a look at our existing Cucumber tests in RapidFTR/capybara_features for examples, and study the Cucumber documentation.) A failing Cucumber test will help guide you in deciding what code needs to be written. Start with a single complete Cucumber test, run it to see it fail (and the reason why) then make it pass one step at a time by writing code. (Remember to rerun the whole suite of Cucumber tests before committing to make sure your changes haven’t broken other functionality!) Once a cucumber test is passing successfully, check in, then write the next cucumber test required to make sure all acceptance criteria on the story are covered.

Note that as part of our migration from Rails 2 to Rails 3, we currently have our Cucumber suite split across /features and /capybara_features. New Cucumber features should go into /capybara_features. There are stories queued up to convert the features in /features to use Capybara instead of Webrat, at which point they are moved to /capybara_features. We hope to have all features converted to use Capybara in the first quarter of 2012.

When writing code to make each step of a Cucumber test pass, we use RSpec tests to drive out those changes. (Rspec tests are unit tests- they specify the behaviour of individual objects, e.g. models, controllers, and views.) Sometimes the implementation of some of that code can be so trivial that it isn’t worth writing an RSpec test around it, and the fact that the Cucumber test has been made to pass is confirmation enough that this code is correct. So in this case we wouldn’t expect a developer to necessarily have RSpec tests for everything in their views and/or controllers. This is something we leave to a developer’s discretion. (Untested model code is not something we would expect!)

As this is an open source project with many different contributors, we consider good test coverage very important to maintain a fully working system at all times. Please help us by having your code submissions accompanied by Cucumber and/or RSpec tests that verifies that code does what it should. Having well named tests will really help us too.

Writing tests to guide bug fixing

• Any defects that creep up can be captured on the “Default Wall” in Mingle or the GitHub bug tracker to be picked up by developers and fixed.
• Automation tests should be written as part of the fix, unless there’s a valid reason not to. A highly recommended approach is:
  • Write an automated test which exposes the defect
  • Make sure the test fails
  • Fix the defect
  • Make sure the test passes
• A story should be closed only after all the major defects against the story have been fixed.

Testing Non-Functional Requirements (Performance & Load):

• As the system evolves over time, we need to ensure that the user experience in terms of page load response time (Performance) is up to the expectations. At the same time we should ensure that the system does not crash when the number of hits exceed a certain limit (Load). It is ideal to have a separate performance environment for these kind of testing.

• Security also needs to be covered here. We need to look out for things like session time outs, multiple logins through multi browsers / locations.

• The user interface and user experience should be verified against the design specifications. Testing the application across well known browsers is highly recommended to ensure compatibility.

Sample Story along with acceptance tests:

Story #41 Edit a user’s details

• Requirements
  • An adminstrator can edit a user’s details from the list of users
  • Any field can be edited
  • Admin can save changes
  • Updated information is shown in list of users

Acceptance Tests

• As an admin,
  • Given that I am on the user listing view page,
  • When I click on the edit link against any record,
  • Then I should be taken to a page where I can edit the user details.

• As an admin,
  • Given that I am on the edit user details page,
  • Then I should not be able to edit the “username” field

• As an admin,
  • Given that I edit user details,
  • When I click on update button
  • Then the changed user details should be updated and the others should remain the same.

• As a user,
  • Given that I am on the edit user record page,
  • Then all the validations of the fields should still hold true