Test Driven Development (TDD)

[lewiswharf]

It's been brought up by @iwyg and @brendo noted how much time it will save getting new releases out. I agree that it must be implemented for Symphony Next.

It adds a bit of extra work up front but there are huge time savings down the road when implementing new features, changes etc. and ensuring minimal bugs.

I do not have any experience with TDD but I understand its benefits and how to implement.

Which testing utilities should we use?

PHP Core

• PHPUnit Considered the standard.
• SimpleTest As the name suggests, more simple :-)
• Behat (Behavior Driven Development) by @iwyg

UI Testing

• CasperJS Mentioned by @brendo
• QUnit Apparently jQuery uses this

Javascript

• mocca by @iwyg
• intern by @iwyg
• Jasmine by @germchaos

Updated include suggestions from comments @brendo

[iwyg]

Let me add

PHP

• Behat (Behavior Driven Development)

Javascript

• mocca

I'd say PHPunit fur core development. Behat could be evaluated for maybe integration?

[bernardodiasc]

Javascript

• Jasmine

[designermonkey]

It might be beneficial to store these tests even after use, and also create tests for extension developers to use to aid in better development throughout the community.

I would say we don't need to leave the tests in the release versions, but have them available to add to a build when required (if that's how it works, I don't know).

[lewiswharf]

Tests are a permanent thing. Each and every time the code base is touched, the tests can be executed with a single command to ensure changes to the code didn't break anything.

[remie]

I think this discussion needs to be split into "which test frameworks to use" and "are we going to do TTD". Because the latter question can have a huge impact on development. TTD is on of those methodologies that can ignite a holy war. Think about it: this would mean that you will have to write a test prior to implementing the code. So you will need to know the input and the output of a method even before you started writing a single letter of PHP. Now you will have to mock objects that do not even exist in order to verify if the returned output of the method to be created is as expected.

From my experience, TTD not only costs to much in development time, it also kills any and all flexibility in coding. It does not play well with Agile development methodologies.

So yes, we should create unit tests, but this can also be done after the method has been deliverd. We can make it part of the acceptance criteria of an issue.

I would strongly recommend using Behavior Driven Development and use testing frameworks like Cucumber (which is extremely easy to use) to test the features rather than the code.

[remie]

ps: although an authority claim is always a bad argument, perhaps it helps to know that in real life I'm managing a QA department of a reasonably large commercial website in the netherlands.

[jonmifsud]

@remie setting up tests before any code is written surely would take more time but then I guess it makes it easier to test pull requests prior to integration. That way you're always sure that whoever is providing the code did not break anything else.

I've never quite used automated testing before; however I do understand over large projects with multiple contributors might be quite beneficial.

[remie]

@jonmifsud: one does not exclude the other. You can have acceptance criteria for pull requests which include automated tests (be it unit tests or feature testsing). Just like doing a code review can be a required step for the person performing the pull request.

However there is no sense in dictating at what point the author of the pull request creates the tests. TTD would require the contributor to create the test first and than start coding. Why would that be any better? The claim of TTD that this would create a better architecture and that the tests would be better constructed (instead of testing the actual outcome it would test the desired outcome) has no real foundation. There is no guarantee that the developer will be able to come up with more test cases or better scenario's when he thinks about the test before he starts writing the code.

However, because we all intuitively feel that it is bad to write a test that will only verify what you've already written because a developer might be biased by that time, there is an alternative. Behavior Driven Development is the perfect match between letting the developer be more flexible in their creative process of creating code and being sure that features, which are predefined, are working as expected.

On could for instance create BDD tests on a dummy front-end and launch these tests on the final product. This is also more agile for it allows parallel development. Front-end developers and QA engineers can create the requirements and tests while developers start implementing it.

[jonmifsud]

@remie basically one can re-define & add tests later down the line to match up with use-cases which were missed in the first place for example.

BDD does look more practical; I think as long as the tests are defined prior to pull requests being done and use cases verified I don't see much problem either way. What you wouldn't want is to have a half-tested repository.

[iwyg]

@remie thanks for the heads up. Btw. I just read that you can use cucumber to test php-applications although I'm not sure if this plays well with travis-ci because of the different languages. Behat seems to be a good alternative.

I don't think we should drop the idea of unittesting. Even when you do it the "wrong" way, it still will have more benefits than writing none at all.

[remie]

@iwyg: yes, we should definitely write PHPUnit tests! TTD is just a very strict methodology (dogmatic even). I would not go with that. However, I'm strongly in favour of adding PHPUnit tests as a requirement for pull requests.

[designermonkey]

That makes me happier for contributing. Also, it will allow others to contribute more freely, which we will want after we release at least a beta.

[iwyg]

You even can run tests automatically on every pullrequest using travis-ci and the travis bot. Quite nice.

[designermonkey]

I'm reading about Behat. Throwing my hat into that ring.

[iwyg]

may I add intern for javascript

[ijy]

Seeing as BDD is TDD done right it seems to be a good way to go. Behat is inspired by Ruby's popular Cucumber project and also uses Gherkin (the language and parser) to describe the feature in terms of a story. The advantage of this includes providing a human readable DSL (Domain Specific Language) to define features and scenarios so potentially that can also be handled by those with less direct programming experience too allowing things to be split up a little more for testing and quality control.

Behat also seems to provide a useful companion tool in Mink (although it's not exclusively for use with Behat) for web acceptance testing. It's benefit seems to be in providing a single API for both headless browser emulation and in-browser emulation so the actual testing driver (i.e Selenium2, Goutte, Sahi, etc.) can be swapped out if required allowing the use of some of the above suggestions. Having a common API at this layer would certainly be useful and flexible.

That would cover the StoryBDD side of things (the business values and priorities) and PHPSpec2 looks like a very promising tool to cover SpecBDD (the 'how' and the implementation). I don't think it's quite as established as PHPUnit but it seems to show promise and have good recommendations. Either could be used however as they'd both be compatible.

[lewiswharf]

BDD does sound good.

[designermonkey]

If you write user stories for your work, you're in for a treat!

[brendo]

Seems like if we go down the BDD path then Jasmine and Behat would be the ideal mix?

[iwyg]

#6 (comment).
We shouldn't drop unit tests, I think they're quite important, especially for underlying modules (e.g. is it desired in BDD to test protected methods?). Although I must admit that Behat/Mink both sound quite interesting to me.

Has anyone had a look at intern as an alternative to jasmine? I mean, it sounds quite promising:

Run tests using the browser or node.js and seamlessly integrate with SauceLabs or Selenium. Write tests using object, TDD, or BDD styles and get full code coverage reporting with Istanbul. Continuous integration? Intern has you covered with Travis CI support out-of-the-box.

[iwyg]

Oh, and sticking to PHPunit would also integrate in the frameworks testsuite.

[iwyg]

Doh! Another one codeception

[ijy]

Oh, and sticking to PHPunit would also integrate in the frameworks testsuite.

If things are going the BDD way with Behat then would a full-stack BDD approach be a better option using PHPSpec2 instead of PHPUnit? PHPUnit seems more TDD. PHPSpec2 and Behat are the equivalent of RSpec and Cucumber in the Ruby world which (it takes PHP a while to catch up but Ruby definitely leads the way in many areas).

The key difference is seems to be that PHPSpec2 describes what the object does instead of what it is. It actually helps you develop rather than yell at you that the code is broken:

In BDD you are not testing you are describing what that classes will do, and so you can continue creating the class (PHPSpec will create the class and the methods for you)

The BDD tests what the object does instead of what it is, and what it does is much more important.

This article is pretty interesting in looking at the comparison of xUnit and SpecBDD (PHPUnit and PHPSpec2). And this quick video shows it in action with a nicely colour coded command line in more of an interactive exchange. Generally it seems more helpful both in focusing on what matters most (what the class/object does rather than what it is) and in determining what is wrong, even offering to help out and create the classes or methods for you. Kinda like having a testing buddy sitting right there with you through those loooong nights of testing (you know, for when the Mrs says ENOUGH!). PHPSpec2 is your buddy, he's your pal, and he'll stay up long after the Mrs has gone to bed keeping the party going. Throw in a Mink and a Gherkin...maybe a few cans of beer... and it can still be one hell of a night!

[iwyg]

to recall codeception: Laravel 4 testing

[DavidOliver]

Seems like a nice intro video for Selenium: Test your web apps with Selenium

[creativedutchmen]

Honestly I don't care if we are using TDD, BDD or whatever methodology, as long as the outcome is the same: Symphony needs to be automatically testable, so that one can be certain a bugfix does not break other things, and to be sure a refactor does not break anything else.

[s-e]

I've been hanging around the Laravel community a bit. Along with the L4 release yesterday, Jeffrey Way of Nettuts released what seems to be (so far) an excellent book on testing with laravel. It also has a fantastic overall introduction to testing practice in php, in terms even I can understand(!). So far it has demystified PHPUnit for me considerably and introduced me to things like Guard. If anyone is feeling uncertain about developing testable code and the tests to go with them, seems like a good place to start.

[ijy]

I saw the book but was somewhat disappointed it didn't cover more modern BDD. I'm new to testing but from what I'm learning at the moment I think the BDD approach is a better way as TDD has evolved. For a full-stack BDD approach Behat and PHPSpec2 would be my choice for learning on the PHP side. For JavaScript Jasmine seems good but Intern seems to be encompass the best of various alternatives and seems to have it all. Just my opinion from what I'm learning at the moment but obviously those with more experience will no doubt be able to identify the pros and cons. There certainly seems to be an embarrassment of riches on the testing side of things at the moment. Quite interesting to learn.

Some references from a learning/comparative point of view if anyone's interested.

REF

BDD: Behat & PHPSpec2

• http://welcometothebundle.com/phpunit-vs-phpspec-theory-on-behaviour-driven-development/
• http://techportal.inviqa.com/2011/08/03/outside-in-behaviour-driven-development-in-php-part-2/
• http://www.uvd.co.uk/blog/moving-to-bdd-in-php/
• http://everzet.com/post/31581124270/fullstack-bdd-2012-wrapup
• http://blog.karolsojko.com/index.php/2012/12/phpspec-how-ive-stopped-testing-and-started-specifying/
• http://techportal.inviqa.com/2011/07/27/behaviour-driven-development-in-php-with-behat/
• http://knpuniversity.com/screencast/behat/bdd

Intern

• http://www.sitepen.com/blog/2013/05/01/intern-javascript-testing/

[s-e]

@ijy The book actually seems quite paradigm agnostic, much more about the practicalities of writing the tests and configuring your setup for testing so far. The last few chapters cover Travis CI and Codeception, so the behavioural testing is there, although perhaps in a less "literate" form than Behat.

[ijy]

I was tempted to get it but wasn't sure if it would be useful or not but from what you say it could be worth the read from a testing integration point of view. You've sold me. :) Codeception is another new one which also looks pretty cool and Travis CI I really need to get my head around. I wonder if Jeff is doing free updates for life like Dayle is with his book?

[s-e]

Just doing some background reading myself, this is a nice article. It seems to me as if BDD and TDD solve different problems, and both have value in development. Where BDD lets you design and specify an application at a high level, TDD allows you to model the actual code much more effectively. One is top-down, the other bottom-up and using both will ensure you meet in the middle!

[s-e]

@ijy I think @JeffreyWay's book is considered a finished article, but hopefully he will correct errors and the like (I am yet to find one!). He is offering hugs for bug reports on it.

[ijy]

I guess it depends on preferred style and what you're used to but you're right they do seem to approach it from very different angles.

"The BDD tests what the object does instead of what it is, and what it does is much more important."

"In BDD you are not testing you are describing what that classes will do, and so you can continue creating the class (PHPSpec will create the class and the methods for you)".

I think I'd prefer to build as I go rather than fail and then fix but that's just from what I get from it. Big Red Angry Text and yelling that something is broken never seems very helpful or motivational. I want someone to tell me and create it if it's missing. :)

Ha, hugs for bugs. I'm sure he'll no doubt add to it if the requests come in or as things evolve even if it's no doubt planned. It'll probably be for the new edition cost though.

[JeffreyWay]

Hey, all -

Updates to the book will be free for life. I plan on writing a couple new chapters in the next two months on PHPSpec, which is pretty exciting. Think of it as Rspec for PHP. You guys should consider it.

About the BDD vs. TDD....it's important to keep in mind that, really, if you're doing TDD well, then you're doing BDD. Plenty of people would say that BDD is nothing more than really well-written TDD, and I agree. A lot of it just comes down to different terminology, though some specifically equate BDD to outside-in testing.

[ijy]

Hey @JeffreyWay thanks for chiming in. Well in that case I'm definitely sold and I'll buy it this afternoon. :) It seems like as thorough run down on testing across the board.

Interesting to hear your views on testing methodologies. BDD is meant to be TDD done right so I guess as long as you're doing it right in the first place the outcome is still the same. Albeit a different approach. Looking forward to the upcoming sections on PHPSpec.

[lewiswharf]

I bought the book yesterday and I'm not disappointed.

[creativedutchmen]

@JeffreyWay thanks for the awesome ebook. I've seen all the 'clean code talks' by Misko Hevery, but they still left me wanting more hands-on advice on how to structure things. Your book seems to fill in that spot perfectly.

A few bucks well spent, I'd say!

[iwyg]

I really recommend this book (german version here).

[iwyg]

Please head over to https://scrutinizer-ci.com/. We should consider using this along with travis.