Floehopper: Tag unit

Nasty Ruby Bug Affecting Test::Unit

James Mead — Sat, 05 Apr 2008 00:10:06 +0000

Introduction

Some time ago, while I was pair-programming with him, Chris alerted me to a Ruby bug he’d come across which was interfering with the diagnosis of a bug in our application. Since then I’ve tried to find out more about it, but couldn’t find much, so I’ve done a bit of investigation and thought I’d post it here in case it’s useful to anyone else. The bug has long since been fixed, but I’m sure there are still people our there with the affected versions of Ruby 1.8.6.

Ruby bug

As far as I understand it, the bug is in Ruby’s Kernel.at_exit hook. A call to Kernel.exit(false) should cause the process to exit with an exit status of 1 indicating the process did not complete successfully. The bug means that calling Kernel.exit(false) from within Kernel.at_exit incorrectly causes the process to exit with an exit status of 0.

The most relevant bug report is #9300 and the most relevant mailing list thread is made up of:- [ruby-core:10746], [ruby-core:10748], [ruby-core:10760].

The fix seems to be in changeset 12126…

r12126 | nobu | 2007-03-23 16:53:42 +0000 (Fri, 23 Mar 2007) | 9 lines

* eval.c (ruby_cleanup): exit by SystemExit and SignalException in END
  block.  [ruby-core:10609]

* test/ruby/test_beginendblock.rb (test_should_propagate_exit_code):
  test for exit in END block.  [ruby-core:10760]

* test/ruby/test_beginendblock.rb (test_should_propagate_signaled):
  test for signal in END block.

Implications for Test::Unit & Rake::TestTask

The bug has some important consequences. Test::Unit makes use of this mechanism to report test failures. Unfortunately, the bug means that a Test::Unit process will always return an exit status of 0 even when there have been test failures.

From test/unit.rb:

at_exit do
  unless $! || Test::Unit.run?
    exit Test::Unit::AutoRunner.run
  end
end

This in turn means that a Rake::TestTask process will also always return an exit status of 0 even when there have been test failures. This is significant, because many continuous integration systems rely on Rake::TestTask processes returning an exit status of 1 to indicate that there have been test failures. Thus you will get false positive passing builds – not good.

Affected versions of Ruby

I’ve built and installed a number of versions of Ruby and run tests on them to try to establish which ones are affected. Although they aren’t comprehensive, here are the results…

affected?	version
`N`	`ruby 1.8.4 (2005-12-24) [i686-darwin8.10.3]`
`N`	`ruby 1.8.5 (2006-08-25) [i686-darwin8.10.3]`
`N`	`ruby 1.8.5 (2007-03-16 patchlevel 37) [i686-darwin8.10.3]`
`N`	`ruby 1.8.5 (2008-03-03 patchlevel 115) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-02-17 patchlevel 0) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-03-16 patchlevel 2) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-03-19 patchlevel 4) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-05-22 patchlevel 5) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-05-22 patchlevel 6) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-05-22 patchlevel 7) [i686-darwin8.10.3]`
`N`	`ruby 1.8.6 (2007-05-22 patchlevel 8) [i686-darwin8.10.3]`
`N`	`ruby 1.8.6 (2007-05-23 patchlevel 9) [i686-darwin8.10.3]`
`N`	`ruby 1.8.6 (2007-05-23 patchlevel 10) [i686-darwin8.10.3]`
`N`	`ruby 1.8.6 (2007-08-22 patchlevel 50) [i686-darwin8.10.3]`
`N`	`ruby 1.9.0 (2007-11-28 patchlevel 0) [i686-darwin8.10.3]`

Testing tidbits

James Mead — Tue, 25 Sep 2007 22:03:00 +0000

A couple of useful nuggets came up while I was pair programming with James today.

Firstly, James had an interesting new use of the Mocha parameter matcher anything, which reminded me of Joe Walnes and his Flexible JUnit assertions with assertThat().

Normally anything is used to determine what parameters are matched by an expectation…

  Product.expects(:do_stuff).with('name', anything)

But James was passing anything into a method to indicate that the value of that parameter was irrelevant in the context of the test. The kind of thing you might already do using nil...

  Product.do_stuff('name', anything)

I wonder if this idea could be extended by having a kind of holy hand-grenade object which “blows up” by raising an AssertionFailedError when any method is invoked on it…?

Secondly, we were running all our tests using the default rake task when we discovered a test that was hanging. The standard rake test task output wasn’t very helpful in identifying which test was stuck (just rows of dots)...

Started
.....................................................................
.....................................................................
.....................................................................
......................................................

Interrupting the process and examining the stack trace didn’t help either (the relevant stack frames had been collapsed)...

   ... 24 levels...
  from /opt/local/lib/ruby/1.8/test/unit/autorunner.rb:216:in `run'
  from /opt/local/lib/ruby/1.8/test/unit/autorunner.rb:12:in `run'
  from /opt/local/lib/ruby/1.8/test/unit.rb:278
  from /opt/local/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/...

So we set the output to verbose which outputs the name of each test as it starts. From this we could work out which was the offending test…

  TESTOPTS="-v" rake

  test_should_do_something(NaughtyTest): .
  test_should_do_something_else(NaughtyTest): .
  test_should_do_something_ideally_without_hanging(NaughtyTest): .

Stub Queries and Expect Commands

James Mead — Sat, 23 Dec 2006 20:02:00 +0000

In Making a Mockery of ActiveRecord, a single test has multiple expectations set up for accessor methods…

  specify "should create EmailMessages and Subscriptions when include_subscribers is true" do
    @message.include_subscribers = true
    @message.should_receive(:lists).and_return([@list])
    @message.should_receive(:campaign).twice.and_return(@campaign)
    @list.should_receive(:people).and_return([@person])
    @person.should_receive(:campaigns).and_return([])

    @person.should_receive(:subscriptions).and_return(@subscriptions)
    @subscriptions.should_receive(:create).and_return(nil)
    @campaign.should_receive(:people).and_return([])

    message.generate # I think this line is missing from the original test
    @message.should_have(1).email_messages
  end

I’ve found Nat Pryce’s rule of thumb – stub queries and expect commands – to be very useful in reducing the brittleness of tests (see Yoga for Your Unit Tests).

A query in this context is a method which does not change the state of the object on which it is called. The accessor methods definitely fall into this category, so I would stub them, not set expectations for them. The one command method in the above example which merits an expectation is the call to create on @subscriptions.

However, I prefer to use in-memory ActiveRecord objects instead of mocks or stubs wherever possible. So I’d write something more like this (using Test::Unit and Mocha)...

  def test_should_create_email_messages_and_subscriptions_when_include_subscribers_is_true
    subscriptions = mock()
    person = Person.new(:campaigns => [])
    person.stubs(:subscriptions).returns(subscriptions)
    list = List.new(:people => [person])
    campaign = Campaign.new(:people => [])
    message = Message.new(:lists => [list], :campaign => campaign)
    message.include_subscribers = true

    subscriptions.expects(:create)

    message.generate
    assert_equal 1, message.email_messages
  end

I haven’t actually run this – so it’s quite likely there are errors in it, but you should get the general idea. The person.stubs line is necessary to avoid the type checking that Luke Redpath mentions in his comment.

More thoughts here.

Fixtures, mock objects or in-memory ActiveRecord objects?

James Mead — Sat, 23 Dec 2006 20:00:00 +0000

Wilson Bilkovich has posted an article about mocking ActiveRecord objects.

A new method mock_model is defined that builds a mock object which will respond the same way as a real ActiveRecord object. As I understand it, this means he can replace…

  @campaign = mock("campaign")
  @campaign.stub!(:is_a?).and_return(true)
  @campaign.stub!(:new_record?).and_return(false)
  @campaign.stub!(:id).and_return(rand(1000))

with…

  mock_model :campaign

Although I agree with him that using fixtures is not a good idea, why not use a real ActiveRecord object…

  @campaign = Campaign.new

Sometimes due to the way ActiveRecord couples your models to the database, it becomes essential to have a model in the database and not just in memory. In which case why not just do this…

  @campaign = Campaign.create!

I’ve written up a couple more thoughts here and here.

Mocha in Java using bytecode manipulation

James Mead — Sat, 16 Sep 2006 14:43:00 +0000

My former colleague and bytecode maestro Stacy Curl has an interesting article explaining how it would be possible to implement Mocha in Java using experience from his PicoUnit project.

It’s good to be reminded that meta-programming is possible in Java, but just a bit more effort.

ThoughtWorks team uses Mocha

James Mead — Tue, 05 Sep 2006 06:35:00 +0000

It’s good to see that a team from my old company has caught the Mocha bug…

Jay Fields describes how his team have been using Mocha to achieve Ruby on Rails Unit Testing in less than 1 second.

Unit tests - Can 100% coverage be a bad thing?

James Mead — Fri, 14 May 2004 12:12:37 +0000

Consider what happens when you have “unit” tests that go more than one unit deep (in call stack terms) i.e. tests that don’t mock out all units external to the unit under test. Although such tests may give you good coverage…

They usually require complex setup code which means it can be hard to identify the intention of a test. It’s generally accepted that self-documenting code is a good thing – unit tests with clear intent can be one of the best ways to achieve such documentation.
They are by definition more fragile than a true unit test, because they’re vulnerable to changes in multiple units. Also, the more of these tests you have, the more likely it is that a number of tests will be vulnerable to changes in a single unit.
When they fail, it often proves difficult to identify the reason for the failure. This difficulty is often a consequence of the unclear intention of the test. A failure caused by an exception thrown deep in the call stack (rather than an assertion in the test) can lead to similar problems.
They usually take longer to run, partly by definition because they execute more code, but also because they often run against a real database or other external resource.
All of these issues will act as a barrier to refactoring. Faced with the choice of performing a sensible refactoring that breaks a number of such tests or adding duplicate logic to a system, many developers will understandably choose to add the duplicate logic. This can start a spiral that leads to the rapid build up of crippling technical debt.

I call that a bad thing!

Update: I see that Tim Bacon has already touched on this subject.