Floehopper

Nasty Ruby Bug Affecting Test::Unit

Posted by James Mead Sat, 05 Apr 2008 00:10:06 GMT

Introduction

Some time ago, while I was pair-programming with him, Chris alerted me to a Ruby bug he’d come across which was interfering with the diagnosis of a bug in our application. Since then I’ve tried to find out more about it, but couldn’t find much, so I’ve done a bit of investigation and thought I’d post it here in case it’s useful to anyone else. The bug has long since been fixed, but I’m sure there are still people our there with the affected versions of Ruby 1.8.6.

Ruby bug

As far as I understand it, the bug is in Ruby’s Kernel.at_exit hook. A call to Kernel.exit(false) should cause the process to exit with an exit status of 1 indicating the process did not complete successfully. The bug means that calling Kernel.exit(false) from within Kernel.at_exit incorrectly causes the process to exit with an exit status of 0.

The most relevant bug report is #9300 and the most relevant mailing list thread is made up of:- [ruby-core:10746], [ruby-core:10748], [ruby-core:10760].

r12126 | nobu | 2007-03-23 16:53:42 +0000 (Fri, 23 Mar 2007) | 9 lines

* eval.c (ruby_cleanup): exit by SystemExit and SignalException in END
  block.  [ruby-core:10609]

* test/ruby/test_beginendblock.rb (test_should_propagate_exit_code):
  test for exit in END block.  [ruby-core:10760]

* test/ruby/test_beginendblock.rb (test_should_propagate_signaled):
  test for signal in END block.

Implications for Test::Unit & Rake::TestTask

The bug has some important consequences. Test::Unit makes use of this mechanism to report test failures. Unfortunately, the bug means that a Test::Unit process will always return an exit status of 0 even when there have been test failures.

at_exit do
  unless $! || Test::Unit.run?
    exit Test::Unit::AutoRunner.run
  end
end

This in turn means that a Rake::TestTask process will also always return an exit status of 0 even when there have been test failures. This is significant, because many continuous integration systems rely on Rake::TestTask processes returning an exit status of 1 to indicate that there have been test failures. Thus you will get false positive passing builds – not good.

Affected versions of Ruby

I’ve built and installed a number of versions of Ruby and run tests on them to try to establish which ones are affected. Although they aren’t comprehensive, here are the results…

Tags bug, code, continuous, exit, hook, integration, ruby, test, testing, unit | no comments

Testing tidbits

Posted by James Mead Tue, 25 Sep 2007 22:03:00 GMT

A couple of useful nuggets came up while I was pair programming with James today.

Firstly, James had an interesting new use of the Mocha parameter matcher anything, which reminded me of Joe Walnes and his Flexible JUnit assertions with assertThat().

Normally anything is used to determine what parameters are matched by an expectation…

  Product.expects(:do_stuff).with('name', anything)

But James was passing anything into a method to indicate that the value of that parameter was irrelevant in the context of the test. The kind of thing you might already do using nil...

  Product.do_stuff('name', anything)

I wonder if this idea could be extended by having a kind of holy hand-grenade object which “blows up” by raising an AssertionFailedError when any method is invoked on it…?

Secondly, we were running all our tests using the default rake task when we discovered a test that was hanging. The standard rake test task output wasn’t very helpful in identifying which test was stuck (just rows of dots)...

Started
.....................................................................
.....................................................................
.....................................................................
......................................................

Interrupting the process and examining the stack trace didn’t help either (the relevant stack frames had been collapsed)...

   ... 24 levels...
  from /opt/local/lib/ruby/1.8/test/unit/autorunner.rb:216:in `run'
  from /opt/local/lib/ruby/1.8/test/unit/autorunner.rb:12:in `run'
  from /opt/local/lib/ruby/1.8/test/unit.rb:278
  from /opt/local/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/...

So we set the output to verbose which outputs the name of each test as it starts. From this we could work out which was the offending test…

  TESTOPTS="-v" rake

  test_should_do_something(NaughtyTest): .
  test_should_do_something_else(NaughtyTest): .
  test_should_do_something_ideally_without_hanging(NaughtyTest): .

Tags anything, matcher, options, rake, test, testopts, unit, verbose | 2 comments

Stub Queries and Expect Commands

Posted by James Mead Sat, 23 Dec 2006 20:02:00 GMT

In Making a Mockery of ActiveRecord, a single test has multiple expectations set up for accessor methods…

  specify "should create EmailMessages and Subscriptions when include_subscribers is true" do
    @message.include_subscribers = true
    @message.should_receive(:lists).and_return([@list])
    @message.should_receive(:campaign).twice.and_return(@campaign)
    @list.should_receive(:people).and_return([@person])
    @person.should_receive(:campaigns).and_return([])

    @person.should_receive(:subscriptions).and_return(@subscriptions)
    @subscriptions.should_receive(:create).and_return(nil)
    @campaign.should_receive(:people).and_return([])

    message.generate # I think this line is missing from the original test
    @message.should_have(1).email_messages
  end

I’ve found Nat Pryce’s rule of thumb – stub queries and expect commands – to be very useful in reducing the brittleness of tests (see Yoga for Your Unit Tests).

A query in this context is a method which does not change the state of the object on which it is called. The accessor methods definitely fall into this category, so I would stub them, not set expectations for them. The one command method in the above example which merits an expectation is the call to create on @subscriptions.

However, I prefer to use in-memory ActiveRecord objects instead of mocks or stubs wherever possible. So I’d write something more like this (using Test::Unit and Mocha)...

  def test_should_create_email_messages_and_subscriptions_when_include_subscribers_is_true
    subscriptions = mock()
    person = Person.new(:campaigns => [])
    person.stubs(:subscriptions).returns(subscriptions)
    list = List.new(:people => [person])
    campaign = Campaign.new(:people => [])
    message = Message.new(:lists => [list], :campaign => campaign)
    message.include_subscribers = true

    subscriptions.expects(:create)

    message.generate
    assert_equal 1, message.email_messages
  end

I haven’t actually run this – so it’s quite likely there are errors in it, but you should get the general idea. The person.stubs line is necessary to avoid the type checking that Luke Redpath mentions in his comment.

More thoughts here.

Tags activerecord, command, mock, query, stub, test, unit | 1 comment

Fixtures, mock objects or in-memory ActiveRecord objects?

Posted by James Mead Sat, 23 Dec 2006 20:00:00 GMT

Wilson Bilkovich has posted an article about mocking ActiveRecord objects.

A new method mock_model is defined that builds a mock object which will respond the same way as a real ActiveRecord object. As I understand it, this means he can replace…

  @campaign = mock("campaign")
  @campaign.stub!(:is_a?).and_return(true)
  @campaign.stub!(:new_record?).and_return(false)
  @campaign.stub!(:id).and_return(rand(1000))

with…

  mock_model :campaign

Although I agree with him that using fixtures is not a good idea, why not use a real ActiveRecord object…

  @campaign = Campaign.new

Sometimes due to the way ActiveRecord couples your models to the database, it becomes essential to have a model in the database and not just in memory. In which case why not just do this…

  @campaign = Campaign.create!

I’ve written up a couple more thoughts here and here.

Tags fixture, mock, object, ruby, stub, test, unit | 1 comment

Mocha in Java using bytecode manipulation

Posted by James Mead Sat, 16 Sep 2006 14:43:00 GMT

My former colleague and bytecode maestro Stacy Curl has an interesting article explaining how it would be possible to implement Mocha in Java using experience from his PicoUnit project.

It’s good to be reminded that meta-programming is possible in Java, but just a bit more effort.

Tags bytecode, dependency, injection, java, meta, mocha, mock, pico, programming, ruby, unit | no comments

ThoughtWorks team uses Mocha

Posted by James Mead Tue, 05 Sep 2006 06:35:00 GMT

It’s good to see that a team from my old company has caught the Mocha bug…

Jay Fields describes how his team have been using Mocha to achieve Ruby on Rails Unit Testing in less than 1 second.

Tags mocha, mock, rails, ruby, test, thoughtworks, unit | 2 comments