Floehopper: Tag testing

Mocha Release 0.9.1

James Mead — Sat, 30 Aug 2008 16:48:00 +0000

Release Notes

Fixed bug #21465 – expects & stubs should support method names as strings (as well as symbols) or fail fast. Convert all expectation method names to a symbol in case they were supplied as a string.
By removing Mock#unexpected_method_called we reduce the number of methods vulnerable to the problem that surfaced in bug #21563.
Fix bug #21563 – stubbing ‘verified?’ method is unsafe. Instance method names on the Mock class should be more obscure.
Performance improvement. StubbaExampleTest goes twice as fast on my local machine.
Added primitive performance test to default rake task.
Fix format of case statements which don’t work in Ruby 1.9 and make others consistent.
There is no point in running (potentially expensive) checks if configuration is set to allow such checks to fail. This is a relatively quick fix in response to Chris McGrath’s performance problems.
Fix for bug #21161 – ‘uninitialized constant Deprecation in stubba.rb’.
It’s more readable to talk about ‘once’ and ‘twice’ rather than ‘1 time’ and ‘2 times’.
Fix bug #20883 – never should raise when called to prevent follow up errors. Fail fast when there are no matching invokable expectations and handle the stub_everything case sensibly. This might not be entirely backwards compatible, but I think the benefits outweigh the risks. The most likely change is that a test that was already failing will now fail faster, which doesn’t seem so awful.

Vim Learning Curve

James Mead — Sat, 12 Jul 2008 22:52:35 +0000

Because I’ve been doing a lot of remote pair programming using vim and screen, I’ve been making a real effort to improve my vim skills in order to be more productive and feel like I’ve made some progress. Chris has rightly suggested that the only way to really improve is to make vim your editor of choice even when not remote pairing, so that’s what I’ve been trying to do. I’ve come across a few useful but random bits and pieces which I thought I’d record in case they were useful to anyone else…

Textile

I’m using a vim syntax file for Textile written by Kornelius Kalnbach to write this article.

svn blame

I’ve found Tammer Saleh’s key mapping for svn blame useful…

  vmap gl :<C-U>!svn blame <C-R>=expand("%:p") <CR> \| sed -n <C-R>=line("'<") <CR>,<C-R>=line("'>") <CR>p <CR>

I decided to invest a bit of time understanding how it worked and broke it down as follows…

  # key mapping for visual mode
  vmap gl
  # remove all characters between the cursor position and the beginning of the line
  :<C-U>
  # current file with full path
  <C-R>=expand("%:p") <CR>
  # pipe std out to sed which only outputs line N to line M
  \| sed -n N,M
  # line number of first line of selection
  <C-R>=line("'<") <CR>
  # line number of last line of selection
  <C-R>=line("'>") <CR>
  # print output
  p <CR>

Search & Replace

At the end of last week, Chris & I found out from Ibrahim Ahmed’s blog how to search and replace across multiple files...

  # select file on which to operate
  :args path/with/wildcards
  # find pattern, replace with replacement, and save file
  :argdo %s/pattern/replacement/ge | update

vimdiff

vimdiff seems like a nice tool, but unfortunately it doesn’t immediately play well with svn diff and its --diff-cmd option. There seem to be a couple of alternatives: (a) write a wrapper script for vimdiff which works with --diff-cmd; or (b) write a script which uses svn cat and then calls vimdiff.

Test::Unit

One of the things I really want to be able to do, is run Ruby Test::Unit tests and view the output, so I can jump to the line where an assertion failed. I haven’t managed to find anything suitable on the web, so here’s my first attempt…

  function! Ruby_run_tests()
    let results = tempname()
    set splitbelow
    silent execute ":! ruby % > " . results . " 2>&1 "
    silent execute ":10 sview " . results
  endfunction

  map <silent> <F7> :call Ruby_run_tests()<cr>
  imap <silent> <F7> <ESC><F7>

I’m sure this isn’t the best way of doing it, but when you put this in your vimrc file, pressing the F7 key runs the current file using Ruby and pipes the results to a temporary file. This temporary file is opened in a read-only window 10 lines high at the bottom of the screen. By moving the cursor onto the relevant line of any stack trace, you can then use “goto file” key sequence gf to goto the failing assertion. Although for some reason this only seems to work the first time round for me.

Ideally I’d like to be able to do the equivalent of “run focussed test” in TextMate, which should be quite straightforward, but that’ll have to wait for another day.

In the meantime, I’d love to hear from anyone else who has useful vim tricks for Ruby development. I’ll be bookmarking any useful links I find on del.icio.us.

Mocha 0.9 Released

James Mead — Tue, 24 Jun 2008 19:14:00 +0000

There’s been quite a bit of work going on in Mocha over recent months, but a release is long overdue. The API is now pretty stable and so this release jumps from version 0.5 to 0.9. Much of the work has been refactoring Mocha’s internals to support new features and make the code more maintainable.

Before attempting the refactoring, extensive acceptance tests were added. One of the benefits of this is that it should now be easier to write new acceptance tests if you want to suggest new features or illustrate a bug ;-)

Here’s a quick summary of the changes in the release. I’ll try to post some code examples here in the near future.

Ordering constraints

Based on the JMock constraints with the same names…

Added “sequences” for constraining the order of expected invocations. See Standalone#sequence and Expectation#in_sequence.
Added “states” for constraining the order of expected invocations. See Standalone#states, Expectation#then, Expectation#when and StateMachine.

Configurable warnings or errors

When a method on a non-public method is stubbed
When a method on a non-existent method is stubbed
When a method on a non-mock object is stubbed (partial mocking)
When a method is stubbed unnecessarily (i.e. the stubbed method is not called during the test)

See Configuration for more details.

Improved error messages

A more readable and complete list of unsatisfied expectations, satisfied expectations and state machines.
Display more sensible failure message for any_instance expectations.

Parameter matchers

New to this release: optionally (allows matching of optional parameters if available), yaml_equivalent (allows matching of YAML that represents the specified object), responds_with (tests the quack not the duck).
Nesting of matchers is now supported.

Syntax shortcut

An optional block can be passed into the Standalone#mock method. The block is evaluated in the context of the new mock instance and can be used as a shortcut to set up expectations.

Ruby & Rails compatibility

Tested with Ruby 1.8.4, 1.8.5, 1.8.6 & 1.9. All related bugs and warnings believed to be fixed.

Tested with Rails 1.2.3 & Rails 2.1.0.

Deprecation

There is no longer any need to have a “require ‘stubba’” statement in your code. A deprecation warning has been added to this effect, because the file will be removed in a future release.

Bug fixes

Nasty Ruby Bug Affecting Test::Unit

James Mead — Sat, 05 Apr 2008 00:10:06 +0000

Introduction

Some time ago, while I was pair-programming with him, Chris alerted me to a Ruby bug he’d come across which was interfering with the diagnosis of a bug in our application. Since then I’ve tried to find out more about it, but couldn’t find much, so I’ve done a bit of investigation and thought I’d post it here in case it’s useful to anyone else. The bug has long since been fixed, but I’m sure there are still people our there with the affected versions of Ruby 1.8.6.

Ruby bug

As far as I understand it, the bug is in Ruby’s Kernel.at_exit hook. A call to Kernel.exit(false) should cause the process to exit with an exit status of 1 indicating the process did not complete successfully. The bug means that calling Kernel.exit(false) from within Kernel.at_exit incorrectly causes the process to exit with an exit status of 0.

The most relevant bug report is #9300 and the most relevant mailing list thread is made up of:- [ruby-core:10746], [ruby-core:10748], [ruby-core:10760].

The fix seems to be in changeset 12126…

r12126 | nobu | 2007-03-23 16:53:42 +0000 (Fri, 23 Mar 2007) | 9 lines

* eval.c (ruby_cleanup): exit by SystemExit and SignalException in END
  block.  [ruby-core:10609]

* test/ruby/test_beginendblock.rb (test_should_propagate_exit_code):
  test for exit in END block.  [ruby-core:10760]

* test/ruby/test_beginendblock.rb (test_should_propagate_signaled):
  test for signal in END block.

Implications for Test::Unit & Rake::TestTask

The bug has some important consequences. Test::Unit makes use of this mechanism to report test failures. Unfortunately, the bug means that a Test::Unit process will always return an exit status of 0 even when there have been test failures.

From test/unit.rb:

at_exit do
  unless $! || Test::Unit.run?
    exit Test::Unit::AutoRunner.run
  end
end

This in turn means that a Rake::TestTask process will also always return an exit status of 0 even when there have been test failures. This is significant, because many continuous integration systems rely on Rake::TestTask processes returning an exit status of 1 to indicate that there have been test failures. Thus you will get false positive passing builds – not good.

Affected versions of Ruby

I’ve built and installed a number of versions of Ruby and run tests on them to try to establish which ones are affected. Although they aren’t comprehensive, here are the results…

affected?	version
`N`	`ruby 1.8.4 (2005-12-24) [i686-darwin8.10.3]`
`N`	`ruby 1.8.5 (2006-08-25) [i686-darwin8.10.3]`
`N`	`ruby 1.8.5 (2007-03-16 patchlevel 37) [i686-darwin8.10.3]`
`N`	`ruby 1.8.5 (2008-03-03 patchlevel 115) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-02-17 patchlevel 0) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-03-16 patchlevel 2) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-03-19 patchlevel 4) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-05-22 patchlevel 5) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-05-22 patchlevel 6) [i686-darwin8.10.3]`
`Y`	`ruby 1.8.6 (2007-05-22 patchlevel 7) [i686-darwin8.10.3]`
`N`	`ruby 1.8.6 (2007-05-22 patchlevel 8) [i686-darwin8.10.3]`
`N`	`ruby 1.8.6 (2007-05-23 patchlevel 9) [i686-darwin8.10.3]`
`N`	`ruby 1.8.6 (2007-05-23 patchlevel 10) [i686-darwin8.10.3]`
`N`	`ruby 1.8.6 (2007-08-22 patchlevel 50) [i686-darwin8.10.3]`
`N`	`ruby 1.9.0 (2007-11-28 patchlevel 0) [i686-darwin8.10.3]`

Mock Object Injection

James Mead — Thu, 29 Nov 2007 12:17:48 +0000

A few months back, in my Introduction to Mock Objects talk at LRUG, I talked about “Mock Object Injection”. At the time I described a number of different ways of replacing a production object with a Mock Object using Mocha. I remember that at the meeting, James Adam (who has since joined the team at Reevoo) asked me why I didn’t like the Any Instance Stub Injection technique.

I’m not sure I gave him a very convincing response and I’ve been meaning for ages to have a better go at explaining what I think are the pros and cons of each of the techniques I mentioned. Here’s the list of techniques with the ones I like best at the top. I still haven’t done a very good job, but I’d be interested to hear what other people think so that I can try and improve my understanding.

Constructor Injection

The ClassUnderTest allows its dependencies to be passed in as parameters to its constructor. A mock object is passed in as a replacement for the “real” collaborator. It may be convenient to specify the production collaborator as a default parameter value.

Advantages

The dependencies of the ClassUnderTest are explicit.

Disadvantages

Can’t think of any at the moment.

  class ClassUnderTest
    def initialize(dependency = Collaborator.new)
      @dependency = dependency
    end
    def do_something
      # use @dependency
    end
  end

  collaborator = mock('collaborator')
  # constructor parameter injection
  instance_under_test = ClassUnderTest.new(collaborator)
  instance_under_test.do_something

Parameter Injection

The ClassUnderTest allows its dependencies to be passed in as parameters to the method under test. A mock object is passed in as a replacement for the “real” collaborator. It may be convenient to specify the production collaborator as a default parameter value.

Advantages

The dependencies of the method under test are explicit.

Disadvantages

Can’t think of any at the moment.

  class ClassUnderTest
    def do_something(local_dependency = Collaborator.new)
      # use local_dependency
    end
  end

  collaborator = mock('collaborator')
  instance_under_test = ClassUnderTest.new
  # method parameter injection
  instance_under_test.do_something(collaborator)

Stubbed New Method Injection

Use Mocha’s Object#stubs to temporarily replace Collaborator#new with a stub implementation that returns a mock object.

Advantages

Better than Any Instance Stub Injection, because you can have more control over different instances of Collaborator.

Disadvantages

Dependencies of the ClassUnderTest are hidden and not explicit.

  class ClassUnderTest
    def initialize
      @dependency = Collaborator.new
    end
    def do_something
      # use @dependency
    end
  end

  collaborator = mock('collaborator')
  # stubbed new method injection
  Collaborator.stubs(:new).returns(collaborator)
  instance_under_test = ClassUnderTest.new
  instance_under_test.do_something

Writer Method Injection

Use an attribute writer method to replace the “real” collaborator with a mock object.

Disadvantages

The ClassUnderTest has to unnecessarily expose a way to modify its internal state. The test is coupled to the implementation of the ClassUnderTest.

  class ClassUnderTest
    attr_writer :dependency
    def initialize
      @dependency = Collaborator.new
    end
    def do_something
      # use @dependency
    end
  end

  collaborator = mock('collaborator')
  instance_under_test = ClassUnderTest.new
  # writer method injection
  instance_under_test.dependency = collaborator
  instance_under_test.do_something

Stubbed Private Method Injection

Use partial mocking to temporarily replace a private builder method with a stubbed version of the method.

Disadvantages

The test is coupled to the implementation of the ClassUnderTest. The partial mocking of the instance_under_test means that the test is not testing a pristine instance of the ClassUnderTest, but a modified one. It also means that the boundaries between test code and production code are less clear.

  class ClassUnderTest
    def do_something
      local_dependency = build_collaborator()
      # use local_dependency
    end
    private
    def build_collaborator
      Collaborator.new
    end
  end

  collaborator = mock('collaborator')
  instance_under_test = ClassUnderTest.new
  # stubbed private method injection
  instance_under_test.stubs(:build_collaborator).returns(collaborator)
  instance_under_test.do_something

Any Instance Stub Injection

Use Mocha’s Class#any_instance method to temporarily replace the method on a collaborator with a stub method.

Disadvantages

The stubbed method is applied to all instances of the collaborating class. If the instance_under_test interacts with the stubbed method on more than one instance of the collaborating class, it isn’t possible to specify different behaviour for the stubbed method on each instance. Even if the instance_under_test only interacts with the stubbed method on one instance of the collaborating class, the test is specifying more stubbed behaviour than strictly necessary which could lead to false positives.

  class ClassUnderTest
    def do_something
      local_dependency = Collaborator.new
      return local_dependency.do_stuff
    end
  end

  # any_instance stub injection
  Collaborator.any_instance.stubs(:do_stuff).return('something useful')
  instance_under_test = ClassUnderTest.new
  instance_under_test.do_something

Instance Variable Set Injection

Use Object#instance_variable_set to replace the reference to a collaborator with a mock object.

Disadvantages

The test is coupled to the implementation of the ClassUnderTest. In particular the test is coupled to the supposedly private instance variable. In my opinion, it would be more honest to expose the instance variable by adding an attribute writer and using Writer Method Injection.

  class ClassUnderTest
    def initialize
      @dependency = Collaborator.new
    end
    def do_something
      # use @dependency
    end
  end

  collaborator = mock('collaborator')
  instance_under_test = ClassUnderTest.new
  # instance_variable_set injection
  instance_under_test.instance_variable_set(:@dependency, collaborator)
  instance_under_test.do_something

Mock Commands, Stub Queries

James Mead — Fri, 03 Aug 2007 10:10:00 +0000

Zach Moazeni has just posted a suggested patch for Mocha over on his blog. My understanding of the patch is that it means expectations are verified even when an assertion error occurs in the test. Here is his example…


  class Car

    def initialize(parts = [])
      @parts = parts
    end

    def start
      started = true
      @parts.each do | part |
        # commenting out for failure
        # started = started && part.start
      end

      started
    end

  end

  class SomeTest < Test::Unit::TestCase

    def test_start
      engine_mock = mock("engine_mock")
      car = Car.new([engine_mock])

      engine_mock.expects(:start).returns(false)
      assert !car.start
    end

  end

I’ve had a friendly & useful conversation with Zach about it, but I’m not convinced this is the right way to go. Using the one assertion per test school of thought, you can achieve the same goal by splitting the test into two so you get a test failure for the expectation and another for the assertion…


  class SomeTest < Test::Unit::TestCase

    def test_should_start_engine
      engine = mock('engine')
      car = Car.new([engine])

      engine.expects(:start)

      car.start
    end

    def test_should_start_if_engine_starts
      engine = stub('engine')
      car = Car.new([engine])

      engine.stubs(:start).returns(false)

      assert !car.start
    end

  end

Something that makes the example less suitable for mocking is that the Car#start method is both a command and a query. If you separate the two, testing with mocks might be easier…


  class Car

    def initialize(parts = [])
      @parts = parts
    end

    def start
      @parts.each { |part| part.start }
    end

    def started?
      @parts.all? { |part| part.started? }
    end

  end

  class SomeOtherTest < Test::Unit::TestCase

    def test_should_start_engine
      engine = mock('engine')
      car = Car.new([engine])

      engine.expects(:start)

      car.start
    end

    def test_should_not_be_started_if_engine_is_started
      engine = stub('engine')
      car = Car.new([engine])

      engine.stubs(:started?).returns(false)

      assert !car.started?
    end

  end

I’d be interested to know what other people think…

One thing I do agree with Zach about is that submitting a suggested patch to an open source project is a great way of initiating a constructive conversation.

Mock Objects in Ruby

James Mead — Sun, 22 Jul 2007 20:48:00 +0000

As promised I’ve finally got round to making the S5 slides of my LRUG talk available here.

You can use various keys to navigate the presentation. If you just want to quickly scan through the content, you might find the outline view useful (press the “T” key to toggle between slideshow & outline modes).

Mocha 0.5 released

James Mead — Fri, 08 Jun 2007 15:46:00 +0000

  sudo gem install mocha

or download one of the latest packages from rubyforge.

Parameter Matchers

I’ve added a few Hamcrest-style parameter matchers which are designed to be used inside Expectation#with. The following matchers are currently available: anything(), includes(), has_key(), has_value(), has_entry(), all_of() & any_of(). More to follow soon. The idea is eventually to get rid of the nasty parameter_block option on Expectation#with.

  object = mock()
  object.expects(:method).with(has_key('key_1'))
  object.method('key_1' => 1, 'key_2' => 2)
  # no verification error raised

  object = mock()
  object.expects(:method).with(has_key('key_1'))
  object.method('key_2' => 2)
  # verification error raised, because method was not called with Hash containing key: 'key_1'

Values Returned and Exceptions Raised on Consecutive Invocations

Allow multiple calls to Expectation#returns and Expectation#raises to build up a sequence of responses to invocations on the mock. Added syntactic sugar method Expectation#then to allow more readable expectations.

  object = mock()
  object.stubs(:method).returns(1, 2).then.raises(Exception).then.returns(4)
  object.method # => 1
  object.method # => 2
  object.method # => raises exception of class Exception
  object.method # => 4

Yields on Consecutive Invocations

Allow multiple calls to yields on single expectation to allow yield parameters to be specified for consecutive invocations.

  object = mock()
  object.stubs(:method).yields(1, 2).then.yields(3)
  object.method { |*values| p values } # => [1, 2]
  object.method { |*values| p values } # => [3]

Multiple Yields on Single Invocation

Added Expectation#multiple_yields to allow a mocked or stubbed method to yield multiple times for a single invocation.

  object = mock()
  object.stubs(:method).multiple_yields([1, 2], [3])
  object.method { |*values| p values } # => [1, 2] # => [3]

Invocation Dispatch

Expectations were already being matched in reverse order i.e. the most recently defined one was being found first. This is still the case, but we now stop matching an expectation when its maximum number of expected invocations is reached. c.f. JMock v1. A stub will never stop matching by default. Hopefully this means we can soon get rid of the need to pass a Proc to Expectation#returns.

  object = mock()
  object.stubs(:method).returns(2)
  object.expects(:method).once.returns(1)
  object.method # => 1
  object.method # => 2
  object.method # => 2
  # no verification error raised

This should still work…

  Time.stubs(:now).returns(Time.parse('Mon Jan 01 00:00:00 UTC 2007'))
  Time.now # => Mon Jan 01 00:00:00 UTC 2007
  Time.stubs(:now).returns(Time.parse('Thu Feb 01 00:00:00 UTC 2007'))
  Time.now # => Thu Feb 01 00:00:00 UTC 2007

Acknowledgements

Thanks to David Chelimsky, Dan North, Jay Fields, Kevin Clark, Frederick Cheung, James Moore, Brian Helmkamp, Ben Griffiths, Chris Roos & Paul Battley for their input. Apologies to anybody I forgot to mention.

RSpec/Mocha compatibility

James Mead — Wed, 04 Apr 2007 16:11:00 +0000

The RSpec team have just announced a beta version of their pending 0.9 release.

Apparently…

You can now use Mocha by saying
config.mock_with :mocha
in a
spec_helper

Sounds good :-)

Mocha 0.4 released

James Mead — Mon, 22 Jan 2007 12:28:00 +0000

So I finally got round to releasing a new version of Mocha. Much of the functionality has been available for some time if you’ve been using the Rails plugin based on subversion HEAD, but now you can get it in all in a gem (or other package). The most recent changes centre around allowing mocking of Object instance methods.

Release notes…

Allow naming of mocks (patch from Chris Roos).
Specify multiple return values for consecutive calls.
Improved consistency of expectation error messages.
Allow mocking of Object instance methods e.g. kind_of?, type.
Provide aliased versions of #expects and #stubs to allow mocking of these methods.
Added at_least, at_most, at_most_once methods to expectation.
Allow expects and stubs to take a hash of method and return values.
Eliminate warning: “instance variable @yield not initialized” (patch from Xavier Shay).
Restore instance methods on partial mocks (patch from Chris Roos).
Allow stubbing of a method with non-word chars in its name (patch from Paul Battley).
Removed coupling to Test::Unit.
Allow specified exception instance to be raised (patch from Chris Roos).
Make mock object_id appear in hex like normal Ruby inspect (patch from Paul Battley).
Fix path to object.rb in rdoc rake task (patch from Tomas Pospisek).
Reverse order in which expectations are matched, so that last expectation is matched first. This allows e.g. a call to #stubs to be effectively overridden by a call to #expects (patch from Tobias Lutke).
Stubba & SmartTestCase modules incorporated into Mocha module so only need to require ‘mocha’ – no longer need to require ‘stubba’.
AutoMocha removed.

Thanks to all who contributed.

Enjoy :-)