Why No One Uses Your Test Framework

http://www.slideshare.net/c.titus.brown/pycon-2012-tip-bof-intro-slides

by raymondh • 14 years ago

38 23 14 years ago

igorgue• 14 years ago

Well there are very good efforts to make this situation better like Konira (RSpec like Python DSL) and Lettuce (BDD but you can also use it for unit tests). But nobody really pays attention to it (well maybe just Yipit and me). And its authors are really cool and active, you can even tweet with them about problems you have, I even had conversations about testing theory (and nomenclature) with them.

jrockway3• 14 years ago

This is pretty much my impression every time I try to test something in Python. I've never liked xunit, and I especially dislike it when it's a direct port from Java. Yes, it's familiar to Java programmers. But when I'm writing Python, I want something that looks like Python. (XML DOM libraries also like to think they're Java. That makes Common Lisp my favorite language for munging XML, since it's the only language bold enough to come up with some other API. But I digress...)

Also, I don't understand mocks in Python. Say I have a class Foo that has a method "bar" that I want to override. OK:

   class FakeFoo(Foo):
      def bar(self):
          print "OH HAI"

Done. It's built in to the freakin' language :)

kbd2• 14 years ago

Well, here's Michael Foord's (@voidspace, referenced in the slides) Mock library so you can see what extra stuff it gives you.

http://www.voidspace.org.uk/python/mock/

raymondh2• 14 years ago

FWIW, Michael Foord's mock library has been approved for inclusion in the Python's standard library.

See http://www.python.org/dev/peps/pep-0417/ and http://www.gossamer-threads.com/lists/python/dev/973274

tdavis• 14 years ago

Now make some other code use your "mock" without editing its source. Now assert on the calls made to "bar()," including arguments. Now make it raise an exception without writing a second class. And make it return something else without writing a third. Now patch a built-in. And a context manager. And a dictionary. When you're done with that, mock a class without writing a new class at all.

If you honestly think that sub-classing is equivalent to mocking/stubbing, you don't understand the purpose of the activity.

jrockway3• 14 years ago

I know libraries can do all those things, but I've never preferred it over the real thing. As an example, I might do something like this (not Python):

   my $bar = 0;
   class FakeFoo {
      sub bar { return $bar }
   }

   my $thing = ThingToTest->new( foo => FakeFoo->new );
   $foo = 42;
   is $thing->make_bar_two_bigger, 44;
   $foo = -10;
   is $thing->make_bar_two_bigger, -12;

Mock enthusiasts would write:

   my $fake_foo = Mock->new( Foo->class )->instance;
   $fake_foo->return_sequence( bar => [42, -10] );
   my $thing = ThingToTest->new( foo => $fake_foo );
   is $fake_foo->make_bar_two_bigger, 44;
   is $fake_foo->make_bar_two_bigger, -12;
   ok $fake_foo->ensure_exhausted('bar');

Yes, it's less lines, but I think it's harder to read because the values are being set up long before they're used.

pjscott• 14 years ago

If you want more pleasant test-writing in Python, nose is commonly used, high quality, and compatible with the standard library's unittest package:

http://readthedocs.org/docs/nose/en/latest/

kbd2• 14 years ago

Thanks for the flashbacks from the TiP BoF. I thought we'd never get to the end, but man that was a fun and nerdy time.

pragmatic• 14 years ago

There seems to be a lot of subtext here that I'd love to see more about. I'm only an occasional python hacker and I'd love to know more about this.

One question, is there a web testing framework for python that they do like/recommend?

keypusher• 14 years ago

For a general purpose testing framework, Nose is great. For web testing, combine it with Selenium.

raymondh2• 14 years ago

Python's unittest.py was a direct port from Java's JUnit.

More recently, a Google contribution to unittest.py added a huge number of self.assertThisThatOrTheOther() methods. The new methods were designed to have better diagnostic messages for test failures (i.e. diffs between expected and actual).

Much needed test discovery was added by Michael Foord.

Third-party testing modules like py.test and nose.py are popular substitutes for unittest.py. They use a light-weight syntax and are joy to use.

jamescarr• 14 years ago

I really liked the shot about web testing. It's a crying shame that a lot of web testing frameworks don't bother supporting javascript.

tmcneal• 14 years ago

It's hard to get the full context of this presentation just from the slides, but I'm assuming the author is talking about Python test frameworks. I've been using nose in combination with coverage (http://pypi.python.org/pypi/coverage) for a year and have been very happy with it.

jarrett• 14 years ago

My pet reason: Your framework doesn't solve any of the genuinely hard problems that annoy us when we're trying to test stuff.

Lots of people have written variations on assertions and mocks. But where's the full-stack test harness for a Rails app with heavy JS usage, that runs at a reasonable speed, gives us useful traces on the client and server, and doesn't break when I slightly rework my HTML?

Or, where's the tool for testing integration with REST services outside your control, that doesn't depend on the real service most of the time, but can be told to run against the real service as a sanity check when you want to? (Like Ruby's VCR, but better.)

Or, how about a test framework that accurately emulates mobile browsers and automatically?

I'm not getting on anyone's case for not having written this stuff. Because hey, I'm just as guilty of not solving these problems as is the rest of the community. But if you want to make a testing tool that people will absolutely love, solve some of these problems really elegantly. (I know, easier said than done. And yes, some tools have solved some of these problems to some extent. But I'd consider each of my examples to be an ongoing pain point for developers.)

Fluxx• 14 years ago

> But where's the full-stack test harness for a Rails app with heavy JS usage, that runs at a reasonable speed, gives us useful traces on the client and server, and doesn't break when I slightly rework my HTML?

If you're talking Python, Lettuce does this is Python, there also is Splinter. You can run selenium with a headless firefox quite easily with Xvfb.

If you're talking rails, Capybara has drivers for Webkit (QI), selenium and even PhantomJS. Their speed is variable, but it's not too bad.

On the "HTML changes break my tests," I'd argue that A) you should write your front end tests not tied to the HTML if at all possible. Refer to elements by semantic names, not XPath or CSS selectors. And be smart about your steps. Make your submit_form() method actually look for a submit button in more than one way. B) you shouldn't have more than a handful of full stack frond end tests. They're hard to write, slow, and don't give much help telling you WHAT went wrong. Most of the time the ROI is only there for verifying a few user paths, but not much more than that.

benjiweber• 14 years ago

The Page Object Pattern can help reduce the brittleness of automated UI tests.

http://code.google.com/p/selenium/wiki/PageObjects

saurik2• 14 years ago

So, while I agree that it isn't right to "get on anyone's case for not having written this stuff", I do think it is reasonable to get on peoples' cases when they do release things if they didn't solve the "real problems": my message to people about to release a test framework would be "do you solve a real problem? if not, just keep it to yourself: please do not pollute the keyword space"; the corollary to this is that if you spent a weekend on it, it almost certainly does not solve a "real problem" ;P.

aaronblohowiak• 14 years ago

Let a hundred flowers bloom.

gaius• 14 years ago

No-one talk to me about unit testing unless you're already using strong typing.

troyjfarrell• 14 years ago

Python (the subject of the OP) is a strongly-typed, dynamically-typed language.

pbiggar• 14 years ago

I think it's pretty clear from context that the OP meant static typing.

Since he's being a dick about it, I very much understand your desire to pedantically point out his error of terminology. Unfortunately, "strong" typing is a very overloaded term, which has never had a widely agreed meaning amongst either academics or practitioners. There are generally two camps: "strong" === "static", and "strong" === "no dynamic coercions", and neither is technically incorrect.

Why No One Uses Your Test Framework

Related Stories