Why I don't mock

http://blog.metaobject.com/2014/05/why-i-don-mock.html

by mpweiher • 12 years ago

90 88 12 years ago

blog.metaobject.com

clubhi• 12 years ago

Mocking helps with feature isolation. If a test failed you can be sure it was because of the use case of the test.

Mocking helps when you are calling into code that you can't control. E.g, when calling into an API that is broken and sometimes returns incorrect results.

I don't write very many tests anymore. I haven't mocked an object in years. I also have a lot of bugs in my code.

mpweiher2• 12 years ago

Mocking helps with feature isolation.

By enabling isolation in the test, mocking reduces the pressure to design more isolated code.

Mocking helps when you are calling into code that you can't control.

Pervasive mocking helps isolate you from the pressure to remove (deep) dependencies on code that you can't control. Redesign your code so it only interacts with such code at the edges. At those places, create APIs that are independent of the code you can't control, and adapt that code to fit your APIs.

pswenson2• 12 years ago

"Redesign your code so it only interacts with such code at the edges."

yeah, and what if you don't have time to do this? sometimes mocks are appropriate because of schedules and other priorities. Much better to mock than to not write tests because mocking is not pure.

SideburnsOfDoom9• 12 years ago

By enabling isolation in the test, mocking reduces the pressure to design more isolated code.

Having just written some tests (with mocks) today, I have read your comment several times and failed to extract any meaning whatsoever from it. It seems like the kind of comment that's counter-intuitive because it's just nonsense. Mocking reduces isolation by enabling isolation? Um.

Redesign your code so it only interacts with such code at the edges.

Yes, and those edges can then be mocked when you want to test the rest of the code. Apis, interfaces etc. are a good fit to mocking. Mocking helps isolation.

TFA's argument seems to be that they ended up not needing a database, and if they'd have "used mocking" they would have started by mocking the database, therefor mocking encourages bad design. Nope, starting with things that you don't need yet encourages bad design. It doesn't matter if you're mocking those things or not.

the_af3• 12 years ago

Pervasive mocking helps isolate you from the pressure to remove (deep) dependencies on code that you can't control. Redesign your code so it only interacts with such code at the edges. At those places, create APIs that are independent of the code you can't control, and adapt that code to fit your APIs.

Yes... and that process you describe can be accomplished easily using mocks or stubs (depending on what you wish to test).

mpdehaan23• 12 years ago

So I do a lot of systems programming type development.

In these cases, whether you're PXE booting a server or talking to 500 different APIs that all range between the latest database or cloud API, flashing firmware on a RAID controller (shudder), whatever, there's a point of diminishing returns. There's also most of the bugs that you won't be able to simulate, because they come from external factors.

The magic lies in automated integration tests.

QA automation is AMAZING stuff. Set up real servers and real test environments when you can. Test real database upgrades.

Mock testing can have it's place, but often you find yourself making internal API contracts more rigid than you want, which can limit refactoring.

humanrebar• 12 years ago

How do you test your system's response to error conditions? Do you have hooks in your integrated system to force errors (throw exceptions, violate contracts, etc.)?

mpdehaan23• 12 years ago

In some cases, yes.

You can do things like set up scenarios you know won't work, such as connecting to machines that don't exist or sending the wrong parameters.

The main thing is if the external inputs can't be made to "violate a contract", the internals aren't the problem.

Ultimately, it depends on what you are building. Simple CRUD web applications are different from systems development type applications in many cases. Many of these classes can be more easily mocked out.

Often the breaking change you'll see is an upstream API or SaaS endpoint changing in ways you didn't expect (or the difference between a Linux utility on the latest Ubuntu, an older RHEL, and Solaris), so it's often better to concentrate on testing the real thing.

And these tests are expensive, so an automated test matrix that deploys a VM fleet with lots of permutations is usually necessary.

pjc50• 12 years ago

I too do systemsish programming in C++. All the places I've ever done this have "unit" tests that link in almost all of the executable and inject test data through the normal frontend. There's very little mocking; sometimes there will be a setFooForUnitTests() function to short-circuit some normal configuration path.

goblin89• 12 years ago

Another magical thing (that I'm yet to try myself) is automated testing against live system. This ensures that the system actually works in production, not just on staging or test environment set up specifically for QA. I was initially against it, but reading about it recently[0] made me reconsider my opinion for some reason.

[0] Meditations by Sean Cassidy, http://blog.seancassidy.me/meditations.html

InclinedPlane• 12 years ago

A big danger with mocks is just shoving the problems under the rug rather than fixing them. Mocks represent assumptions, typically untested assumptions. There is generally nothing enforcing a mock's behaviors to be in accordance with whatever it is mocking, so that leaves it as a free parameter which can easily be incorrect if someone makes an error (which obviously can and does happen).

tomstuart5• 12 years ago

This isn’t a danger, it’s a benefit: your tests of one unit (whatever you consider a “unit” to be) are isolated from the behaviour of the rest of the system. For example, if you break one part of your implementation, you get one failing unit test, not a thousand.

Checking these assumptions is what integration and acceptance tests are for.

meric2• 12 years ago

I don't use mocks much, except with regard to the "now" method. There's some code that relies on the current time, I can use mock to mock many values of 'current time' instead of being forced to only test the case where the current time is when I run the unit test. This is especially important when dealing with code involving time zone and day light savings.

mpweiher2• 12 years ago

My experience with time is that whenever you build a system that is time-dependent in any meaningful way, you want to have "synthetic time" that you control, which you only feed with real time in a single, controllable place.

Otherwise you get nasty effects, such as time moving on while your system is processing etc.

meric2• 12 years ago

I'm guessing you mean like:

    def has_expired(now=None):
        #.. implementation

instead of:

    def has_expired():
        #.. implementation

But at some point you need to call that method somewhere else...

    def get_request_handler(request):
        #...code
        now = timezone.now()
        if has_expired(now=now):
           raise Error403.

And `get_request_handler` still needs to be tested, which I 'stub' the API for getting the current time in the correct timezone within the test for the request. (Taking into account the other comment), using a library called 'mock'. (https://pypi.python.org/pypi/mock)

nahname• 12 years ago

If you are only changing the value of what an API returns, that is not a mock. It is a stub.

blt• 12 years ago

I agree. But another option is to make all the methods that would call now() take an extra parameter of the current time. Talk about making your dependencies explicit...

GrinningFool• 12 years ago

But now you've...

a) changed your interface in a way that invites use cases you never intended (or wanted) to support.

b) added branching code anywhere that requires now()

nope_42• 12 years ago

I had a random idea the other day to create a program that monitors/logs system level calls and lets you replay them for a program. This would effectively let you use your program as normal, log any actual pieces that use external resources, and then automatically use the saved results as part of your test framework by overriding those system calls at the appropriate time.

Is anyone aware of a solution like this? I certainly don't have time to create it but I would definitely find it useful.

ap22213• 12 years ago

IIRC, SoapUI does something similar.

AlexanderDhoore• 12 years ago

We have something like this at the company I work for. We have a "simulation layer" which wraps ALL third party code. This lets us simulate the failure of file system calls, network drivers... And we use it to write our tests.

lucaspiller• 12 years ago

Although not quite the same, VCR for Ruby does this for network requests.

phaedrus• 12 years ago

I did this for a gdb wrapper I wrote. It logs all the stdin/stdout between itself and gdb and between itself and the user. You can then take that log file and replay it to simulate both the user input and the gdb output.

giulianob2• 12 years ago

I really want to see people's complex projects and example code. There have been way too much opinion on this topic lately without any concrete evidence to show the strengths/weaknesses of using or not using mocks.

pling3• 12 years ago

I can only give you an anecdote as our product is closed source. Mocks went like this, in a bell curve fashion:

1. Mocks are great, we'll use them everywhere.

2. Mocks are preventing our tests from being refactored.

3. Mocks are preventing our code form being refactored.

4. Mocks are preventing progress on the platform.

5. Code is being modified and tests and mocks are being removed to keep productivity up.

6. There are no mocks and few tests.

7. Tests increase again as defects are found. No mocks required.

This is over the space of 8 years and our problem was the amount of coupling they caused.

emsy3• 12 years ago

Mocks don't cause coupling. Programmers do.

giulianob2• 12 years ago

I have had the exact opposite experience. People weren't using mocks therefore when you changed something at a low level, it caused tests all over the system to break that didn't particularly care about that behavior. It also makes test data set up much more complicated for both unit and integration tests.

I think there might be subtle differences is how we are mocking objects and how we organize our code that might make mocking have different outcomes. That's why I would like to see actual examples that can be deeply analyzed. I suspect that it's in fact the way the code is written and what the underlying platform allows that makes one more suitable than the other rather than mocks being fundamentally a poor choice.

jerdfelt• 12 years ago

The Openstack project is a good example of a complex project with complicated testing. Merges are gated on successful test runs of a handful of different tests (unit, integration, etc)

An example of complicated mocking can be seen here:

https://github.com/openstack/nova/blob/master/nova/tests/vir...

In particular, SpawnTestCase.

Disclaimer: While I didn't write this example, I am responsible for some of the other mess

lmm3• 12 years ago

Mocks shouldn't be the first resort; they're there as a way to allow you to test code that you can't improve right now. I wouldn't want to use them on new code, but as a stepping-stone for testing and improving a large existing codebase they have their place.

tomstuart5• 12 years ago

That’s not what they’re for! Mocks allow you to design the interface of a new object by implementing that object’s collaborators before you implement the object itself. This is a very high-fidelity source of information about what the interface should look like; anything else is just (educated?) guessing.

mattdeboard2• 12 years ago

That may not be what they're "for" (wtf does that even mean?) but they're a damn good use of mocks. You can't fix everything right now.

lmm3• 12 years ago

A mock is overkill for that case; a stub or a dummy implementation (e.g. in-memory datastore) suffices when the interface with the collaborators is small and well-defined, as it should be when making a new design. The case where you need a full-on mock object is, IME, only when the collaborator is not decoupled enough from the primary object.

ChrisGaudreau• 12 years ago

Mocking is the primary problem with TDD as it exists right now. "Isolated testing" (or isolated TDD) is advocated by people like Uncle Bob. Many developers listen to Uncle Bob and end up believing that unit testing without complete isolation is not "real" unit testing.

Isolated unit testing is supposed to improve your designs. But when you mock everything, you end up with tests that know everything about your implementation (white box testing). The result is that refactoring breaks your tests. Your tests no longer document how to use the system; they are merely a mirror of its implementation.

Kent Beck's central point in his book is that TDD mitigates fear, allows refactoring, and gives you immediate feedback. Extensive mocking makes developers fearful to refactor (thus hurting design) and reduces the quality of feedback. Not to mention it makes your tests hard to refactor too.

Martin Fowler explains the difference between "classical" and "mockist" testing:

http://martinfowler.com/articles/mocksArentStubs.html#Classi...

CMCDragonkai• 12 years ago

I spent more time building my mocks instead of building my tests. Sometimes a file might have 90% mocks and 10% tests. Or maybe that's just my mocking framework...

glenjamin• 12 years ago

The idea is that if you end up with code like this, it shows you have too many collaborators and should refactor them away.

edgyswingset• 12 years ago

It could just be that the module of code under test doesn't do enough. If you need to mock a service, database, other service, error-handler, etc ... but the core logic is just verifying that <some condition> is met, then this is going to be the case.

Personally I like to avoid mocking for cases like this, since integration tests can usually suffice.

As you said, though, it could also be the framework. If you're in C#-land, Moq is pretty terse with the syntax so the actual mocking code isn't that big.

CmonDev2• 12 years ago

https://github.com/FakeItEasy/FakeItEasy

simplify• 12 years ago

I generally agree. I think Uncle Bob has recently realized all his advocating without public explanation causes a lot of confusion. He's been blogging quite a bit this past month, two posts of which are related to mocking:

http://blog.8thlight.com/uncle-bob/2014/05/10/WhenToMock.htm...

http://blog.8thlight.com/uncle-bob/2014/05/14/TheLittleMocke...

crashedsnow2• 12 years ago

I really don't get the debate. If you want to test state vs. behavior you still need to orchestrate and verify that state. You're still "mocking" because you're (presumably) not using a production system on which to test and you're using predictable, idempotent tests and test data. The only difference is you're not using a mocking "framework". Seems to me all you're doing is inheriting a lot of external dependencies that have little or nothing to do with your test. I don't see mocking as behavior vs. state, but rather just a way to remove dependencies that lead to brittle tests.

What am I missing?

falcolas• 12 years ago

you still need to orchestrate and verify that state.

Only if you limit yourself to testing using unit tests. Two of the strongest testing tools care not one bit about internal state.

Combinatorial testing tests all possible combinations of input pairs, and looks for the system to fail.

Fuzz testing creates a fairly large volume of noise as input, and looks for the system to fail.

Integration testing should be in your toolbox right alongside unit testing, and will exercise objects which are not appropriate for unit tests.

inheriting a lot of external dependencies that have little or nothing to do with your test.

I've found that this kind of thinking leads to development practices which rely on unit tests & production as their sole methods of testing. If a component is part of your production system, it should also be part of your test system.

jqjester• 12 years ago

In many cases, isolated testing does not require test doubles. There are alternatives to mocks: as long as the system under test is not manipulating its dependencies, you can get the state you need from the external resources and pass that into your classes.

CmonDev2• 12 years ago

A lot of unit tests are merely a poor man's design-by-contract / static analysis.

pswenson2• 12 years ago

Good points... but what happens when you walk into a legacy code base with few tests that wasn't designed to be tested and is tightly coupled to a component such as a database?

mattdeboard2• 12 years ago

There's a time and place for various tools. Mocking isn't all good, it's not all bad. Sometimes it's a good tool for accomplishing an immediate goal. Sometimes it's a crutch that encourages poor design. Keep an open mind and make evidence-based decisions. Only Siths deal in absolutes.

crashedsnow2• 12 years ago

"Do or do not. There is no try". Absolutely.

emsy3• 12 years ago

I found the article disappointing. Instead of giving reasonable arguments against mocking, the author offers an example where mocking was used wrong and therefore propagates that it shouldn't be used at all.

The problem is not mocking. I found it to be absolutely helpful if used right (and right often means sparsely). The problem is this "all or nothing" mentality that many programmers have. Either they mock the hell out of the codebase or they don't use it at all.

I don't use mock objects, unless it's message based business logic, often located at the topmost layer of abstraction.

programminggeek2• 12 years ago

I think OOP is exposing serious flaws in how humans write software and solve problems. I'm not sure, but it seems there is an impedance mismatch in how we write code, and how we'd like to test code, but there is a reluctance or outright defiance toward the idea of writing testable code.

The FP community doesn't seem to have a problem so much with testing simply because they mostly write pure functions that are data in -> data out and that is the easiest thing in the world to test.

It occurs to me that most developers aren't writing testable OO code, and thus we have a never-ending argument about the most clever way to test (or not test) code that is not easily tested in isolation.

If we never get around to writing testable code, we aren't going to have systems that are easy to test. If we don't have easy to test systems, testing is going to suck and people won't do it.

I don't know that FP is the answer, but just asking people to write better OO systems isn't working. The problem certainly is more fundamental than simply TDD or mocking or whatever language or tool we are using.

The problem is in how we look at solving problems in code and how that relates to system maintenance over time. It's a people problem, not a tooling one.

keeperofdakeys• 12 years ago

I don't think there is enough of an FP community doing large, heavily tested systems to allow accurate generalisations like that. I can think of plenty "We used FP, and everything went better than expected!" articles, but you probably don't hear from the people who tried, but came across problems.

programminggeek2• 12 years ago

I agree, and I don't know that FP is the solution.

I just know that the OO design approach isn't working super well with the people who are using it, and we now have a couple decades where design hasn't progressed significantly on the average project.

So, how to we change the human behavior that is leading to less than ideal outcomes?

camus2• 12 years ago

So, a lot of ideas are thrown here without any real use cases.People say "mocks are bad,whitebox testing is bad".But how can I unit test anything without mocks?

let's take javascript.I cant generate mocks from my code,i have to write them.I have this data access object that takes a connnection.

    var dao=new Dao(connection);
    dao.getById(1,function(err,res){ console.log(res) });

Tell me how do I unit test getById in isolation without mocking connection,without doing whitebox testing,without knowing what actually happens in my getById.

the obvious thing here is to spy on a hypothetic connection.query, fake a db call and return a predictable result.I cant do anything without a mock.

john2x• 12 years ago

I'm new to mocking (to serious unit testing, really), is mocking functions of 3rd-party libraries and/or web API's recommended? It sounds like a good idea, especially for functions that need to hit the network for a web API, but maybe I'm missing something?

michaelmior3• 12 years ago

I would say it's generally a bad idea to mock 3rd party libraries. It's code that's not under your control and testing with a mock which may not match the actual behaviour could cause you trouble. In the case of network requests, something like VCR[1] helps a lot.

[1] https://github.com/vcr/vcr

SideburnsOfDoom9• 12 years ago

I would disagree. If you want to unit-test your code that depends on those 3rd party libraries, then your choices are to:

1) Use the real thing. This can be slower and a lot less reliable.

2) Capture some typical output of the 3rd party api, and make a mock that emits it.

Which technique is useful? Both are. But the second one gives you fast-running fine-grained repeatable unit tests so therefore is your first line of testing.

So you discover in an integration test (#1) that the live api has a behaviour that the mock doesn't, maybe an occasional malformed response due to an error at the other end. Great, capture it and add it to the mock. Now you can reproduce it at will and write fast tests on how you handle it.

lmm3• 12 years ago

For something like a web API I think most people would support mocking. If it's just a library it's a lot less clear-cut; my approach would be to use the real thing until it gets to be a problem (because it's making your tests slow, or because it doesn't expose the right interface, or because it's too hard to get it into the state that you want to test). Certainly use mocks when you need to, but the rest of the time there are arguments for both sides and you probably need to figure out what you prefer yourself.

SideburnsOfDoom9• 12 years ago

TFA says "For example, when building SportStats v2 at the BBC we thought we needed a database for persistence. But we didn't build it in until we needed it, and we didn't mock it out either. We waited until the code told us that we now needed a database. It never did"

So they seem to be saying "if you write code upfront before you need it, you might not need it, therefor mocking is bad"

IMHO, that is not a valid conclusion to draw.

the_af3• 12 years ago

I agree with you, that seems to be what the author is saying, and his conclusion doesn't follow. Also, I'm not sure if he really means mocking or any kind of test doubles.

I thought everyone agreed that writing code upfront before you need it is always a bad idea, especially specific implementations like a database.

ytechie• 12 years ago

I don't understand how you can be 100% against mocking. When I want to visualize testing, I often find it useful to think about how it's applied in the real world. For example, when building a car, the parts are tested in isolation (for the most part). Individual parts are tested - size, shape, stress, etc. Groups of parts are also tested in isolation. For example, the engine is actually connected to a mock drivetrain, mock computer, mock user inputs, etc. I see that as useful, and I think the same goes for software. It can simply be too costly in terms of time, bandwidth, etc to test against real resources.

stinos• 12 years ago

While I think I get where the OP is going, this seems just too easy to claim without any real-life code. So let's try this then. The last time I used mocks: we're interfacing with motor control hardware.

In code we have interfaces for axis configuration, axis movement and so on. All implemented for different controllers. On top of that there's a layer defining specific movements. Say you want to go from point a to b while avoiding c, there's a piece of code that calculates a route (so it needs axes configurations for max velocity/range/...) and another piece that applies this route to a set of axes (so it needs axes to drive).

Apart from dragging around a bunch of motors everywhere I code, how am I supposed to properly test those two pieces without any mocks? Telling me the design is all wrong to start with is fine, but then you also have to come up with a better one of course :). Telling me that route calculation shouldn't depend directly on axis configuration but should use it's own configuration type and I should then add a conversion between the two is also no solution as it a) introduces an extra layer and b) still requires a mock to test the conversion, d'uh :)

trustfundbaby• 12 years ago

As always theres a middle ground between the extremes of MOCK-ALL-TEH-THINGS or mock nothing!!

Since reading Sandi Metz's very fantastic "Practical Object-oriented design in ruby" it clear to me that mocking is great for checking that objects other the one under test receive messages sent by the methods being tested (makes tests easier to setup and write usually). If I find that I'm doing too much mocking, then its usually because the method I'm using is doing way too much. FWIW.

krazydad• 12 years ago

Was hoping to read an admonition against excessive snark and bullying in Hacker News comments. Ah well.

webhat• 12 years ago

I dislike the "that's not supposed to throw an exception" developer statement, if something is not supposed to throw an exception under normal operating you can almost guarantee that it will throw exceptions in the demo. So one of the few cases where I use lots of mock is in Exception Handling.

Some exceptions are almost impossible to generate consistently in tests, or can't be simulated easily next to test that are supposed to pass when that exception state doesn't occur - think API failures, databases being yanked mid-query, etc.

Why I don't mock

Related Stories