When to Mock

https://enterprisecraftsmanship.com/posts/when-to-mock/

by vkhorikov • 6 years ago

59 79 6 years ago

pgt• 6 years ago

Panic closed the tab when a giant form popup appeared asking for my email. Hate to add noise to the discussion, but I hope the author sees this and understands the market doesn't want this and reduces readership.

corebit• 6 years ago

Same. I wish HN had a filter I could set up to hide entries form domains that do this. Won't catch one offs like this but I can add it along with NYT and other paywalls so I don't waste my time.

xhrpost• 6 years ago

These kind of CTAs are everywhere these days and it's rather frustrating. Wish there was some solution to filter out sites that use completely not-ignorable (like in the corner) CTAs.

SAI_Peregrinus• 6 years ago

Wish there was some solution to filter out sites that use completely not-ignorable (like in the corner) CTAs.

uBlock Origin, if the default filter lists don't catch it just right-click and block the element.

john-radio• 6 years ago

ublock origin seems to have stepped on it; I didn't encounter this popup.

mabbo• 6 years ago

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying.

Use real instances of managed dependencies in tests.

Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code may depend on an large, complex database. I could have a continually running instance just for my unit tests, but that's expensive (time, money, hosts, etc) and means I can't run my unit tests if the wifi goes down.

If we're talking about large, complete systems running very large suites of system tests as part of a CI/CD pipeline, then sure, I'm on board. Let's use a real database. But there's lots of other small, simple tests that absolutely don't need anything more than a mock while I work on business logic bugs.

And no, I'd not like to get into a debate of "you shouldn't bother to write small unit tests". I find them very useful.

jaitsu• 6 years ago

Writing unit tests is debatable?

ashtonkem• 6 years ago

That argument gets floated occasionally.

The utility of a unit test suite is roughly proportional to the amount of in-process computation you do. “Glue” systems which mostly transform from one protocol to another usually need more integration systems than unit, making skipping unit tests not disastrous.

jerf• 6 years ago

Oh, yes. We've still got a bunch of Real Programmers [1] out there. Real Programmers have moved on in my lifetime from insisting that everything should be written in assembly to insisting that they're Real Programmers enough to handle C directly and anyone who writes undefined behavior or buffer overflows or memory management issues just aren't Real Programmers enough and should put down their keyboards and walk away in shame, perhaps taking up goat farming or plumbing.

In all seriousness, their numbers seem to be diminishing. But you certainly can't expect the Real Programmers to write test code. Why should a programmer write test code when they know they didn't write any bugs?

[1] http://catb.org/~esr/jargon/html/R/Real-Programmer.html - I'd say as time goes on, this term is relative.

commandlinefan3• 6 years ago

On HN, it's not debatable: you should definitely do it. Everywhere I've ever worked, it's also not debatable: no time or effort should be "wasted" writing unit tests, that's what QA is for.

Tomis023• 6 years ago

Unit tests are code, and all code has a cost. Writing unit tests without any benefit is harmful behaviour. Real world example: testing dumb getters/setters.

uk_programmer• 6 years ago

Yes. For a lot of smaller (web) applications where it is obvious if it isn't working correctly it just become a time sink. Everything in engineering is a set of trade-offs.

With regards to TDD specifically. It isn't for everyone and many consider it to be a bit of a cult. While I don't consider it a cult, It doesn't work with how I personally solve problems. I normally for example get something extremely rough working and then iterate until I consider it to be perfect and then write my tests to define how it should behave.

squiggleblaz2• 6 years ago

On legacy code? Alas. The amount of changes you might have to make to make the legacy code unit testable versus the benefit of unit tests can be exceedingly significant.

For instance, if the implementation involves calling up a bunch of value objects from a database, each of which do the same, and all of the code is inhouse so there is no standard mock or stub libraries available, adding unit tests is tantamount to rewriting the whole system without tests.

Existing codebases can also be too complicated for a few people to formalise into unit tests. The algorithm itself might be simple, but again, to discover that, you need to rewrite the whole system without tests. (You can add tests, of course, but when you're adding tests, you're making an assertion you cannot prove. Since you don't know what the code is meant to do, you don't know whether the tests are complete or even accurate.)

Once you're coding in a world where tests passing or tests failing has almost no predictive value on success or failure in release, you're working in a self-fulfilling prophecy where the code will never be tested because the tests literally make things worse.

Many codebases clearly do not have any automated tests at all, but unit tests can be the hardest to add onto a system after the fact.

bluejekyll4• 6 years ago

Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about.

They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system.

The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less clear it becomes where errors were introduced.

sanderjd• 6 years ago

Do people usually mock the database at the integration / functional level? I haven't seen that... I interpreted this to mean the unit test level, which similarly to the thread starter, seems crazy to me.

pydry2• 6 years ago

However, the further down the pyramid you go:

* The more unrealistic the tests generally become (larger % of false positives - tests that fail when they shouldn't - and false negatives - unit tests that simply don't catch bugs at all).

* The less reusable the test infrastructure becomes. Stubbing/mocking individual method calls to the database is an ongoing cost of development whereas building scripts to start the database and shut it down is an investment cost that pays dividends.

On the whole I think the pyramid idea enforces a wrong view that there is a "right" mix of "test levels" across any project. The best mix is determined by the kind of bugs and code you have (integration vs logical) and the kinds of abstractions you need or already have (in general, the worse your abstractions, the higher level you need your tests to be).

A lot of projects are best done with 100% integration tests while others can be done with 100% unit (especially small, self contained, simple-to-interact-with code bases that are 99% about calculations/logical decision making).

kerkeslager• 6 years ago

The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less clear it becomes where errors were introduced.

I disagree. A failing unit test doesn't even necessarily indicate that an error was introduced: if a unit doesn't do what it's supposed to but the user doesn't see that, then an error wasn't introduced. Sure, when a unit test fails there's often an error, but if an end-to-end test fails, there's always an error--E2E tests are testing from the user's perspective, so what they're testing is actually errors. (This is assuming that both the unit tests and the E2E tests are correctly written).

You're positioning unit tests as a debugging tool, but I'd argue that there are much better debugging tools: REPLs and debuggers give you a lot more information than a unit test, and allow you to ask new questions quickly.

I don't want to come across as being anti-unit tests. On the contrary, I think unit tests are highly valuable. But I don't think the value comes from gathering information, debugging, or even catching bugs (in most cases). I think the value comes from a few things:

1. TDD forces you to design units for reuse from the start. Immediately you're using the code in two contexts: the application and the unit test. So right away your code is inherently reusable (in a binary sense) because de-facto you've re-used it. Reusability is more complicated than that (it's really more of a spectrum than a binary) but having at least two uses from the beginning pushes you toward the reusable side of the spectrum.

2. Unit tests act as living documentation for units. It is often unclear by reading code what the code does, because production concerns such as performance and security can lead you to do things in seemingly complicated ways. But unit tests don't have these concerns (at least not in the same way) so you can write code in unit tests that clearly communicates what a unit does. And unit tests don't fall out of sync with code like plain text documentation does.

3. TDD is incredibly motivating. Moving red->green on a quick cycle takes advantage of the dopamine reward system to increase productivity.

barrkel• 6 years ago

Unit tests on code that has no dependencies (and I mean no dependencies, neither injected nor direct) is great.

Unit tests on code with dependencies (whether injected so that they can be mocked, or directly referenced so that they end up more like mini integration tests) are less excellent. They're brittle, inhibit refactoring, and either don't test as much as you think they do (if mocking dependencies) or are slow (if not mocking).

The further up the testing pyramid you go, the less work it is to refactor things, because you don't need to rewrite as many tests. OTOH test are more complex to write and take longer to run.

And now I get to my point: I don't think the blanket statement of "your largest set of test should be ... unit testst that run quickly" is well-founded. There are trade-offs, and they shouldn't be trivially waved aside.

grey-area2• 6 years ago

When you're developing, aren't you running a db server with a dev database for that? Just add another separate test db and use that for tests where appropriate.

Also, most unit tests don't need a db, it's more useful for integration tests.

pc862• 6 years ago

If you're using a Repository pattern or similar you'll have a lot of unit tests where a database or db provider is an injected dependency.

amw-zero3• 6 years ago

Hitting a database in tests is an order of magnitude slower and also involves worrying about a lot more code, i.e. the database engine. You may or may not want to take on that burden.

davedx2• 6 years ago

Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test?

So what we do where I work is for unit tests, we mock services and repositories, unit tests don't go anywhere near the database.

For integration tests, we use an in memory database.

BUT! Be careful before you embark down the path of running integration tests against a different database than the one you use when running the application. There are SO many pitfalls, nasty bugs, and other warts along that road. EntityFramework alone has quite some weirdness there. Expect these kinds of integration tests to cost a lot of developer effort to build and maintain. For us, it took months of effort to get these kinds of tests working usefully.

Personally after working on applications that have a solid test pyramid, I would recommend:

* Write unit tests as per the standard advice (bottom of pyramid kind of quantity), but try and keep it sane (don't go for 90% coverage just for the sake of hitting an arbitrary number; don't pointlessly unit test your framework/libraries/other dependencies)

* Write some integration tests where they really add value (interaction between 2 or a few complicated components in your system, for example places where the state of a component changes a lot depending on input from another component). Make sure when you start out writing an integration test that it doesn't turn into an "almost" end-to-end test along the way. They have a habit of doing this and it can really cost you later. Integration tests should still be focused.

* Write end-to-end tests that test as much of your system as possible, including a database (preferably the same vendor; one approach is to truncate all tables before each run). IME it is very good here to have one e2e test that covers a big chunk of functionality in one run, than lots of separate tests covering different things. Why? Because no matter how "different" the things being tested by the e2e test there will still be LOTS of overlap by its very nature, and changing anything where that overlap is will tend to break ALL your e2e tests. Not a fun workflow.

One last comment on E2E tests. This is more opinionated. But try and limit E2E tests to things that are really business critical (e.g. your "sign up" process and your "renew subscription" process, but not every single form in your dashboard). This is just a cost-benefit thing. E2E tests help, but they also slow you down, and the more you have, the more slowly you will be able to change your software. Sometimes minor bugs slipping through is an acceptable cost if it means you release 1 week sooner.

matwood• 6 years ago

For integration tests, we use an in memory database.

I think this can work for running the tests locally when speed is what you want, but you should still have the tests run on the actual database on commits. It requires keeping a database running just for tests to run (it can be a smaller instance and startup/shutdown on demand), but it will save so much pain later.

dtech2• 6 years ago

With docker and the other arguments the article points out I find there's very little value in an in-memory database. Just use either a stub/mock/dummy for unit test or spin up the real thing for integration test.

pc862• 6 years ago

I've yet to have a good testing experience when using anything more than a trivial database if it has EntityFramework on top. The de facto standard seems to be Sqlite for unit tests, which causes some blocking issues if you have the same database name in two different schemas, or composite primary keys, to use my two most recent examples.

tybit• 6 years ago

A big part of the authors unit testing philosophy is that you should separate IO from business logic.

That way you can unit test without mocks and without heavy real dependencies either, and leave that for integration tests.

je42• 6 years ago

yep. the test pyramid helps figuring when to use what test setup. https://martinfowler.com/bliki/TestPyramid.html

amw-zero3• 6 years ago

What I got out of the article was that, in unit tests, you should stub the response of a database query. However, you shouldn't directly assert that some database API method is called, because that is one additional level of coupling.

What I don't understand is, a stub to me is almost the same level of coupling as a mock. Your test setup needs to know about the dependency on what you're stubbing. So when you want to change the interface of the dependency or use a different one, or remove the dependency, your tests have to change.

It's not like you can add a query to an operation that didn't have one previously and the result will magically appear without connecting to a real database. So you have to modify the tests to provide the stub. Seems like a logical contradiction to me.

BenoitP• 6 years ago

should spin up an entire instance of Oracle

I'm sort of doing that at the moment, with postgres and series of tests. But it is still useful for unit testing too: pull up the right CREATE tables, INSERT your test data, execute your tests, then drop tables (lots of safeguards here), and repeat. The container loads up in 1.5 sec, and all my tests (~100) are done in 10 seconds.

It's been great in my use case. I'm on postgres, Java, and using testcontainers. They have handy containers available, not limited to RDBMS and not even limited to databases; and here is Oracle Express edition, which should be enough for most tests:

https://www.testcontainers.org/modules/databases/oraclexe/

stickfigure2• 6 years ago

IMO, the classic test pyramid with integration tests at the narrow top and unit tests at the wide bottom is inverted. We test software that way because integration tests are hard and unit tests are easy, but integration tests provide more value than unit tests. It's the streetlight effect: "Did you lose your car keys here?" "No, but the light is much better here."

I built some framework code so that integration tests as easy to write as unit tests. Almost all of my tests are "integration tests". I never mock the database. My test harness clones a template database at the start of each test; that database is maintained with migrations just like every other database. I run against test accounts at all the 3rd party services I use. It's not super fast - a full CI run takes about 15 minutes. And occasionally a 3rd party service will cause a test to flake. But it's fast enough and most importantly it's thorough.

I still have some unit tests that run without the expensive setup harness, but they're for components that have a lot of algorithmic complexity. Anything that touches the database gets a real instance.

This works pretty well with Postgres, which is free and I can run locally and has a fast db clone operation. YMMV with other databases.

Tainnor• 6 years ago

That's not the sole reason for avoiding too many integration tests. The main reason for that IMHO is combinatorial explosion of code paths which you'd have to test for if you wanted to be thorough.

regularfry• 6 years ago

I find it utterly bizarre that this should still be a current topic of contention. Running unit tests against a local database was a norm in Rails a decade ago, because the framework made it trivial to set up.

Yes, there were knock-on performance questions that wanted answering - how do you avoid the database setup costs for those tests that don't care about it, for instance, the "fast Rails test" movement was a big thing - but by and large those were solved problems by the time I stopped writing Rails code professionally around the 5.1 era.

The answer is, of course, no, you don't spin up an entire instance of Oracle to run a single unit test. You run against a local instance, and you use whatever tricks your-RDBMS-of-choice gives you to make resetting the test tables to a known-good state extremely fast. That way you can have your tests running continuously, giving you fast feedback as you develop, only suffering db overheads when you actually need to run tests that hit it.

If your chosen stack makes it difficult to do this, it's worth asking why.

lifeisstillgood• 6 years ago

If I am honest, at this point I would rather not use a mock, but a stub / injection.

I know this gets horrbily pedantic but its easier to see in code

def difficult_function(dbconn, a,b,c): dbconn.execute(Select * from tbl) <comlicated stuff invlvoing results set and a,b,c>

I would not want to mock the dbase at this point. Please can we instead do this

def difficult_function(dbconn, a,b,c): resultset_as_dict = dbconn.execute(select * from tbl) insider_function(a,b,c,results_set_as_dict)

def insider_function(a,b,c,results_set_as_dict): This can now be tested without mocks quite easily.

I think if you are doing 'difficult' stuff with a exernal database you are de facto, writing integration tests.

In fact i would say anything involving a database MUST be treated as a integration test. If it takes ten minutes thats fine - its an integration test.

If you want fast and external connections, use sqllite as part of your testing CI suite. But dont moan.

and do not use Mocks.

steerablesafe2• 6 years ago

Retrieving data from the database is an incoming interaction — it doesn’t result in a side effect.

Unless you count the time taken to retrieve the data as a side effect. If you are implementing a cache with a well specified behavior then you might want to test incoming interactions.

Anyway, this just shows that having a side effect can be a matter of perspective.

mumblemumble4• 6 years ago

So, disclaimer: I'm about to focus on the finger instead of where it's pointing. Because that's what I feel like jabbering about right now. TFA isn't just about databases; it makes some interesting points on test practice in general, and is well worth a read.

Anyway,

A story I've seen all too often when mocking the database: A large development effort goes into creating test infrastructure. And then there end up being scads of bugs that weren't caught by the unit tests, because the test doubles for the database don't accurately emulate important behaviors and semantics of the real database.

This isn't just a problem with mocking, mind - it's also a problem I've seen (albeit less often) when using some other DBMS during testing because it's designed to operate in-memory.

Nowadays it's not too hard to configure a RAM disk for the DBMS to use. Especially if your test stack runs it in Docker. If you're having performance problems with your test suite, start there. You might never achieve the same run times as you could with mocking, but, if there's one thing they hammered on in my Six Sigma training that I wholeheartedly agree with, it's that you shouldn't sacrifice quality or correctness for the sake of speed.

It's also not too difficult (not any more difficult than going hog-wild with mocks, anyway) to set up a mechanism that uses transactions or cleanup scripts or similar to ensure test isolation, so I don't find that complaint to be particularly compelling. You can even parallelize your tests if you can set things up so that each test is isolated to its own database or schema.

dtech2• 6 years ago

There's nothing wrong with mocking the DB in unit tests, the kind of tests you're describing are at least integration tests.

commandlinefan3• 6 years ago

scads of bugs that weren't caught by the unit tests

Which is fine! Unit tests will never catch all the bugs - neither will type safety. Neither will code reviews. Neither will manual testing. But unit tests do catch the kinds of bugs that unit tests are good at catching, which eases the burden on the manual testers.

monkpit• 6 years ago

because the test doubles for the database don't accurately emulate important behaviors and semantics of the real database.

Commenter implies that bugs occurred in code that was assumed to be tested and correct (according to the test specs) because it did have tests. Which is decidedly Not Fine.

Now, would I consider them to be “unit tests” in this case? Probably not. But the label you decide to slap on the test doesn’t change the fact that a spec was written and code was tested against it and passed (falsely) due to mocking the db.

UK-Al052• 6 years ago

A good compromise is you write an in-memory version of the dependency and the real version.

You write a comprehensive test of integration tests against the real object.

You then set it up so those same tests can run both the in-memory and real version.

Any differences should show up for the interactions you have specified, and tests should fail.

You can write sociable unit tests against the in-memory version, knowing it matches the same behaviour as closely as you have specified in the tests.

mumblemumble4• 6 years ago

That's what I typically do when I must. I find it yields much more readable tests than mocking does, and they're less likely to accidentally become tautological.

That said, discrepancies can still sink in, so I think it's best to also have some baseline level of tests that run against the real dependency, even if they're typically only run overnight on the CI server.

axegon_• 6 years ago

There are cases in which this is nearly impossible, especially when you rely on products/services delivered by other teams/companies. That said, whenever mocking can be avoided, should be avoided at all cost. I've been a witness of the "but it works on my computer" situation precisely because of that(and subsequently suffered from it immensely). Especially during the development phase(even if that means pulling 6-7-8 containers, do it): Someone misread the documentation and instead of "package_signature" used camel case in their code and used that. Of course it works on your local... Now could you give us back the 3 hours trying to solve it, please?

aidenn0• 6 years ago

This article uses the Mock/Stub distinction exactly reversed from how I've always heard it. Naming things is hard.

ooOOoo• 6 years ago

The article uses a Mock/Stub distinction very close to the famous Mocks Aren't Stubs article by Martin Fowler https://martinfowler.com/articles/mocksArentStubs.html

devit2• 6 years ago

You should never mock anything. Test against the same dependencies that exist in production.

If that's impossible (i.e. you get charged for the backends or you are controlling physical objects), then generalize the program to support alternative backends and frequently test only those that can work in the test environment, using some more ad-hoc methodologies for the others.

Also, in general, if you need to change your tests for valid changes in the implementation, then your approach to testing is completely broken. An example are dumb testing strategies where you check that the code produces specific SQL queries instead of checking that the code returns correct results.

doublesCs• 6 years ago

Test against the same dependencies that exist in production.

Interesting. I said this at my work recently and I got a condescending explanation about how production things are production, we don't touch them. If we need stuff for development, those are dev things.

I now think that whether this is or isn't a good idea depends on specifics. Most often than not, I think it makes sense.

detaro• 6 years ago

I think parent means test against the same code, not against the same instance as production. Tests don't get to talk to the production database, but should use the same database software, deployed as similarly as possible, as production does. Against test/demo versions of APIs if possible. ...

tyurok• 6 years ago

That's where "works on my machine" comes from.

The further you steer dev environments from production, the more you'll have these kinds of issues.

Would you mind expanding on their explanation? The way you described doesn't sound like an argument at all.

devit2• 6 years ago

I don't mean testing against the production instance, but rather against an instance configured like the production instance, except for being scaled/sized appropriately for the testing data and workload instead of the production one.

It's also possible to test against live production instances and data itself (mostly useful for performance optimization work and testing), although that's more of an ad-hoc process and some kinds of tests are not possible because the actual data there is arbitrary. Also, those tests, if not used for development, might be better expressed as self-checks done on system start and monitoring systems.

steerablesafe2• 6 years ago

Also, in general, if you need to change your tests for valid changes in the implementation, then your approach to testing is completely broken. An example are dumb testing strategies where you check that the code produces specific SQL queries instead of checking that the code returns correct results.

I don't see how this relates to mocking at all. You can write good or bad tests this way regardless of mocking.

BiteCode_dev• 6 years ago

This is one of the reason I don't like to put application logic directly in the database.

It makes everything harder: caching, testing, abstracting, balancing, etc.

Tomis023• 6 years ago

Where do you draw the line between a query and application logic? Does a join or aggregate function count as application logic?

On another note, a few hours/days spent implementing application code can save you 5 minutes of writing SQL.

UK-Al052• 6 years ago

Most of the reason for 'mocking' databases is because of performance reasons. If you have 8000 tests, having a fake database drastically increases performance.

One trick I use is that I write a in-memory version. I use that in unit tests.

I then write integration tests, that check behaviour of the InMemory and RealVersion are exactly the same. I inject either version into the same tests. They also check I haven't broken any code in the RealVersion which isn't been covered by unit tests mostly because its just external interaction in there.

If you have to verify inter-service interactions, use contract tests.

harrisonjackson4• 6 years ago

For a long time, we ran our Django test suite against an in-memory SQLite DB. It was super fast, which encouraged more tests to be written and a CI/CD process that allowed everyone to confidently ship code often.

Our production database is postgres.

We kept bumping against things we wanted to do in the application code that worked well with postgres, but would fail in sqlite. We limited our development so that we could keep running the tests. We knew that we could run them against a postgres db, but the development time to rewrite our CI test runner to spin up a fresh database was not worth it.

Recently we migrated from github + Jenkins to gitlab + our own gitlab-runners in AWS. During that switch, we prioritized testing against a Postgres DB and got it all running in a containerized way that spins up at fresh DB for every test run. The tests are slower but the runners scale horizontally so we don't mind queueing up as many merge requests as we need to and deployments through Gitlab environments are a big improvement over our Jenkins deployment job, so we still ship as often as we want.

Now our biggest testing issue is keeping fixtures up to date.

malinens• 6 years ago

if you keep using sqlite for tests it forces Your database logic to be universal. You could confidently switch to other db like mysql any time.

taffer2• 6 years ago

This also means that you have to stick to the lowest common denominator between all databases.

harrisonjackson4• 6 years ago

This was specifically why we moved away from sqlite. We wanted to take advantage of features postgres offers that are not universal.

mumblemumble4• 6 years ago

It also means that you have to stick to the lowest common denominator, which is approximately equivalent to the state of the art as of a quarter century ago.

Every benefit has a cost. The benefit doesn't always justify the cost.

edit: To add to that - I've seen more than a couple major database engine migrations in my day, so it's not to say that that isn't a concern. But none of them has ever been from one SQL RDBMS to another. More common is migrating among different classes of database. MySQL to Mongo, Oracle to BigTable, Couch to Cassandra, something like that. MS Access to MS SQL Server a couple times, but even those are different enough that it was never going to be as simple as changing the connection string and having a carefree life.

The speculative future proofing that you do almost never manages to work for the future you end up actually living.

taffer2• 6 years ago

Can't you use transactions to set up and tear down fixtures? That should be a lot faster than spinning up a new database every time.

harrisonjackson4• 6 years ago

Yes, that is how the test suite handles it. The CI uses a fresh db for each run of the test job but within that job, it uses 1 instance of the database.

Locally we have a flag to keep the test db alive between runs which speeds the tests up and can help with debugging.

The slowest part of the test run in the CI is building the application container and pulling down the postgres container. I'm sure there are improvements to how we are handling this but it isn't enough of an issue to prioritize it now.

Our issue with fixtures has more to do with changing application code and not having a great way to generate/regenerate the fixtures from live data. We've tried a few different libraries to do this but haven't found any that we love.

stickfigure2• 6 years ago

It isn't clear if you're already doing this, but Postgres has a 'create database from template' feature. You can initialize your template database with migrations once for a whole test suite, then clone a new database at the start of each test.

It's quite fast. I run almost all of my tests this way.

I agree, forcing your app into the lowest common denominator of portable SQL is crippling. JSONB columns in particular are extremely useful in Postgres.

harrisonjackson4• 6 years ago

Nice! I did not know Postgres had that. We'll probably do something similar next time we prioritize improvements to the test runner.

Right now we are pulling a postgres:10 container down from our ECR on every run :laughcry: so definitely some low hanging fruit around that.

I think we will rebuild the postgres container up to our most recent migrations in prod branch then bake that container onto the gitlab runner AMI daily. Then the test runner can just start that and apply any migrations in the merge request and proceed with the tests.

anarazel• 6 years ago

It obviously makes sense that creating a new PG instance will be slower than creating a sqlite database. There also is some inherent speed difference for simple queries just by virtue of sqlite not needing to context switch to a separate process and to marshal the query/results across process boundaries.

But if the difference is more substantial than those factors would suggest, I'd be interested to see if we can do something about it from the PG side.

bob1029• 6 years ago

Our trick is to use SQLite to completely side-step the concern of maintaining any sort of central testing database. Each developer can clone a fresh copy, load the .sln, hit F5, and every database required by the application is immediately available within the same process. There is absolutely no other software required to be installed.

This also makes it infinitely easier to coordinate complex schema changes. Each developer can sort it all out on their local branch before we even know about it. If we had to share some common test database, this would become a much more painful process.

Also, I do not believe in using mock layers for the database interactions. Our service implementations are tightly-coupled with their backing datastore. This is the only way we are able to make SQLite a viable storage medium for a high-throughput business application. As a consequence, testing our services in absence of their concrete datastores would be an extremely disingenuous endeavor for us.

squiggleblaz2• 6 years ago

Are you using SQLite as your main production datastore or you're using sqlite as a standin for something else? It sounds like production. One db file or several? How big? Can you summarise the non-test benefits and costs?

slifin• 6 years ago

If your database is mutable then you have a third party network resource, essentially you can't test it because you have no stable basis in time and no means of dealing in values

All you have is a shared place, mocking is essentially putting up a fence around it to prevent testing into that boundary

If your database is immutable and is indexed for time travel, you can rewind or fast-forward your database into your desired state, if your database supports speculative writes you can even build up a non-committable state

I'm sure Rich Hickey will have a talk on this in relation to databases

When to Mock

Related Stories