radanskoric

Author here. I didn't mention it because I wasn't writing an evaluation of fixtures. Just writing about how to make better use of fixtures. I actually use both fixtures and factories depending on the project specifics and also whether it is even my decision to make. :)

Personally, I even slightly prefer to use Factories and I also previously wrote about a better way to use them: https://radanskoric.com/articles/test-factories-principal-of...

It's required for tests to clean up after themselves. With Rails and fixtures this is handled by default: each test runs inside a transaction which is rolled back at the end of the test. That way each test starts with the same initial state.

I'm with you on the one assertion per test. I bundled two assertions into the same test here because my whole point was to have them effectively together describe a single test, just in a more maintainable manner.

Regarding the fact that I'm not fully testing the contract of the method, you're absolutely correct. But also, no example based test suite is fully doing that. As long as the test suite is example based it is always possible to find a counter-case where the contract is violated but the test suite misses it.

These counter-cases will be more contrived and less likely the better the test suite. So all of us at some point decide that we've done enough and that more contrived cases are so unlikely and the cost of mistake is so small that it's not worth it to put in the extra testing effort. Some people don't explicitly think about it but that decision is still made one way or another.

This is a long way of saying that I both agree with you but that also, in most cases, I would still take the tradeoff and go for more maintainable tests.

Ah, ok, now I understand. Ok, I wasn't talking about that. From what I understand about property based testing it's sort of half way between regular example based testing and formal proofs: It tries to prove a statement but instead of a symbolic proof it does it stohastically via a set of examples?

Unfortunately, I'm not aware of a good property based testing library in Ruby, although it would be useful to have one.

Even so I'm guessing that property based testing in practice would be too resource intensive to test the entire application with it? You'd probably only test critical domain logic components and use regular example tests for the rest.

Ah, Ok, yes, for API endpoints it makes a lot of sense. Especially if it's a public API, you need to inspect the output anyway, to ensure that the public contract is not broken.

But, I spend very little or no time on API endpoints since I don't work on projects where the frontend is an SPA. :)

Author here. Yes, what you describe sound where much like what I call Factories (and that's what they're usually called in Ruby land, and some other languages).

The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite considerably.

If you use them to generate purely in memory objects, this problem goes away and then I also prefer to use factories (or generators, as you describe them).

I think I'm getting what you mean and I almost completely agree with you, let me address one part, the only part where I don't agree:

Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project.

Imagine this:

1. You start with some fixtures. You crafted the fixtures and you're happy that the fixtures are good for the test you're about to write.

2. You write a test where you assert the EXACT collection that is returned. This is, as you say, a test that "confirms the scope returns _every_ active project".

3. You now rewrite the test so that it checks that the collection includes ALL active projects and excludes all inactive projects.

Do you agree that nothing changed when you went from 2 to 3? As long as you don't change the fixtures, those 2 version of the test will behave exactly the same: if one passes so will the other and if one fails so will the other. As long as fixtures don't change they have exactly the same testing power.

If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?

I'm familiar with snapshot testing for UI and I agree with you, they can work really well for this because they're usually quick to verify. And especially if you can build in some smart tolerance to the comparison logic, it can be really easy to maintain.

But how would you do snapshot testing for behaviour? I'm approaching the problem primarily from the backend side and there most tests are about behaviour.

When you say inheritance do you mean DRY as in "Don't repeat yourself"?

I'm not sure what you mean by inheritance in tests but DRY is criminally overused in tests. That could be a whole separate article but the tradeoffs are very different between test and app code and repetition in the test code is much less problematic and sometimes even desirable.

In theory you're 100% right, a true unit test is completely isolated from the rest of the system and than a lot of the problems disappear.

In reality, that is also not free. It imposes some restrictions on the code. Sometimes being pragmatic, backing off from the ideal leads to faster development and quicker deliver of value to the users. Rails is big on these pragmatic tradeoffs. The important thing is that we know when and why we're making the tradeoff.

Usually I go with Rails defaults and usually it's not a problem. Sometimes, when the code is especially complex and perhaps on the critical path, I turn up the purity dial and go down the road you describe exactly for the benefits you describe.

But when I decide that sticking to the defaults the is right tradeoff I want to get the most of it and use Fixtures (or Factories) in the optimal way.

Your thinking is sound. At the end of the day Rails default fixtures is nothing more than some code that reads yaml files and creates records once at the start of test suite run.

So you can definitely use FactoryBot to create them. However, the reason I think that's rarely done is that you're pretty likely to start recreating a lot of the features of Rails fixtures yourself. And perhaps all you need to do is to dynamically generate the yaml files. Rails yaml fixtures are actually ERB files and you can treat is an ERB template and generate its code dynamically: https://guides.rubyonrails.org/testing.html#embedding-code-i...

If that is flexible enough for you, it's a better path since you'll get all the usual fixture helpers and association resolving logic for free.

Author here. I'm a big fan of factories but the slowness is a real drag on large test suites. If you're considering switching, remember that you can do it gradually, there's no law against using both fixtures and factories in the same project, in some cases (mostly on very complex domain data models) even makes sense: fixtures for the base setup that all tests share, factories for additional test specific records.

Btw, I also have an article with some of my learnings using factories and I make a remark on how it helps with test speed: https://radanskoric.com/articles/test-factories-principal-of...

Author here. Thanks for writing up your thoughts on this!

The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that:

``` refute_includes active_projects, projects(:inactive) ```

Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ...

Regarding the fact that the test does not check that the scope returns "all" active projects, that's a bit more complex to address but let me let tell you how I'm thinking about it:

The point of tests is to validate expected behaviours and prevent regressions (i.e. breaking old behaviour when introducing new features). It is impossible for tests to do this 100%. E.g. even if you test that the scope returns all active projects present in the fixtures that doesn't guarantee that the scope always returns all active projects for any possible list of active projects. If you want 100% validation your only choice is to turn to formal proof methods but that's whole different topic.

You could always add more active project examples. When you write a test that is checking that "Active projects A,B and C" are returned that is the same test as if your fixtures contained ONLY active projects A,B and C and then you tested that all of them are returned. In either case it is up to you to make sure that the projects are representative.

So by rewriting the test to check: 1. These example projects are included. 2. These other example projects are excluded.

You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain.

Does that make sense? Let me know which part is still confusing and I'll try to rephrase the explanation.

Author here, thanks for posting. :)

Woah, this thread escalated quickly.

Can you explain why you say they would be better off? What else would be a better choice and why?

Have you used it over the last few years? It has it been rapidly improving, mainly because Shopify put a team full time on it. It doesn’t take a lot of people to optimize a VM/interpreter it just has to be the right people.

And the question is always “fast enough for what?” Different languages are more suitable for different types of projects. I wouldn’t code a rendering engine in Ruby but for web apps it’s amazing.

It's a sharp knife. You can create a messy nightmare or a clean super readable codebase, it's up to how good the author is.

No worries, I didn't think that's the case. :)

It's an interesting conversation.

That was my misunderstanding as well. That's why I wrote the article.

Btw, it's not even about the RubyLLM gem. The gem abstracts away the calling of various LLM providers and gives a very clean and easy to use interface. But it's not what gives the "agentic magic". The magic is pretty much all in the underlying LLMs.

Seeing all the claims made by some closed source agent products (remember the "world's first AI software engineer"?) I thought that a fair amount of AI innovation is in the agent tool itself. So I was surprised when I realised that almost all of the "magic" parts are coming from the underlying LLM.

It's also kind of nice because it means that if you wanted to work on an agent product you can do that even if you're not an AI specialised engineer (like I am not).

Interesting. I've not used it but this sounds like the tools that were exposed to it need improvement. Everything you describe seems like it should be mostly in reach of current larger models.

Yes, exactly, Ruby has the human readability expressiveness.

If you say that expressivity is the ability to implement a program in less lines of code then Ruby is more expressive than most but less than for example Clojure. Well written Clojure can be incredibly expressive. However, you can argue that for most people it's going to be less readable than a comparable Ruby program.

It's hard to talk about these qualities as there's a fair amount of subjectivity involved.

It's good that you commented. I see more than a few people are getting caught up on the number of lines so it's good that I clarify.

Why are people always the reason why we can't have nice things... :D

That's a very neat idea, maybe even add something like browser-use to allow it to implement a Rails app and try it out automatically. I think you should try it. :)

I'm being serious. This sounds like a fun project but I have to turn my attention to other projects for the near future. This was more of an experiment for me, but it would be cool to see someone try out that idea.

I didn't put the number into the title to make it a competition. LoC is a poor metric. I put it to communicate to the reader that they won't have to spend a lot of time reading the article to get a full understanding.

I always put extra effort into trying to make my blog posts shorter without sacrificing the quality. I think good technical writing should transfer the knowledge while requesting the least amount of time possible from the reader.

It depends on the flexibility of the API. If you're making an API for just one specific use case, you can make it a one liner in any language, even assembler: just push the exact specific functionality into the one function.

Language expressiveness is more about making the interface support more use case while still being as concise. And Ruby is really good at this, better than most languages.

I put the lines of code into the title to communicate to the reader that they can get a good understanding just by reading this article.

Basically, what I wanted to say was: "Here is an article on building a prototype coding agent in Ruby that explains how it works and the code is just 94 lines so you'll really be able to get a good understanding just by reading this article."

But that's a bit too long for a title. :)

When understanding a certain concept, it's very useful to be able to see just the code that's relevant to the concept. Ruby language design enables that really well. Also, Ruby community in general puts a lot of value on readability. Which is why with Ruby it's often possible to eliminate almost all of the boilerplate while still keeping the code relatively flexible.

That's an excellent point.

Code was always read more than written. With AI it shifts even more towards reading so language readability becomes even more important. And Ruby really shines there.

If a certain user is susceptible to having the LLM convince them to run an unsafe command, I fear we can't fix that by trying to trick the LLM. :D

Either the user needs to be educated or we need to restrict what the user themselves can do.