Fundamentals of Object Storage in Elixir

https://underjord.io/fundamentals-of-object-storage.html

by lawik • 2 years ago

122 39 2 years ago

Good write-up!

I will add:

- make sure to “stream” your transfers in general, instead of keeping whole file blobs in RAM. This to avoid refactoring at the least comfortable moment when a file gets too big in production

- it is quite easy to use Mox for isolated testing in general, and it works nicely with ExAws, so make sure to integrate those tests early on

sph4• 2 years ago

Seconding the Mox recommendation, it's pretty good and enforces good separation of concerns in your application. It was created by José himself, so you know it's got a well-designed API. You just need to get used with using behaviours, and using the Application config to tell your code when to use the mock vs the real thing.

I use it to separate and test external API calls (i.e. to test the payment layer without making any request), to maintain separation between my code and an external library I might want to swap out at a later date, or even between various components of the application. Then writing unit tests becomes a breeze, and you don't need many integration tests.

https://hexdocs.pm/mox/Mox.html

sarchertech8• 2 years ago

I hate Mox with a passion. Because I hate separating out the interface and implementation of something that is only ever going to have one real implementation.

The extra layer of indirection just for testing bugs me so much. And I just can’t justify it with “one day I might want to swap out the implementation”. YAGNI and if you do, deal with it then.

So much legacy elixir code is an unreadable mess because of all the extra layers of interfaces that only get used for Mox. There’s nothing I dislike more than diving into an unfamiliar section of code and having to keep 15 files open to follow the logic for something that should have been in one module.

SSLy2• 2 years ago

what's up with the swedes and their insistence of using : in lieu of apostrophe?

emj• 2 years ago

It is often used in swedish technical writing to separate and foreign word or an acronym from the plural so "VPS:es" in english is VPS:erna in swedish. I do not know if this is a swedish tradition, it most probably exists in Germany or the US. I've seen this use in Sweden for several decades.

SSLy2• 2 years ago

I've seen it used by Swedes and Swedish speaking Finns, at least.

bobwaycott• 2 years ago

Even in English, the use case you’re referring to would not have an apostrophe—apostrophes are used for contractions and possession, not plurality.

penguin_booze• 2 years ago

Meet our hero: https://www.bbc.co.uk/news/av/uk-39459831.

atombender• 2 years ago

It's a fairly old tradition specific to Swedish and Finnish. Wikipedia has an explanation. [1]

[1] https://en.wikipedia.org/wiki/Colon_(punctuation)

lawik2• 2 years ago

If I knew what was up with us swedes I'd be consulting on that instead.

losvedir2• 2 years ago

Nice post. One part that raised a red flag for me was:

     defp bucket!, do: System.fetch_env!("BUCKET_NAME")

This will look at that environment variable at build time. For many workflows this doesn't work because you may build the app in CI and run it with different buckets in production and staging.

I always default to using Application.get_env/2 and setting the value in runtime.exs, now that that's a thing in recent Elixir.

For things where performance matters in a hot loop, it might be worth compiling in the value, but I wouldn't imagine that's the case here where it's a part of a network call to AWS.

ilikehurdles• 2 years ago

This will look at it at runtime not build time. `System.fetch_env!("BUCKET_NAME")` is still in the body of the function and won't get executed until it's called after application start.

It is still a good reason to use Application.get_env, and it should be preferred over peppering system environment variable fetches all over the application.

losvedir2• 2 years ago

Oh, of course. Brain fart moment. I pattern matched (ha!) to code I've seen too much where that's at compile time.

Yeah, runtime here works but I agree is still not as good as Application.get_env. Aside from (IMO) code clarity reasons, last I checked it is (or can be) much quicker. Application is a quick ETS lookup, while System reaches out to the system and I forget the details but at one point that was much slower.

thibaut_barrere4• 2 years ago

As pointed out, it's not compile time but runtime here unless I'm missing anything. And it would be a good reason to build a Credo check exactly for this (I've done a bit of research about this).

Any calls to System in regular compile time config (e.g. config/prod/dev/test or included) should be forbidden, same for attribute-level / meta code.

I hope we will get this one day!

lawik2• 2 years ago

There are a bunch of things that are kind of quick and dirty about these snippets. I think I mentioned that in the post, that these are not best practices. But not quite so ill as compile-time in this case :)

29athrowaway• 2 years ago

All Elixir projects I've seen, when you run "mix xref" you will see spaghetti.

josevalim2• 2 years ago

You are not wrong but you may be hinting at the wrong cause. It is hard to generate a useful graph or summary that gives actionable feedback once you reach a certain amount of nodes. Even mix app.tree, which has no cycles, is barely useful after 30-50 deps because it becomes dense.

Using mix xref to evaluate your project will, at best, give you a very blurry picture. Think of it more of a database: it will give you good information when you need to answer certain queries.

leftsaidtim• 2 years ago

Oh boy another underjord post. It's always such a joy getting to read his writing.

Fundamentals of Object Storage in Elixir

Related Stories