How to implement a disk cache plugin for Elixir's Req HTTP client?

https://thibautbarrere.com/2022/09/09/implementing-a-disk-cache-for-elixir-req

60 36 3 years ago

38• 3 years ago

    Req.get!("https://hex.pm/api/packages/req").body["meta"]["description"]

wow. coming from Go I have to say I do not like this. no error checking at all (I assume it just panics or exception?) no mention of JSON at all, even though thats what its doing. and if "body" is JSON, how do you even get the raw body, or can you? just seems over engineered/over fitted whatever you want to call it.

strken2• 3 years ago

From memory https://wiki.c2.com/?LetItCrash is a big part of Erlang's (and therefore Elixir's) philosophy. You put everything in a process, monitor it with a supervisor, and just crash and let the supervisor recover unless the error is immediately recoverable in the calling code, e.g. trying a second provider or skipping the cache or something.

metadat2• 3 years ago

How do you report on / bubble up what the precise nature of the problem was under this paradigm?

E.g. Was it a network issue, content type issue, malformed JSON, unexpected JSON format, etc?

I mostly enjoy Go, Python, and Java (in this order), but I must say the self-contained, self-dependency-installing script thing in TFA seems pretty appealing.

metadat2• 3 years ago

It's actually not just Go, most procedural languages have similar HTTP request flows (e.g. Java, Python, C++, TCL).

Cool GitHub, btw, though I couldn't help but notice you have only Go repositories, which I found amusing (never seen a profile quite like yours before!). Cheers.

jolux3• 3 years ago

no error checking at all

First of all: this example is a one-off script, so error handling is not as much of a concern (you'll notice the first expression installs packages, this is not how production apps should handle dependencies).

Second, functions that raise always end in `!` in Elixir, or at least they are supposed to. Most have alternatives that return error tuples instead which you can pattern match on (this is what I recommend). You'll hear "let it crash" a lot as it applies to Erlang/OTP programming (and by extension Elixir), and this is a good high-level organizing principle. You shouldn't go out of your way to program against the unknown in Elixir. However, HTTP requests failing is common enough that in my opinion at least you really should just check the error. You can read the docs for `get/2` (as opposed to `get!/2` which raises) here: https://hexdocs.pm/req/Req.html#get/2.

A common pattern is for the `!` version to call the version that doesn't raise, check the result, and raise on error, which is the case here: https://github.com/wojtekmach/req/blob/9de30de0df481ee557ccc...

and if "body" is JSON, how do you even get the raw body, or can you?

You would set `decode: false` when calling `get!/2: https://hexdocs.pm/req/Req.html#new/1. You can also set this as configuration with https://hexdocs.pm/req/Req.html#default_options/1.

As a closing note I'll mention that Req is intended to be a very high-level, scripting-friendly requests library, similar to Requests in Python (the author also mentions in the OP that his use case is scripting). If you don't want conveniences like Req provides, you can either turn them off or use something different, like Finch (which Req is based on, https://github.com/sneako/finch).

Other than Req and Finch I'm personally only familiar with HTTPoison, which is significantly older than all of the libraries derived from Mint (like Finch and Req, https://github.com/elixir-mint/mint) but is very popular and still works. There are many others though, like Gun and Tesla and such. I just think it's important for folks to know that this level of "magic" is far from the default in Elixir, but it's there if you want it.

ricketycricket• 3 years ago

no error checking at all (I assume it just panics or exception?)

In Elixir, bang functions per convention will raise on error. `get/2` will return error tuples allowing you to handle errors. In fact, get!/2 just calls get/2 and raises for you[^1].

no mention of JSON at all

Req is the most "batteries included" Elixir HTTP lib out there. I can't speak for Wojtek, but I believe the goal was to make Req extremely easy to use in scripting or things like LiveBook without having to do much work. That being said, the automatic decoding is mentioned in the readme[^2] and the docs[^3].

if "body" is JSON, how do you even get the raw body, or can you?

Per the docs[^3], you can either skip with the `:decode_body` option, or just build your own request using only the steps you want.

just seems over engineered/over fitted whatever you want to call it.

Fair, but again, this library is designed to be on that end of the spectrum. There are plenty of other libraries further down the stack that you can use. I am partial to Finch[^4], upon which Req is built.

To address the sibling comment about "Let it Crash", the language allows you to easily recover from crashes, but that is for resiliency, not error handling. In practice you would use the non-bang get/2, pattern match on the response, handle any errors, perhaps use Kernel.get_in/2 to safely traverse the map, etc. The example provided by the author is not "production ready".

[^1]: https://github.com/wojtekmach/req/blob/v0.3.11/lib/req.ex#L3...

[^2]: https://github.com/wojtekmach/req

[^3]: https://hexdocs.pm/req/Req.Steps.html#decode_body/1

[^4]: https://github.com/sneako/finch

thibaut_barrere2• 3 years ago

Thanks for detailing while I was asleep :-)

Indeed, that + pattern matching does error checking just right.

I must admit that the automatic JSON decoding initially turned me off a bit!

antran22• 3 years ago

It's a piece of example code to demo a concept. You are asking people to error check a something akin to a curl call in a bash command.

nesarkvechnep2• 3 years ago

Mind that this is not an idiomatic code. One would usually use pattern matching.

   case Req.get!("https://hex.pm/api/packages/req") do
     %{body: %{"meta" => %{"description" => desc}}} ->
       # use the value of desc here
     %{body: body} ->
       # report for unexpected body
   end

As others mentioned, the ! at the end of the function name tells that the user should expect an exception.

jolux3• 3 years ago

That depends. I think that's a bit deep for pattern matching personally but I've evolved on this over time.

thibaut_barrere2• 3 years ago

I would not say this is not idiomatic - it depends on the situation.

On production code or scripts where I want to be sure (which is the case mostly), I would indeed use something like:

%{status: 200, body: json} = Req.get!(".....")

And then later restructure as needed.

Sometimes it would be:

meta = json |> Map.fetch!("meta") |> Map.fetch!("description")

Or pattern matching as you did!

How to implement a disk cache plugin for Elixir's Req HTTP client?

Related Stories