Defining a new HTTP method: HTTP Search (2021)

https://httptoolkit.com/blog/http-search-method/

by thunderbong • 3 years ago

171 129 3 years ago

httptoolkit.com

Spivak• 3 years ago

See also QUERY, imho a better name than SEARCH which is semantically a GET but with a request body -- it's cacheable!

https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-meth...

mlhpdx• 3 years ago

Yep, I’ve already implemented QUERY. Sorry SEARCH.

buildfocus• 3 years ago

They're effectively the same I think - this post is from 2021 but a later version of the same draft renamed the concept to QUERY.

montroser• 3 years ago

This seems reasonable, but unlikely to happen. GET is already pretty much sufficient, especially given that clients generally support megabytes worth of query strings in these modern times. If we're doing this though, I'll cast my vote for naming it more semantically as QUERY, and forgoing whatever little WebDAV compatibility would be had otherwise.

ZiiS• 3 years ago

QUERY would be more compatible with WebDAV as the would be no confusion. The relevant compatibility is with middle boxes like proxy servers that mostly already forward SEARCH but would block QUERY.

Raqbit• 3 years ago

Middle boxes can arbitrarily limit the request URL size in certain cases.

rcme• 3 years ago

Sure, but in the times of encrypted traffic, the only middle boxes capable of limiting the query should be ones explicitly added by the service owner.

dragonwriter7• 3 years ago

Funny you should say that:

https://httpwg.org/http-extensions/draft-ietf-httpbis-safe-m...

austin-cheney• 3 years ago

I understand the benefits of QUERY over GET because GET is expected to return a resource in the response. Why not just use POST though? Send a post message to a server asking to conduct a search and sending the results of that query in the response?

wilg• 3 years ago

I’ve, amazingly, managed to hit the arbitrary megabyte limit in real production software.

linuxdude314• 3 years ago

Same, that's configurable though, and 1mb sounds like nginx. It's a lot higher on many other web servers including Apache.

afiori3• 3 years ago

I can report that cloudlfare workers fail on 44kb of url

paulddraper3• 3 years ago

Agreed.

Could be used for e.g. SQL over HTTP

Nurbek-F• 3 years ago

Agree. I was also thinking about the "Query" naming

AtNightWeCode4• 3 years ago

Or just use GET since it already works. I don't see any problems with large query params. You can also split the search into a POST request and a GET request for the result.

insanitybit• 3 years ago

I don't see any problems with large query params.

Most servers will reject URLs after a certain size, 4-8KB.

le-mark• 3 years ago

This is correct, and one only needs to consider the impact of large or unlimited length urls on server memory consumption per request to see that a relatively small limit is desirable and even necessary.

AtNightWeCode4• 3 years ago

A query of that size will likely not benefit from a GET anyway so use a POST.

time0ut2• 3 years ago

A situational issue that I've run into a number of times with GET is various middle boxes inadvertently logging sensitive data that was in the query string of a GET. By sensitive data, I mean PII and CU information. While we have used POST to side step the issue, it'd be nice if there was a more appropriate method.

AtNightWeCode4• 3 years ago

Many logging systems exclude query parameters from the logging by default. It is also fine to include PII data in the logs as long as they are handled correctly and deleted after 30 days. But I have got burned by this problem as well. Like people using PIIs for identifiers.

Beldin• 3 years ago

But a new HTTP method would require an update to the middle boxes... if you're doing that, you could also consider updating them to avoid the issue you described.

Overall, I'm not necessarily inherently against new HTTP methods. If equipment that is supposed to be transparent to HTTP isn't, should be solved elsewhere, not by a new HTTP method.

slondr4• 3 years ago

I’m pretty sure most browsers have fairly low limits for uri length.

tkiolp42• 3 years ago

Wouldn’t be enough to “extend” GET so that it supports a body payload as well?

Makhini• 3 years ago

It already does.

dwb• 3 years ago

Sort of, but it’s not that solid: https://datatracker.ietf.org/doc/html/rfc9110#name-get

(paragraph starting “Although request message framing is independent of the method used”)

electroly• 3 years ago

The RFCs are clear that a server should ignore any body sent by a client after a GET request header. But RFCs don't matter; real life matters, and in real life there are lots of HTTP clients that can't make a GET request with a body. We had to go into our API and add POST support to every endpoint that was GET + body because there was a long tail of clients that couldn't manage to make the GET request. In practice, support is NOT widespread enough to rely on GET + body requests.

simpsond• 3 years ago

It’s not exhaustive though. Many servers and proxies support it, but not all. For instance, GCP load balancers do not, but ALB does. At least that was case a few years ago.

time0ut2• 3 years ago

It sort of does, but support is spotty. I've tried using it only to have things like MITM proxies on my customers' corporate networks drop the body. It'd be a large push to get universal support. It may be more difficult than simply adding a new method.

tkiolp42• 3 years ago

But if a new method is needed, servers have to support it (e.g., by updating them to the next version that supports the new method). In the same sense, couldn’t that new version “just” not drop the body of GET requests?

In any case we are talking about updating servers.

simonw• 3 years ago

The only software I have seen that does this is Elasticsearch - it works with their client libraries but I'm always worried there may be proxies and suchlike which don't support it.

They support POST as a fallback.

dragonwriter7• 3 years ago

No, because that would make GET not-cachable by URL alone, which breaks systems that rely on the fact that it is.

A new method doesn’t break existing systems.

jillesvangurp• 3 years ago

You can actually do that already. It's one of those things where the specification is a bit loose on this. Elasticsearch is an example of a product that supports doing a GET with a body to do search. Works fine but not all http clients allow you to do this so they also support POST.

Nice discussion on HTTP GET with a body here: https://stackoverflow.com/questions/978061/http-get-with-req...

TLDR. the http 1.1 spec was a bit vague on this and allowed people to send a body but specified that servers should ignore that if they did. Later updates of the 1.1 spec removed that sentence and actually allow this. The caveat that the spec mentions that some implementations may not support it.

So an update is not needed; it's already allowed.

I tend to not get too hung up on the meaning of http verbs. I sort of lost interest in the endless debates on this years ago. I like a well designed REST API of course and I try to follow the principle of the least amount of surprise when I implement one myself. Using exotic new experimental http verbs would be surprising. Using GET this way is also a bit surprising. I actually steer clear of using PATCH as well. Just use a PUT or a POST and move on.

Adding new verbs to HTTP is redundant as far as I'm concerned. Not really against it but where does it stop? And what do we get out of it? It doesn't sound like it's worth the trouble.

hgl• 3 years ago

Many people seem to be unenthusiastic about it because the limitation on query strings are usually large enough.

I personally like this addition, because it no longer requires all queries be shoehorned into query strings. You can use any syntax you like, be it SQL or GraphQL etc.

cratermoon3• 3 years ago

I'm trying to imagine an example of the authors primary use case: "a complicated data retrieval, sending lots of data but not changing the server state". Is he imagine sending something like a SQL statement SELECT with multiple WHERE and JOIN clauses using a recursive CTE?

If defining the attributes of the thing to return takes that much complexity, maybe that's a sign the endpoint is poorly designed, rather than GET query string length being insufficiently large.

bastawhiz2• 3 years ago

A great example of this today is GraphQL. Even if you disagree with the premise of GQL, you can understand how you can have complicated filtering and sorting logic, and conditional expansion of nested resources. For a nontrivial data model, you may want to select only a minimal number of fields on each nested resource. Having lots of different variants of the same endpoint to serve the same query with different shapes of output is difficult to maintain, so this makes some sense.

cratermoon3• 3 years ago

I knew someone would bring up GraphQL, as I consider it a sort of way to present a database schema in JSON, so of course it would result in complex SQL-like queries. I understand why it works for Facebook, but it's not something I've ever reached for to solve any programming problem I've had.

slondr4• 3 years ago

We actually run into the query limit problem super often where I work. User-facing search pages for very complicated data models create very complicated URLs quickly.

cratermoon3• 3 years ago

That reminds me of reporting software like Crystal Reports. A thin veneer over a relation model (or, often, a collection models abstracted). I understand why businesses choose them, but they've always struck me as a bit of a shortcut. I'm always more curious about what the business does with the reports, and why we can't program to those requirements.

ThaFresh• 3 years ago

being from 2021, I guess this one didnt pan out

dragonwriter7• 3 years ago

Its still actively being worked (the most recent workgroup draft, and IIRC several of the internet drafts since the posted one, names it QUERY, SEARCH as a name for thr verb is probably dead) though, yes, the particular 2021 draft didn’t get adopted as anything more.

IshKebab• 3 years ago

The advantage of GET is that you can always link to it. Doesn't matter with POST because you're never going to want to link to a POST.

So, GET URLs with base64 encoded data is definitely ugly, but at least you can save them and share them easily.

asattarmd• 3 years ago

One advantage of GET is I can just copy the URL and share it. The article makes no mention of that.

While I love the proposal (apart from the name, I can see the SEARCH verb being used for something that's not search), they should also address the URL share-ablity aspect.

Something like https://google.com/search<some-special-character><query> where query can be arbitrarily large (>2000 URL length restriction) and the browser is smart enough to treat only https://google.com/search as the URL and anything after that as the body. The complete "URL" can be big and shared anywhere else.

hartator• 3 years ago

Yes, that’s also lost when you do POST. Which is by design though. A HTTP Search seems like only drawbacks.

paulddraper3• 3 years ago

SEARCH can have a request body.

Many systems limit the length of the URL, so this is significant.

2h• 3 years ago

One advantage of GET is I can just copy the URL and share it.

no, you cant. if the server requires any headers such as Authorization or Cookie, this method will fail.

johngalt_• 3 years ago

They are obviously talking about public urls.

ivan_gammel• 3 years ago

Typical scenario is to redirect to login and after successful authentication return back to the requested URL, so the method doesn’t fail if server is implemented correctly.

hombre_fatal• 3 years ago

It’s not a “no you can’t” just because you know of some exceptions.

And even then, they are still correct while you are not. You can copy the GET url even if it ultimately it requires authentication in a way that you can’t do it all for a POST request.

candiddevmike• 3 years ago

Related, I wish there was a more standard way to include things like verb and headers in a URI. I hacked an implementation that parses /something#a:b&c:d to set the headers a and c, I was thinking for verb I could do https+get.

dragonwriter7• 3 years ago

verbs don't identify a resource, so why would they be in a URI?

A separate format to serialize a request spec is a good idea, sure, but it is a distinctly different thing than the URI of the resource referenced by the request.

sceutre• 3 years ago

Sounds like what curl does. Maybe curl:{curl commandline}

GloomyBoots• 3 years ago

This is an interesting idea for a browser extension. Maybe needs a change in name from URL/URI. Could be a DURI, Discrete Universal Record Interaction. Just spitballing. You could share one-liners similar to how one might share curl command-lines but expect them to work in multiple environments.

graypegg• 3 years ago

Guess that needs to be different than the ? query params. I wonder if you could use ?key=value as your signal for GET, and ?value as your signal for SEARCH. I think it's up to the service to parse the string that comes after ? in a URI so maybe that's a way to go.

justinator• 3 years ago

That would unfortunately break the internet as a GET query without ?key=value and just ?value is perfectly reasonable.

phailhaus• 3 years ago

The primary use case of SEARCH is programmatic, e.g., making complex requests to a search API in order to render results. Those are API requests, they're not being shared around.

BugsJustFindMe2• 3 years ago

One advantage of GET is I can just copy the URL and share it.

Often but not always.

The article is wrong when it says message bodies for GET are defined to be meaningless. They are in fact just not defined to be meaningful, which is very much not the same thing.

Nothing in the spec for GET blocks using message bodies with it. Elastic search famously uses(used?) bodies with GET requests.

preseinger• 3 years ago

unless you control every hop between client and server, GET bodies can be arbitrarily dropped, and can't be relied upon

deepzn• 3 years ago

The proposal has changed the name to QUERY, just yesterday.

imnotjames2• 3 years ago

This is why the RFC suggests you redirect to a GET like /resource/queryABCD

based2• 3 years ago

The missing ones are HTTP OPEN/CLOSE to improve the sessions management and security audits.

brabel2• 3 years ago

Did you know that HTTP is an inherently stateless protocol?

From the very first line in the RFC[1]:

"The Hypertext Transfer Protocol (HTTP) is a stateless application- level protocol for distributed, collaborative, hypertext information systems."

Which is why you need to send a header every time you make a request with authorization data.

Given that: no, stateful methods like OPEN/CLOSE would not fit at all with how HTTP is designed to work.

[1] https://www.rfc-editor.org/rfc/rfc7231

phendrenad2• 3 years ago

Can anyone steelman this idea for me? Seems like an incredible waste of time to make this.

hackerbrother• 3 years ago

I want this. SEARCH/POST APIs forever.

mabbo• 3 years ago

What I like about this proposal is that no one has to use it. There's lots of comments here saying why they prefer GET, and those people are welcome to keep using it- GET will remain unchanged.

But for those applications where having a complex body in the query request is a better choice, this tool can be available. I think that's a win.

zeroimpl2• 3 years ago

TIL sending a POST request to a URL invalidates the cache for GET requests of the same URL.

Going to have to review my REST APIs to make sure that’s not a problem.

tacone2• 3 years ago

Incredible, I did not know that. Does anybody point out to some resources to learn more about this behavior?

brabel2• 3 years ago

I'm sorry, but isn't that the most basic thing possible about GET and POST?? GET is used to read a resource, POST to write to it... why would you NOT expect writing to a resource to invalidate caches of that resource??

yashap2• 3 years ago

I’d like to see this (or something like it) ship. Right now the two most common alternatives are:

1) GET, but base64 encode a json payload as a query param. Main downsides are that you can hit URL size limits, it makes the client and server slightly more complex (base64 encode/decode), and it sucks with browser dev tools - can’t nicely inspect the request in the browser, have to copy and decode it

2) Use POST to search. This confuses both machines and people. On the machines side, it doesn’t play well with caches, retry middleware, etc. On the people side, it confuses developers, ppl monitoring metrics, etc. - you think it’s creating something but it isn’t. Basically goes against all the reasons we have separate GET/POST/PUT/PATCH/DELETE methods in the first place

Yeah, we have workarounds, but they have pretty significant downsides. I’d rather have this new method. Honestly, I think one of the main reasons ppl reach for RPC frameworks over RESTful APIs is the awkwardness around making search queries - this would really fix that issue.

afiori3• 3 years ago

I would say that people should be more open to using HTTP in non-REST ways.

Sometimes you want to send a request and receive a response, with arbirtrary restrictions and side effects that suit your cases.

There are a lot of good reason for rest to exists, but also sometime you want to POST /open-garage-door?t=5-minutes and call it a day

yashap2• 3 years ago

Sure, the odd RPC style HTTP call is fine - not everything is cleanly modelled as a resource. But the lack of a clean way to make GET requests with a more complex request payload is simply a wart/shortcoming in the HTTP protocol IMO, and one that’s very correctable. Sucks to have to abandon RESTfulness even when I have a very “clean fit for the resource model” call to make, but the request body is too complex/large for HTTP query params.

revskill• 3 years ago

What if modern browser support advanced URL bar with POST method ? (Like Postman UI)

niij• 3 years ago

This new method would be idempotent and safe. POST is neither.

But, adding other methods than GET to the address bar in browsers is an interesting idea. It'd affect such a small percent of people it'd most likely be a net time sink than time saver, though.

politician• 3 years ago

I used a SEARCH verb for an REST API years ago. Granted, we only had to support clients that we wrote ourselves. Semantically, however, it made a lot of sense to include a payload in the request body describing the query.

No one else needed to know or care that it existed.

nonethewiser• 3 years ago

Alternative naive solution: Add a body to GET requests.

As I said, this is totally naive. Is there some obvious reason why this is a terrible idea, or not really possible?

earthling8118• 3 years ago

There are a lot of client implementations that don't support this. You already can add a body to GETs but you'll likely run into issues somewhere due to this. I believe that JavaScript's fetch doesn't allow for this as one example. That's a big one that is probably important to have but there are many others.

kevincox• 3 years ago

Because a lot of existing proxies and other middleware will just drop the body. Or even worse not consider it in caching decisions but still proxy it.

samwillis• 3 years ago

If they do add this I hope they also add support to the Browser History pushState API to set the request body. You can obviously set an arbitrary JS state object, but for server rendered "html over the wire" single page apps it would be nice to be able to do a pushState that is sent to the server on refresh.

LinuxBender• 3 years ago

I was under the impression that new methods could not be amended into an existing RFC by revision but would require a new major.minor version all together is that not the case? Or is this worked around with the concept of extensions and if so doesn't that mean vendors of middle boxes are not required to recognize it?

throwaway14356• 3 years ago

i suppose you could post and store a query, give it a number and use get.

malablaster• 3 years ago

One of the arguments for a separate SEARCH verb is to differentiate between the cacheing needs of POST. Aren’t cache headers sufficient for using the existing verbs?

hcarvalhoalves• 3 years ago

    Right now, you have two main options:
    
    Use a GET, and squeeze all the parameters you need in the URL or headers somewhere
    Use a POST, and have the request considered as unsafe & uncacheable

Third option: you POST or PUT to a resource representing the search, then you’re free to redirect and subsequently GETs of this resource can be cached.

Wanting a special method for search hurts the conceptual integrity of HTTP, where resource representation is the core idea to build on top, it isn’t supposed to be just a request/response protocol.

leftnode• 3 years ago

That's exactly what we do: submitting a search/filter form through a POST request which creates a record in the database representing the search/filter criteria and then redirects to the same endpoint with the ID of that record. The endpoint looks for the ID in the query, attempts to load the same record, and if found, executes and returns the search results.

It works very well, is completely transparent to the end user, has automatic query logging, and provides a clean URL that can be easily shared.

zulu-inuoe• 3 years ago

I really wish I'd thought of this method when I was building a search based application. It was my first time and I struggled with structuring everything

tedyoung• 3 years ago

POST or PUT to a resource representing the search, then you’re free to redirect and subsequently GETs of this resource can be cached.

But that only means that specific instance of the search query is cacheable, not the search query itself, no? I presume the POST would create a new identifier for the resource, so that yes, that specific resource is cacheable. (Even if the server says "oh, I've just seen this POSTed query, let me return the same resource ID", another client will still have to POST to the server, a non-cacheable action.)

The idea of cacheable isn't just by the server, but by anything in between the client and the resource. By using QUERY, the query itself can be cached.

pzlarsson• 3 years ago

This sounds promising as a lot of use cases are shoe horned into the current verbs. A bit surprised EVALUATE isn't discussed as a name since it would make sense both for search queries and other "get using body" scenarios.

wizofaus• 3 years ago

One of the user cases listed - being able to send a large body of data to a server for it to encrypt (though it could be any sort of transformation, e.g. reformatting, encoding, format conversion, or even operations like "spell check" etc.) strongly suggests to me we don't need a new verb, but should simply redefine the expected behaviour for GET requests for when they do or don't have a body - even if it means there's a expectation and method for clients to first query servers whether they have that ability (as servers that don't support "search" will return an error if that verb is used, whereas servers, esp. proxies/load balancers etc, may well simply ignore the body if GET is used.). And to not define caching semantics seems somewhat crazy - it's bound to result in implementations inventing their own (the obvious one would be to combine some hash of the body with the URL to uniquely identify the request).

Too2• 3 years ago

By adding the new verb, it implicitly formalizes that body in GET should not be used. Previously it was allowed, although not always expected.

This may be a good thing and it could also be a bad thing if middleware starts relying on this new expectation and break old applications that previously assumed get bodies to be passed through.

They should clarify the expected behavior of GET bodies more explicit, whether it is allowed or not doesn’t really matter, as long as it is crystal clear.

pan69• 3 years ago

Another way to think about it, if you're doing REST as it seems the author of this article seems to strive for is to introduce a "search resource". You'd then POST to e.g. POST /searches with the query in the body, and on the server side a search resource is created. The server performs the search query and returns the result with a search ID. To do the same search again just do a GET /searches/ID.

BugsJustFindMe2• 3 years ago

GET requests ... can't have a request body. It's not specifically banned, but it is defined as being completely meaningless

This is widely believed but false. And repeating it leads me to believe that the author is not careful enough to create new standards. They are just not defined to be meaningful, which is not the same thing.

In fact the standard says that sending a message body with a GET request should not (specifically SHOULD NOT, not MAY NOT) be done unless you have confirmed that the servers accept it, because servers are not required to accept it, but that if you are talking to a server directly that you know supports it then it's perfectly fine and within the standard.

poisonborz• 3 years ago

As other commenters pointed out, support is spotty and it's generally a greyzone. A new verb (I'd also support "QUERY") would clarify things.

akvadrako• 3 years ago

But it doesn't actually work in many browsers.

mdale• 3 years ago

Maybe read only query getting larger than the GET length limit indicates it's post-processing the object for you and not a "cachable" resource from a key value retrieval perspective? I.e your POST'ing a read only query since an application has to process the query for your at that point ?

Defining a new HTTP method: HTTP Search (2021)

Related Stories