Binance built a 100PB log service with Quickwit

https://quickwit.io/blog/quickwit-binance-story

by samber • 2 years ago

228 194 2 years ago

AJSDfljff9• 2 years ago

Unfortunate the interesting part is missing.

Its not hard at all to scale to PB. Junk your data based on time, scale horizontally. When you can scale horizontally it doesn't matter how much it is.

Elastic is not something i would use for scaling horizontally basic logs, i would use it for live data which i need live with little latency or if i do constantly a lot of log analysis live again.

Did Binance really needed elastic or did they just start pushing everything into elastic without every looking left and right?

Did they do any log processing and cleanup before?

fulmicoton11• 2 years ago

This is their application logs. They need to search into it in a comfortable manner. They went for a search engine with Elasticsearch at first, and Quickwit after that because even after restriction the search on a tag and a time window "grepping" was not a viable option.

AJSDfljff9• 2 years ago

Would be curious what they are searching exactly.

At this size and cost, aligning what you log should save a lot of money.

jcgrillo10• 2 years ago

This position has always confused me. IME logs search tools (ELK and their SaaS ilk) are always far too restrictive and uncomfortable compared to Hadoop/Spark. I'd much rather have unfettered access to the data and have to wait a couple seconds for my query to return than be pigeonholed into some horrible DSL built around an indexing scheme. I couldn't care less about my logs queries returning in sub-second time, it's just not a requirement. The fact that people index logs is baffling.

randomtoast• 2 years ago

I wonder how much their setup costs. Naively, if one were to simply feed 100 PB into Google BigQuery without any further engineering efforts, it would cost about 3 million USD per month.

AJSDfljff9• 2 years ago

Good question. I thought it would be a no brainer to put it on s3 or similiar but thats already way to expensive at 2m/month without api requests.

Backplace storage pods are an initial investment of 5 Million, thats probably the best bet you could do and on that savings level, having 1-3 good people dedicated to this is probably still cheaper.

But you could / should start talking to the big cloud providers to see if they are flexible enough going lower on the price.

I have seen enough companies, including big ones, being absolut shitty in optimizing these types of things. At this level of data, i would optimize everyting including encoding, date format etc.

But i said it in my other comment: the interesting questions are not answered :D

orf• 2 years ago

The compressed size is 20pb, so it’s about 500k per month in S3 fees

Aurornis• 2 years ago

They provide some big hints about the number of vCPUs and the size of the compressed data set on S3:

Size on S3 (compressed): 20 PB

There are also charts about vCPUs and RAM for the indexing and searching clusters.

gaogao• 2 years ago

Yeah, doing some preferred cloud Data Warehouse with an indexing layer seems fine for this sort of thing. That has an advantage over something specialized like this of still being able to easily do stream processing / Spark / etc, plus probably saves some money.

Maybe Quickwit is that indexing layer in this case? I haven't dug too much into the general state of cloud dw indexing.

fulmicoton11• 2 years ago

Quickwit is designed to do full-text search efficiently with an index stored on an object storage.

There are no equivalent technology, apart maybe:

- Chaossearch but it is hard to tell because they are not opensource and do not share their internals. (if someone from chaossearch wants to comment?)

- Elasticsearch makes it possible to search into an index archived on S3. This is still a super useful feature as a way to search punctually into your archived data, but it would be too slow and too expensive (it generates a lot of GET requests) to use as your everyday "main" log search index.

Daviey• 2 years ago

"Object storage as the primary storage: All indexed data remains on object storage, removing the need for provisioning and managing storage on the cluster side."

So the underlying storage is still Object storage, so base that around your calculations depending if you are using S3, GCP Object Storage, self hosted Ceph, MinIO, Garage or SeaweedFS.

onlyrealcuzzo2• 2 years ago

A lot.

1PB with triple redundancy costs around ~$20k just in hard drive costs per year. That's ~$2.5M per year just in disks.

I'd be impressed if they're doing this for less than $1.5M per month (including SWE costs).

Obviously, if they can, saving $1.5M a month vs BigQuery seems like maybe a decent reason to DIY.

BiteCode_dev5• 2 years ago

Why per year? If they buy their own server, they keep the disk several years.

The money motivation to self host on bare metal at this scale is huge.

the_arun• 2 years ago

DIY also comes with the cost of managing it. We need a team to maintain, bug fix etc., not hard but cost

francoismassot3• 2 years ago

Good question.

Let's estimate the costs of compute.

For indexing, they need 2800 vCPUs[1], and they are using c6g instances; on-demand hourly price is $0.034/h per vCPU. So indexing will cost them around $70k/month.

For search, they need 1200 vCPUs, it will cost them around $30k/month.

For storage, it will cost them $23/TB * 20000 = $460k/month.

Storage costs are an issue. Of course, they pay less than $23/TB but it's still expensive. They are optimizing this either by using different storage classes or by moving data to cheaper cloud providers for long term storage (less requests mean you need less performant storage and usually you can get a very good price on those object storages).

On quickwit side, we will also improve the compression ratio to reduce the storage footprint.

[1]: I fixed the num vCPUs number of indexing, it was written 4000 when I published the post, but it corresponded to the total number of vCPUs for search and indexing.

rcaught• 2 years ago

Savings plans, spot, EDP discounts. Some of these have to be applied, right?

endorphine3• 2 years ago

What would you use for storing and querying long-term audit logs (e.g. 6 months retention), which should be searchable with subsecond latency and would serve 10k writes per second?

AFAICT this system feels like a decent choice. Alternatives?

jjordan• 2 years ago

NATS?

packetlost3• 2 years ago

NATS doesn't really have advanced query features though. It has a lot of really nice things, but advanced querying isn't one of them. Not to mention I don't know if NATS does well with large datasets, does it have sharding capability for it's KV and object stores?

bojanz• 2 years ago

You'll find many case studies about using Clickhouse for this purpose.

hipadev23• 2 years ago

Do you know any specific case studies for unstructured logs on clickhouse?

I think achieving sub-second read latency of adhoc text searching over ~150B rows of unstructured data is going to be quite challenging without a high cost. Clickhouse’s inverted indices are still experimental.

If the data can be organized in a way that is conducive to the searching itself, or structured it into columns, that’s definitely possible. Otherwise I suppose a large number of CPUs (150-300) to split the job and just brute force each search?

SSLy2• 2 years ago

What if I don't have such latency requirements? I'm willing to trade that for flexibility or anything else

AJSDfljff9• 2 years ago

I would question first if the system needs to search with subsecond latency and if the same system needs to be which can handle 10k writes/sec.

Even google cloud and others let you wait for longer search queries. If not business ciritical, you can definitly wait a bit.

And the write system might not need to write it in the endformat. Especially as it also has to handle transformation and filtering.

Nonetheless, as mentioned in my other comment, the interesting details of this is missing.

endorphine3• 2 years ago

Let's say that it powers a "search logs" page that an end user wants to see. And let's say that they want last 1d, 14d, 1m, 6m.

So subsecond I would say is a requirement.

And no, it doesn't have to be the same system that ingests/indexes the logs.

jakjak123• 2 years ago

10k audit logs per sec? I think we have different definitions of audit logs.

kstrauser3• 2 years ago

How? If Binance had a trillion transactions, that’s 100KB per transaction. What all are they logging?

francoismassot3• 2 years ago

They have 181 trillion logs

kstrauser3• 2 years ago

But of what? What has Binance done 181 trillion times?

Obviously they have. I don’t think they’re throwing away money for logs they don’t generate or need. I just can’t imagine the scope of it.

That is, I know this is a failing of my imagination, not their engineering decisions. I’d love to fill in my knowledge gaps.

askl• 2 years ago

Don't forget the logs produced by the logging infrastructure.

kstrauser3• 2 years ago

And what if that infra goes down? Who's watching that?

TacticalCoder• 2 years ago

Yeah I don't get it either, something seems deeply wrong here.

These are impressive numbers but I wonder about something... Binance is a centralized cryptocurrencies exchange. But AIUI "defi" is a thing: instead of using a CEX (Centralized EXchange), people can use a DEX (Decentralized EXchange).

And apparently there's a gigantic number of trades happening in the defi world. And it's all happening on public ledgers right? (I'm asking, I don't know: are the "level 2" chains public?).

And the sum of all the public ledgers / blockchains transactions do not represent anywhere near 1.6 PB a day.

And yet DEXes do work (and often now, seen that volume picked up, have liquidity and fees cheaper than centralized exchanges).

Someone who know this stuff better than I do could comment but from a quick googling here's what I found:

    - a full Ethereum node today is 1.2 TB (not PB, but TB)
    - a full Bitcoin node today is 585 GB

There are other blockchains but these two are the two most successful ones?

So let's take Ethereum... For 1.2 TB you have the history of all the transactions that ever happened on Ethereum, since 2015 or something. And that's not just Ethereum but also all the "tokens" and "NFTs" on ethereum, including stablecoins like Circle/Coinbase's USDC.

How do we go from a decentralized "1.2 TB for the entire Ethereum history" to "1.6 PB per day for Binance transactions"?

That's 1500x the size of the freaking entire Ethereum blockchain generated by logs, in a day.

And Ethereum is, basically, a public ledger. So it is... Logs?

Or let's compared Binance's numbers to, say, the US equities options market. That feed is a bit less than 40 Gb/s I think, so 140 TB for a day of actual equities options trading datafeed.

I understand these aren't "logs" but, same thing...

How do we go from a daily 140 TB datafeed from the CBOE (where the big guys are and the real stuff if happening) to 10x that amount in daily Binance logs.

Something doesn't sound right.

You can say it's apples vs oranges but I don't it's that much of apples vs oranges.

I mean: if these 1.6 PB of Binance logs per day are justified, it makes me think it's time to sell everything and go all-in in cryptocurrencies, because there may be way more interest and activity than people think. (yeah, I'm kidding)

EDIT: 1.2 TB for Ethereum seems to be a full but not an "archive" node. An "archive" node is 6 TB. Still a far cry from 1.6 PB a day.

arandomusername2• 2 years ago

Archive node can go as low as ~2TB depending on the client, but regardless you do want to compare it to a full node, as the only difference is an "archive" node keeps the state at any block (e.g you want to replay a transaction that occured in the past, or see a state at a certain block). The full nodes still contain all events/txs (DEXes will emit events, akin to logs, for transfers) - so it's fair to compare it to a full node.

On CEXes you see a lot more trades, market makers/high frequency trades just doing a ton of volume/creating orders and so on, since it's a lot cheaper. CEXes without a doubt have more trades than DEX by orders of magnitude.

Additionally, they probably log multiple stuff for every request, including API request, which they most definitely get a ton of.

tommek40774• 2 years ago

Defi is a fraction of whats happening on cefi exchanges. I have a customer placing hundreds of billions of orders per day on binance. And they are by far not the biggest players there.

namjh• 2 years ago

Albeit 1.6PB/day may be somewhat exaggerated, comparing Binance to Ethereum needs much more consideration including:

- Ethereum's transaction throughput is normally 12~20tx/sec so there cannot be a "high-frequency trades" on Ethereum smart contracts with naive contract interaction(it will cost enormous fees). There are scaling concept like "layer-2" or "layer-3", but they still cannot beat highly optimized centralized server applications. Decentralized exchanges have different schemes to centralized ones to reduce txs to discover the price(keyword: AMM, "automated market maker")

- The transactions per sec metrics are just recording "confirmed" txs by the blockchain, and many "retail-squeezing" trading txs (called MEV, maximal value extraction) are competing behind the blockchain and only one tx is chosen by the blockchain, which will rebate most profits to the blockchain validator(which is analogous to the HFTs on the centralized exchanges).

- The blog post's argument would count all logs of intermediate hops, like L7/L4 proxy and matching engine and so on, and Ethereum's full node storage is only a single component which is almost like a non-parallelized matching engine. Maybe we should also count logs of public RPC nodes of Ethereum? (Also many txs are not gossiped to the public mempool so these are hard to count)

tommek40774• 2 years ago

High frequency traders are making hundreds of billions of orders per day. And there are many bigger and smaller players.

KaiserPro13• 2 years ago

A word of caution here: This is very impressive, but almost entirely wrong for your organisation.

Most log messages are useless 99.99% of the time. Best likely outcome is that its turned into a metric. The once in the blue moon outcome is that it tells you what went wrong when something crashed.

Before you get to shipping _petabytes_ of logs, you really need to start thinking in metrics. Yes, you should log errors, you should also make sure they are stored centrally and are searchable.

But logs shouldn't be your primary source of data, metrics should be.

things like connection time, upstream service count, memory usage, transactions a second, failed transactions, upsteam/downstream end point health should all be metrics emitted by your app(or hosting layer), directly. Don't try and derive it from structured logs. Its fragile, slow and fucking expensive.

comparing, cutting and slicing metrics across processes or even services is simple, with logs its not.

p-o2• 2 years ago

I would say from my experience, for _application logs_, it's the exact opposite. When you deal with a few GB/day of data, you want to have logs, and metrics can be derived from those logs.

Logs are expensive compared to metrics, but they convey a lot more information about the state of your system. You want to move towards metrics over time only one hotspot at a time to reduce cost while keeping observability of your overall system.

I'll take logs over metrics any day of the week, when cost isn't prohibitive.

KaiserPro13• 2 years ago

I was at a large financial news site, They were a total splunk shop. We had lots real steel machines shipping and chunking _loads_ of logs. Every team had a large screen showing off key metrics. Most of the time they were badly maintained and broken, so only the _really_ key metrics worked. Great for finding out what went wrong, terrible at alerting when it went wrong.

However, over the space of about three years we shifted organically over to graphite+grafana. There wasn't a top down push, but once people realised how easy it was to make a dashboard, do templating and generally keep things working, they moved in droves. It also helped that people put metrics emitting system into the underlying hosting app library.

What really sealed the deal was the non-tech business owners making or updating dashboards. They managed to take pure tech metrics and turn them into service/business metrics.

lmpdev• 2 years ago

I’m not well versed in QA/Sysadmin/Logs but surely metrics suffer from Simpson’s paradox compared to properly probed questions only answered through having access to the entirety of the logs?

If you average out metrics across all log files you’re potentially reaching false or worse inverse conclusions about multiple distinct subsets of the logs

It’s part of the reason why statisticians are so pedantic about the wording of their conclusions and to which subpopulation their conclusions actually apply to

FridgeSeal• 2 years ago

Logs are expensive compared to metrics, but they convey a lot more information about the state of your system.

My experience has been the kind of opposite.

Yes, you can put more fields in a log, and you can nest stuff. In my experience however, attics tend to give me a clearer picture into the overall state (and behaviour) of my systems. I find them easier and faster to operate, easier to get an automatic chronology going, easier to alert on, etc.

Logs in my apps are mostly relegated to capturing warning error and error states for debugging reference as the metrics give us a quicker and easier indicator of issues.

zarathustreal• 2 years ago

I’ve got to disagree here, especially with memoization and streaming, deriving metric from structured logs is extremely flexible, relatively fast, and can be configured to be as cheap as you need it to be. With streaming you can literally run your workload on a raspberry pi. Granted, you need to write the code to do so yourself, most off-the-shelf services probably are expensive

KaiserPro13• 2 years ago

memoization and streaming,

memoization isn't free in logs, you're basically deduping an unbounded queue and its difficult to scale from one machine. Its both CPU and Memory heavy. I mean sure you can use scuba, which is great, but that's basically a database made to look like a log store.

deriving metric from structured logs is extremely flexible

Assuming you can actually generate structured logs reliably. but even if you do, its really easy to silently break it.

With streaming you can literally run your workload on a raspberry pi

no, you really can't. Streaming logs to a centralised place is exceptionally IO heavy. If you want to generate metrics from it, its CPU heavy as well. If you need speed, then you'll also need lots of RAM, otherwise searching your logs will cause logging to stop. (either because you've run out of CPU, or you've just caused the VFS cache to drop because you're suddenly doing no predictable IO. )

greylog exists for streaming logs. hell, even rsyslog does it. Transporting logs is fairly simple, storing and generating signal from it is very much not.

reisse• 2 years ago

Metrics are only good when you can disregard some amount of errors without investigation. But they're a financial organization, they have a certain amount of liability. Generalized metrics won't help to understand what happened to that one particular transaction that failed in a cumbersome way and caused some money to disappear.

fells• 2 years ago

It's always struck me that these are two wildly different concerns though.

Use metrics & SLOs to help diagnose the health of your systems. Derive those directly from logs/traces, keep a sample of the raw data, and now you can point any alert to the sampled data to help go about understanding a client-facing issue.

But, for auditing of a particular transaction, you don't need full indexing of the events? You need a transactional journal for every account/user, likely with a well-defined schema to describe successful changes and failed attempts. Perhaps these come from the same stream of data as the observability tooling, but I can only imagine it must be a much smaller subset of the 100PB that you can avoid doing full inverse indexes on this, because your search pattern is simply answering "what happened to this transaction?"

KaiserPro13• 2 years ago

You can still have logs. What I'm suggesting is that vast amounts of unstructured logs, are worse than useless.

Metics tell you where and when something when wrong. Logs tell you why.

However, a logging framework, which is generally lossy, and has the lowest level of priority in terms of deliverability is not an audit mechanism. especially as nowhere are ACLs or verifiability is mentioned. How do they prove that those logs originates from that machine?

If you're going to have an audit mechanism, some generic logging framework is almost certainly a bad fit.

pavlov• 2 years ago

"But they're a financial organization, they have a certain amount of liability."

In the loosest possible sense. Binance is an organization that pretended it doesn't have any physical location in any jurisdiction. Its founder is currently in jail in the United States.

_boffin_2• 2 years ago

Hogwash. I’ll agree that it’s not as simple with logs, but amazingly powerful, and even more so with distributed tracing.

They both have their places and are both needed.

Without logs, I would not have been able to pinpoint multiple issues that plagued our systems. With logs, we were able to tell google, Apigee, it was there problem, not ours. With tracing, we were able to tell a legacy team they had an issue and was able to pinpoint it after them telling us for 6 months that it was our fault. Without logging and tracing, we wouldn’t have been able to tell our largest client, that we never received a 1/3 of their requests they sent us as our company was running around frantically.

They’re both needed, but for different things…ish.

KaiserPro13• 2 years ago

You're missing my main point: logs should not be your primary source of information.

Without logs, I would not have been able to pinpoint multiple issues that plagued our systems.

Logs are great for finding out what went wrong, but terrible at telling there is a problem. This is what I mean by primary information source. If you are sifting through TBs logs to pinpoint a issue, it sucks. Yes, there are tools, but its still hard.

Logs are shit for deriving metrics, it usually requires some level of bespoke processing which is easy to break silently, especially for rarer messages.

pawelduda• 2 years ago

With logs you can get an idea of what events happened in what order during some complex process, stretched over long timeframe, and so on. I don't think you can do this with a metric

KaiserPro13• 2 years ago

With logs you can get an idea of what events happened in what order

Again, if you're at that point, you need logs. But thats never going to be your primary source of information. if you have more than a few services running at many transactions a second, you can't scale that kind of understanding using logs.

This is my point, if you have >100 services, each with many tens or hundreds of processes, your primary (well it shouldn't be, you need pre SLA fuckup alerts)alert to something going wrong is something breeching an SLA. That's almost certainly a metric. Using logs to derive that metric means you have a latency of 60-1500 seconds

Getting your apps to emit metrics directly means that you are able to make things much more observable. It also forces your devs to think about _how_ their app is observed.

derefr• 2 years ago

I would note that a notional "log store", doesn't have to just be used for things that are literally "logs."

You know what else you could call a log store? A CQRS/ES event store.

(Specifically, a "log store" is a CQRS/ES event store that just so happens to also remember a primary-source textual representation for each structured event-document it ingests — i.e. the original "log line" — so that it can spit "log lines" back out unchanged from their input form when asked. But it might not even have this feature, if it's a structured log store that expects all "log lines" to be "structured logging" formatted, JSON, etc.)

And you know what the most important operation a CQRS/ES event store performs is? A continuous streaming-reduction over particular filtered subsets of the events, to compute CQRS "aggregates" (= live snapshot states / incremental state deltas, which you then continuously load into a data warehouse to power the "query" part of CQRS.)

Most CQRS/ES event stores are built atop message queues (like Kafka), or row-stores (like Postgres). But neither are actually very good backends for powering the "ad-hoc-filtered incremental large-batch streaming" operation.

• With an MQ backend, streaming is easy, but MQs maintain no indices for events per se, just copies of events in different topics; so filtered streaming would either have the filtering occur mostly client-side; or would involve a bolt-on component that is its own "client-side", ala Kafka Streams. You can use topics for this — but only if you know exactly what reduction event-type-sets you'll need before you start publishing any events. Or if you're willing to keep an archival topic of every-event-ever online, so that you can stream over it to retroactively build new filtered topics.

• With a row-store backend, filtered streaming without pre-indexing is tenable — it's a query plan consisting of a primary-key-index-directed seq scan with a filter node. But it's still a lot more expensive than it'd be to just be streaming through a flat file containing the same data, since a seq scan is going to be reading+materializing+discarding all the rows that don't match the filtering rule. You can create (partial!) indices to avoid this — and nicely-enough, in a row-store, you can do this retroactively, once you figure out what the needs of a given reduction job are. But it's still a DBA task rather than a dev task — the data warehouse needs to be tweaked to respond to the needs of the app, every time the needs of the app change. (I would also mention something about schema flexibility here, but Postgres has a JSON column type, and I presume CQRS/ES event-store backends would just use that.)

A CQRS/ES event store built atop a fully-indexed document store / "index store" like ElasticSearch (or Quickwit, apparently) would have all the same advantages of the RDBMS approach, but wouldn't require any manual index creation.

Such a store would perform as if you took the RDBMS version of the solution, and then wrote a little insert-trigger stored-procedure that reads the JSON documents out of each row, finds any novel keys in them, and creates a new partial index for each such novel key. (Except with much lower storage-overhead — because in an "index store" all the indices share data; and much better ability to combine use of multiple "indices", as in an "index store" these are often not actually separate indices at all, but just one index where the key is part of the index.)

---

That being said, you know what you can use the CQRS/ES model for? Reducing your literal "logs" into metrics, as a continuous write-through reduction — to allow your platform to write log events, but have its associated observability platform read back pre-aggregated metrics time-series data, rather than having to crunch over logs itself at query time.

And AFAIK, this "modelling of log messages as CQRS/ES events in a CQRS/ES event store, so that you can do CQRS/ES reductions to them to compute metrics as aggregates" approach is already widely in use — but just not much talked about.

For example, when you use Google Cloud Logging, Google seems to be shoving your log messages into something approximating an event-store — and specifically, one with exactly the filtered-streaming-cost semantics of an "index store" like ElasticSearch (even though they're actually probably using a structured column-store architecture, i.e. "BigTable but append-only and therefore serverless.") And this event store then powers Cloud Logging's "logs-based metrics" reductions (https://cloud.google.com/logging/docs/logs-based-metrics).

londons_explore• 2 years ago

When you have metrics, you should also keep sampled logs.

Ie. 1 per million log entries is kept. Write some rules to try and keep more of the more interesting ones.

One way to do this is to have your logging macro include the source file and line number the logline came from, and then, for each file and line number emit/store no more than 1 logline per minute.

That way you get detailed records of rare events, while filtering most of the noise.

hooverd• 2 years ago

There are also different types of logs. Maybe you want every transaction action but don't need a full fidelity copy of every load balancer ping from the last ten years.

ryukoposting3• 2 years ago

Metrics are useful when you know what to measure, which implies that you already have a good idea for what can go wrong. If your entire product exists in some cloud servers that you fully control, that's probably feasible. Binance probably could have done something more elegant than storing extraordinary amounts of logs.

However, if you're selling a physical product, and/or a service that integrates deeply with third party products/services, it becomes a lot more difficult to determine what's even worth measuring. A conservative approach to metrics collection will limit the usefulness of the metrics, for obvious reasons. A "kitchen sink" approach will take you right back to the same "data volume" problem you had with logs, but now your developers have to deal with more friction when creating diagnostics. Neither extreme is desirable, and finding the middle ground would require information that you simply don't have.

On a related note, one approach I've found useful (at a certain scale) is to shove metrics inside of the logs themselves. Put a machine-readable suffix on your human-readable log messages. The resulting system requires no more infrastructure than what your logs are already using, and you get a reliable timeline of when certain metrics appear vs. when certain log messages appear.

temporarely2• 2 years ago

Any system has a 'natural set' of metrics. And metrics are not about "what [went] wrong" rather system health. So Metrics -> Alert -> Log Diagnostics.

BonusPlay• 2 years ago

When performing forensic analysis, metrics don't usually help that much. I'd rather sift 2PB of logs, knowing that information I'm looking for is in there, than sit at the usual "2 weeks of nginx access logs which roll over".

Obviously running everything with debug logging just burns through money, but having decent logs can help a lot other teams, not just the ones working on the project (developers, sysadmins, etc.)

anonygler• 2 years ago

"Once in a blue moon" -- you mean the thing that constantly happens? If you're not using logs, you're not practicing engineering. Metrics can't really diagnose problems.

It's also a lot easier to inspect a log stream that maps to an alert with a trace id than it is to assemble a pile of metrics for each user action.

hot_gril• 2 years ago

I think the above comment is just saying that you shouldn't use logs to do the job of metrics. Like, if you have an alert that goes off when some HTTP server is sending lots of 5xx, that shouldn't rely on parsing logs.

zzyzxd2• 2 years ago

Most log messages are useless 99.99% of the time. Best likely outcome is that its turned into a metric. The once in the blue moon outcome is that it tells you what went wrong when something crashed.

If it crashes, it's probably some scenario that was not properly handled. If it's not properly handled, it's also likely not properly logged. That's why you need verbose logs -- once in a blue moon you need to have the ability to retrospectively investigate something in the past that was not thought through, without using a time machine.

This is more common in the financial world where audit trail is required to be kept long term for regulation. Some auditor may ask you for proof that you have done a unit test for a function 3 years ago.

Every organization needs to find their balance between storage cost and quality of observability. I prefer to keep as much data as we are financially allowed. If Binance is happy to pay to store 100PB logs, good for them!

"Do we absolutely need this data or not" is a very tough question. Instead, I usually ask "how long do we need to keep this data" and apply proper retention policy. That's a much easier question to answer for everyone.

KaiserPro13• 2 years ago

If it's not properly handled, it's also likely not properly logged

Then you're blue moon probability if it being useful rapidly drops. Verbose logs are simply a pain in the arse, unless you have a massive processing system. but even then it just either kneecaps your observation window, or makes your queries take ages.

I am lucky enough to work at a place that has really ace logging capability, but, and I cannot stress this enough, it is colossally expensive. literal billions.

but, logging is not an audit trail. Even here where we have fancy PII shields and stuff, logging doesn't have the SLA to record anything critical. If there is a capacity crunch, logging resolution gets turned down. Plus logging anything of value to the system gets you a significant bollocking.

If you need something that you can hand to a government investigator, if you're pulling logs, you're already in deep shit. An audit framework needs to have a super high SLA, incredible durability and strong authentication for both people and services. All three of those things are generally foreign to logging systems.

Logging is useful, you should log things, but, you should not use it as a way to generate metrics. verbose logs are just a really efficient way to burn through your infrastructure budget.

jen20• 2 years ago

It is quite unlikely that a regulator will ask you for proof you have a unit test for anything (also, that's not what a unit test is - see [1] for a good summary of why).

It _is_ likely a regulator will ask you to prove that you are developing within the quality assurance framework you have claimed you are, though.

Finally though, logs are not an audit trail, and almost no-one can prove their logs are correct with respect to the state of the system at any given time.

[1] https://www.youtube.com/watch?v=EZ05e7EMOLM

pphysch• 2 years ago

But logs shouldn't be your primary source of data, metrics should be.

Metrics, logs, relational data, KVs, indexes, flat files, etc. are all equally valid forms of data for different shapes of data and different access patterns. If you are building for a one-size-fits-all database you are in for a nasty surprise.

galkk• 2 years ago

Just to be sure, I'm speaking below about application/system logs, not as "our event sourcing uses log storage"

Yes, you probably don't want to store debug logs of 2 years ago, but logs and metrics solve very different problems.

Logs need to have determined lifecycle, e.g. most detailed logs are stored for 7/14/30/release cadence days, then discarded. But when you need to troubleshoot something, metrics give you signal, but logs give you information about what was going on.

andrewf4• 2 years ago

As an engineer I generally want logs so I can dive into problems that weren't anticipated. Debugging.

I get a lot of pushback from ops folks. They often don't have the same use case. The logs are for the things that'll be escalated beyond the ops folks to the people that wrote the bug.

Yes, most (> 99.99%) of them will never be looked at. But storage is supposed to be cheap, right? If we can waste bytes on loading a copy of Chromium for each desktop application, surely we can waste bytes on this.

My argument is completely orthogonal to "do we want to generate metrics from structured logs".

andmarios• 2 years ago

Most probably, said ops folks have quite a few war stories to share about logs.

Maybe a JVM-based app went haywire, producing 500GB of logs within 15 minutes, filling the disk, and breaking a critical system because no one anticipated that a disk could go from 75% free to 0% free in 15 minutes.

Maybe another JVM-based app went haywire inside a managed Kubernetes service, producing 4 terabytes of logs, and the company's Google Cloud monthly usage went from $5,000 to $15,000 because storing bytes is supposed to be cheap when they are bytes and not when they are terabytes.

I completely agree that logs are useful, but developers often do not consider what to log and when. Check your company's cloud costs. I bet you the cost of keeping logs is at least 10%, maybe closer to 25% of the total cost.

jiggawatts2• 2 years ago

Something I’ve discovered is that Azure App Insights can capture memory snapshots when an exception happens. You can download these with a button press and open in Visual Studio with a double-click.

It’s magic!

The stack variables, other threads and most of the heap is right there as-if you had set a breakpoint and it was an interactive debug session.

IMHO this eliminates the need for 99% of the typical detailed tracing seen in large complex apps.

mianos2• 2 years ago

Storage is not cheap at this scale. That would be 100s of thousands a year at the very least. (How I know, I work in an identical area and have huge budget problems with rando verbose logging).

__0x01• 2 years ago

Error level logging can exist with a metrics focused approach.

Log_out_• 2 years ago

My system has a version number and input + known starting state dbwise. Now assuming i have determenistic reprodible state, is a log just a replay of that game engine at work?

9dev• 2 years ago

My response to that would be that you can enable logging locally, or in your staging environment, but not in production. If an error occurs, your telemetry tooling should gather a stack trace and all related metadata, so you should be able to reproduce or at least locate the error.

But all other logs produced at runtime are breadcrumbs that are only ever useful when an exception occurs, anyway. Thus, you don’t need them otherwise.

ansgri• 2 years ago

You probably don't need full text search, but only exact match search and very efficient time-based retrieval of contiguous log fragments. As an engineer spending quite a lot of time debugging and reading logs, our Opensearch has been almost useless for me (and a nightmare for our ops folks), since it can miss searches on terms like filenames and OSD UX is slow and generally unpleasant. I'd rather have a 100MB of text logs downloaded locally.

Please enlighten me, what are use cases for real full-text search (with fuzzy matching, linguistic normalization etc.) in logs and similar machine-generated transactional data? I understand its use for dealing with human-written texts, but these are rarely in TB range, unless you are indexing the Web or logs of some large-scale communication platform.

sgarland• 2 years ago

I simply doubt that most of these logs (or anyone’s, usually) are that useful.

I worked at a SaaS observability company (Datadog competitor) that was ingesting, IIRC, multiple GBps of metrics, spread across multiple regions, dozens upon dozens of cells, etc. Our log budget was 650 GB/day.

I have seen – entirely too many times – DEBUG logs running in prod endlessly, messages that are clearly INFO at best classified as ERROR, etc. Not to mention where a 3rd party library is spamming the same line continuously, and no one bothers to track down why and stop it.

tuyguntn• 2 years ago

Most log messages are useless 99.99% of the time.

Things are useless until first crash happens, same thing applies to replication, you don't need replication until your servers start crashing.

But logs shouldn't be your primary source of data, metrics should be.

There are different types of data related to the product:

    * product data - what's in your db
    * logs - human readable details of a journey for a single request
    * metrics - approximate health state of overall system, where storing high cardinality values are bad (e.g. customer_uuid)
    * traces - approximate details of a single request to be able to analyze request journey through systems, where storing high cardinality values might still be bad.

Logs are useful, but costly. Just like everything else which makes system more reliable

giancarlostoro• 2 years ago

Most log messages are useless 99.99% of the time. Best likely outcome is that its turned into a metric. The once in the blue moon outcome is that it tells you what went wrong when something crashed. Wonder if just keeping timestamps in a more efficient table of each unique textual log entry would be better? Or rather log entry text template. Then store the arguments also separate.

sebstefan• 2 years ago

On a given high-throughput Kafka topic, this figure goes up to 11 MB/s per vCPU.

There's got to be 2x to 10x improvement to be made there, no? No way CPU is the limitation these days and even bad hard drives will support 50+mB/s write speeds.

ddorian433• 2 years ago

Building inverted index is very CPU intensive.

fulmicoton11• 2 years ago

Building an inverted index is actually very cpu intensive. I think we are the fastest on that (if someone knows something faster than tantivy at indexing I am interested).

I'd be really surprised if you can make a 10x improvement here.

ZeroCool2u2• 2 years ago

There was a time at the beginning of the pandemic where my team was asked to build a full text search engine on top of a bunch of SharePoint sites in under 2 weeks and with frustratingly severe infrastructure constraints, (No cloud services, single box on prem for processing, among other things), and we did and it served its purpose for a few years. Absolutely no one should emulate what we built, but it was an interesting puzzle to work on and we were able to cut through a lot of bureaucracy quickly that had held us back for a few years wrt accessing the sensitive data they needed to search.

But I was always looking for other options for rebuilding the service within those constraints and found Quickwit when it was under active development. I really admire their work ethic and their engineering. Beautifully simple software that tends to Just Work™. It's also one of the first projects that made me really understand people's appreciation for Rust as well outside of just loving Cargo.

fulmicoton11• 2 years ago

Thank you for the kind word @ZeroCool2u ! :)

hanniabu4• 2 years ago

we were able to cut through a lot of bureaucracy quickly that had held us back for a few years wrt accessing the sensitive data they needed to search

Doesn't sound like a benefit for your users

shortrounddev22• 2 years ago

In what way?

totaa• 2 years ago

I don't know what brings me more happiness in this career. Building systems with no political constraints, or building something that's functional with severe restraints.

elchief• 2 years ago

maybe drop the log level from debug to info...

piterrro• 2 years ago

Reminds me of the time Coinbase paid DataDog $65M for storing logs[1]

[1] https://thenewstack.io/datadogs-65m-bill-and-why-developers-...

inssein2• 2 years ago

Why do so many companies insist on shipping their logs via Kafka? I can't imagine deliverability semantics are necessary with logs, and if they are, they shouldn't be in your logs?

bushbaba• 2 years ago

Don’t forget about all the added cost. never got it as many shops can tolerate data loss for their melt data. So long as it’s collected 99.9% of the time it’s good enough.

mdaniel• 2 years ago

My experience has been a mixture of "when all you have is a hammer ..." and Pointy Haired Bosses LOVE kafka, and tend to default to it because it's what all their Pointy Haired Boss friends are using

In a more generous take, using some buffered ingest does help with not having to choose between a c500.128xl ingest machine and dropping messages, but I would never advocate for standing up kafka just for log buffering

inssein2• 2 years ago

at that point you are likely slowing down your applications - I think a basic OpenTelemetry collector mostly solves this, and if you go beyond the available buffer there, then dropping it is the appropriate choice for application logs.

jcgrillo10• 2 years ago

Kafka is a big dumb pipe that moves the bytes real fast, it's ideal for shipping logs. It accepts huge volumes of tiny writes without breaking a sweat, which is exactly what you want--get the logs off the box ASAP and persisted somewhere else durably (e.g. replicated).

ecnahc515• 2 years ago

In log shipping cases it’s good as a buffer so you can batch writes to the underlying SIEM. This prevents tons of small API calls with a few hundred or thousand log lines each. Instead Kafka will take all the small calls and the SIEM can subscribe and turn them into much larger batches to write to the underlying storage (eg S3).

RIMR2• 2 years ago

I am having trouble understand how any organization could ever need a collection of logs larger than the size of the entire Internet Archive. 100PB is staggering, and the idea of filling that with logs, while entirely possible, just seems completely useless given the cost of managing that kind of data.

This is on a technical level quite impressive though, don't get me wrong, I just don't understand the use case.

jcgrillo10• 2 years ago

Same, also I'd love to know more about the technical details of their logging format, the on-disk storage format, and why they were only able to reduce the storage size to 20% of the uncompressed size. For example, clp[1] can achieve much, much better compression on logs data.

[1] https://github.com/y-scope/clp

EDIT: See also[2][3].

[2] https://www.uber.com/blog/reducing-logging-cost-by-two-order...

[3] https://www.uber.com/blog/modernizing-logging-with-clp-ii/

fulmicoton11• 2 years ago

It is pretty much the same as Lucene. The compression ratio is very specific logs and depends on the logs themselves. (Often it is not that good)

tommek40774• 2 years ago

These are order and trade logs probably. You want to have them and you need them for auditing. Binance wants to be more professional in that way probably. HFT is making billions of orders per day per trader.

jcgrillo10• 2 years ago

OK, so let's do some napkin math... I'm guessing something like this is the information you might want to log:

user ID: 128bits

timestamp: 96bits

ip address: 32bits

coin type: idk 32bits? how many fake internet money types can there be?

price: 32bits

quantity: 32bits

So total we have 352bits. Now let's double it for teh lulz so 704bits wtf not. You know what fuck it let's just round up to 1024bits. Each trade is 128bytes why not, that's a nice number.

That means 200Pb--2e17 bytes mind you--is enough to store 1.5625e16 trades. If all the traders are doing 1e9 trades/day, and we assume this dataset is 13mo of data, that means there are 38772 HFT traders all simultaneously making 11574 trades per second.. That seems like a lot..

In other words, that means Binance is processing 448.75 million orders per second.. Are they though?

EDIT: No, indeed some googling indicates they claim they can process something like 1.4 million TPS. But I'd hazard a guess the actual figure on average is less..

EDIT: err sorry, shoulda been 100Pb. Divide all those numbers by two. Still two orders of magnitude worth of absurd.

cletus• 2 years ago

Just browsing the Quickwit documentation it seems like the general architecture here is to write JSON logs but stores them compressed. Is this just something like gzip compression? 20% compressed size does seem to align to ballpark estimates of JSON GZIP compression. This is what Quickwit (and this page) calls a "document": a single JSON record (just FYI).

Additionally you need to store indices because this is what you actually search. Indices have a storage cost when you write them too.

When I see a system like this my thoughts go to questions like:

- What happens when you alter an index configuration? Or add or remove an index?

- How quickly do indexes update when this happens?

- What about cold storage?

Data retention is another issue. Indexes have config for retention [1]. It's not immediately clear to me how document retention works, possibly from S3 expiration?

So, network transfer from S3 is relatively expensive ($0.05/GB standard pricing [2] to the Internet, less to AWS regions). This will be a big factor in cost. I'm really curious to know how much all of this actually costs per PB per month.

IME you almost never need to log and store this much data and there's almost no reason to ever store this much. Most logs are useless and you also have to question what the purpose is of any given log. Even if you're logging errors, you're likely to get the exact same value out of 1% sampling of logs than you are with logging everything.

You might even get more value with 1% sampling because your query and monitoring might be a whole lot easier with substantially less data to deal with.

Likewise, metrics tend to work just as well from sampled data.

This post suggests 60 day log retention (100PB / 1.6PB daily). I would probably divide this into:

1. Metrics storage. You can get this from logs but you'll often find it useful to write it directly if you can. Getting it from logs can be error-prone (eg a log format changes, the sampling rate changes and so on);

2. Sampled data, generally for debugging. I would generally try to keep this at 10TB or less;

3. "Offline" data, which you would generally only query if you absolutely had to. This is particularly true on S3, for example, because the write costs are basically zero but the read costs are expensive.

Additionally, you'd want to think about data aggregation as a lot of your logs are only useful when combined in some way

[1] https://quickwit.io/docs/overview/concepts/indexing

[2] https://aws.amazon.com/s3/pricing/

JackSlateur2• 2 years ago

You have very good questions, I can only guess one answer: s3 network transfer is free for AWS services

Your link[1] said:

  You pay for all bandwidth into and out of Amazon S3, except for the following:
  [...]
  - Data transferred from an Amazon S3 bucket to any AWS service(s) within the same AWS Region as the S3 bucket (including to a different account in the same AWS Region).

fulmicoton11• 2 years ago

Quickwit (like Elasticsearch/Opensearch) stores you data compressed with ZSTD in a row store, builds a full text search index, and stores some of your fields in a columnar. The "compressed size" includes all of this.

The high compression rate is VERY specific to logs.

- What happens when you alter an index configuration? Or add or remove an index?

Changing an index mapping was not available in 0.8. It is available in main and will be added in 0.9. The change only impacts new data.

- Or add or remove an index?

This is handled since the beginning.

- What about cold storage?

What makes Quickwit special is that we are reading everything is on S3. We adapted our inverted index to make it possible to read straight from S3. You might think this is crazy slow, but we typically search into TBs of data in less than a second. We have some in RAM cache too, but they are entirely optional.

2. Sampled data, generally for debugging. I would generally try to keep this at 10TB or less;

Sometimes, sampling is not possible. For instance, some of Quickwit users (including Binance) use their logs for user support too. A user might come asking details about something fishy that happened 2 months ago.

ATsch2• 2 years ago

It's always very amusing how all of the blockchain companies wax lyrical about all of the huge supposed benefits of blockchains and how every industry and company is missing out by not adopting them and should definitely run a hyperledger private blockchain buzzword whatever.

And then, even when faced with implementing a huge, audit critical, distributed append-only store, the thing they tell us blockchains are so useful for, they just use normal database tech like the rest if us. With one centralized infrastructure where most of the transactions in the network actually take place. Who's tech stack looks suspiciously like every other financial institution.

I'm so glad we're ignoring 100 years of securities law to let all of this incredible innovation happen.

tommek40774• 2 years ago

Binance is not a blockchain company. It is a centealized exchange. Nothing is happening on-chain unless getting coins from or to the exchange. And this has nothing tondo with them then.

rijoja• 2 years ago

And then, even when faced with implementing a huge, audit critical, distributed append-only store, the thing they tell us blockchains are so useful for, they just use normal database tech like the rest if us. With one centralized infrastructure where most of the transactions in the network actually take place. Who's tech stack looks suspiciously like every other financial institution.

Right but surely you must understand that the blockchain transactions are already stored in the blockchain, and what this is about is logs that might be useful for debugging purposes, and as such would be more verbose than what's required and also could contain sensitive information?

Apart from that isn't it obvious that the performance requirement would make this unrealistic, with no added benefits, whatsoever.

I'm so glad we're ignoring 100 years of securities law to let all of this incredible innovation happen.

Storing all these logs on a blockchain might very well (apart from being totally asinine) breach privacy regulations as well, as it might very well store sensitive data?

Surely you must understand this?

ATsch2• 2 years ago

Surely you must understand this?

Yes, I understand why blockchains are bad, have no benefits, terrible performance and are a privacy nightmare. Thanks for explaining it in more detail. And binance understands it too, that's why they're not using it (not even a private one!) despite all of their talk about how it's a revolutionary technology.

ram_rar• 2 years ago

Limited Retention: Binance was retaining most logs for only a few days. Their goal was to extend this to months, requiring the storage and management of 100 PB of logs, which was prohibitively expensive and complex with their Elasticsearch setup.

Just to give some perspective. The Internet Archive, as of January 2024, attests to have stored ~ 99 petabytes of data.

Can someone from Binance/quickwit comment on their use case that needed log retention for months? I have rarely seen users try to access actionable _operations_ log data beyond 30 days.

I wonder how much $$ can they save more by leveraging tiered storage and engs being mindful of logging.

drak0n1c• 2 years ago

Government regulators take their time and may not investigate or alert firms to identified theft, vulnerability, criminal or sanctioned country user trails for months. However, that does not protect those companies from liability. There is recent pressure and targeted prosecution from the US on Binance and CZ along this angle. They've been burned on US users getting into their international exchange, so keeping longer forensic logs helps surveil, identify, and restrict Americans better (as well as the bad guys they're not supposed to interact with).

evdubs• 2 years ago

Lots of storage to log all of the wash trading on their platform.

ukuina2• 2 years ago

Does QuickWit support regex search now? The underlying store, Tantivy, already does.

This is what stopped a PoC cold at an earlier project.

PSeitz• 2 years ago

tantivy has two dictionaries FST and SSTable. We added SSTable in tantivy because it works great with object storage, while FST does not. With some metadata we can download only the required parts and not the whole dictionary.

SStable does not support Regex queries, it would require a full load and scan, which would be very expensive.

Your best bet currently would be to make it work with tokenizing, which is way more efficient anyways.

prefix queries are supported btw

ukuina2• 2 years ago

Are in-order queries supported? e.g., TERM1*TERM2 should return matches with those terms in that specific order.

kristopolous• 2 years ago

So people don't build this out themselves?

Regardless, there's some computer somewhere serving this. How do they service 1.6 PB per day? Are we talking tape backup? Disks? I've seen these mechanical arms that can pick tapes from a stack on a shelf, is that what is used? (example: https://www.osc.edu/sites/default/files/press/images/tapelib...)

For disks that's like ~60/day without redundancy, do they have people just constantly building out and onlining machines in some giant warehouse?

I assume there's built in redundancy and someone's job to go through and replace failed units?

This all sounds like it's absurdly expensive.

And I'd have to assume they deal with at least 100x that scale because they have many other customers.

Like what is that? 6,000 disks a day? Really?

I hear these numbers of petabyte storage frequently. I think Facebook is around 5PB/daily. I've never had to deal with anything that large. Back in the colo days I saw a bunch of places but nothing like that.

I'm imagining forklifts moving around pallets of shrink wrapped drives that get constantly delivered

Am I missing something here?

Places like AWS should run tours. It'd be like going to the mint.

jiggawatts2• 2 years ago

The article talks about 1.6 PB / day, which is 150 Gbps of log ingest traffic sustained. That's insane.

A change of the logging protocol to a more efficient format would yield such a huge improvement that it would be much cheaper than this infrastructure engineering exercise.

I suspect that everyone just assumes that the numbers represent the underling data volume, and that this cannot be decreased. Nobody seems to have heard of write amplification.

Let's say you want to collect a metric. If you do this with a JSON document format you'd likely end up ingesting records that are like the following made-up example:

    {
      "timestamp": "2024-07-12T14:30:00Z",
      "serviceName": "user-service-ba34sd4f14",
      "dataCentre": "eastFoobar-1",
      "zone": 3,
      "cluster: "stamp-prd-4123",
      "instanceId": "instance-12345",
      "object": "system/foo/blargh",
      "metric: "errors",
      "units": "countPerMicroCentury",
      "value": 391235.23921386
    }

It wouldn't surprise me if in reality this was actually 10x larger. For example, just the "resource id" of something in Azure is about this size, and it's just one field of many collected by every logging system in that cloud for every record. Similarly, I've cracked open the protocols and schema formats for competing systems and found 300x or worse write amplification being the typical case.

The actual data that needed to be collected was just:

    391235.23921386

In a binary format that would be 4 bytes, 8 if you think that you need to draw your metric graphs with a vertical precision of a millionth of a pixel and horizontal precision of a minute because you can't afford the exabytes of storage a higher collection frequency would require.

If you collect 4 bytes per metric in an array and record the start timestamp and the interval, you don't even need a timestamp per entry, just one per thousand or whatever. For a metric collected every second that's just 10 MB per month before compression. Most metrics change slowly or not at all and would compress down to mere kilobytes.

Binance built a 100PB log service with Quickwit

Related Stories