Kafka is dead, long live Kafka

https://www.warpstream.com/blog/kafka-is-dead-long-live-kafka

by richieartoul • 3 years ago

392 287 3 years ago

ryanworl13• 3 years ago

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges.

If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

rubiquity2• 3 years ago

Congrats! "The SQLite of Kafka" is an item from my side projects pile I'm happy to delete.

One reason I never built it is because it felt paradoxical that users might want a scaled down Kafka rather than using SQLite directly if the scale didn't matter. But you may find out that people enjoy the semantics of the Kafka protocol or are already using Kafka and have learned they don't have the scale they thought they did to warrant the complexity. Best of luck!

derefr5• 3 years ago

it felt paradoxical that users might want a scaled down Kafka rather than using SQLite directly if the scale didn't matter.

I don't need to push very many messages (not enough to justify running Kafka), but each of the messages that I do push are both 1. very important and must be cross-AZ durable, and 2. very urgent and must not be blocked by e.g. contended writes in a regular RDBMS.

Currently, the winner of this use-case for IaaS customers is "whatever cloud-native message-queue service your IaaS offers." (And those customers would also be the extent of WarpStream's Total Addressable Market here, given that WarpStream's architecture fundamentally relies on having a highly-replicated managed object store available.)

I'm therefore curious: in what ways does WarpStream win vs. Amazon SQS / Google Cloud Pub/Sub / Azure Queue Storage?

episiarch• 3 years ago

How do you replace ZooKeeper?

aduffy• 3 years ago

Kafka replaced ZooKeeper with Kafka itself already a few years ago https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

richieartoul36• 3 years ago

(WarpStream founder) WarpStream has a completely different architecture than Kafka: https://docs.warpstream.com/warpstream/background-informatio...

That said it does require a lot of metadata to orchestrate all the different concurrent operations over S3. We handle this with a custom metadata store that we run in our cloud control plane.

dottedmag• 3 years ago

Do you have a reference documentation for S3 data layout?

richieartoul36• 3 years ago

(WarpStream founder here) Not currently. One of the things we're looking to do next is make it so any topic can be "automatically" turned into a standard format in S3, something like Parquet/Iceberg/Deltalake so its easier to consume for application that don't particularly care about the Kafka protocol.

jiaaro• 3 years ago

The blog post mentions that partitions are too low-level an abstraction to program against. Does that mean WarpStream doesn't use partitions?

Do you provide any ordering guarantees like Kafka does at the partition level?

richieartoul36• 3 years ago

(WarpStream founder) No WarpStream has partitions internally and provides the same ordering guarantees Kafka does at the partition level. We're just saying that we think for most streaming applications this is not a great programming model, and we think there is an opportunity to do something better (but we haven't done that yet).

winddude• 3 years ago

Kafka itself no longer requires zookeeper.

pdeva12• 3 years ago

1. dont producers now have much higher latency since they have to wait for writes to s3.

2. if the '5-10x cheaper' is mostly due to cross AZ savings, isnt that offered by AWS MSK offering too?

richieartoul36• 3 years ago

(WarpStream founder)

1. Yeah, we mention at the end of the post the P99 produce latency is ~400ms. 2. MSK still charges you for networking to produce into the cluster and consumer out of it if follower fetch is not properly configured. Also, you still have to more or less manage a Kafka cluster (hot spotting, partition rebalancing, etc). In practice we think WarpStream will be much cheaper to use than MSK for almost all use-cases, and significantly easier to manage.

kbumsik• 3 years ago

Does it support S3-compatible services, notably Cloudflare R2? I heard that there might be special handling for each S3-compatible providers, due the slightly different API behavior and different consistency models, and etc.

If it support Cloudflare R2 then it would be great for multi-cloud too.

geekone• 3 years ago

What was used to generate the diagrams in the post?

cduzz2• 3 years ago

Couple questions:

1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage?

2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clusters, etc.

3. What's done to preserve order-of-operation within a kafka partition?

ilikehurdles• 3 years ago

Re #1 I don’t think Amazon charges for data transfers from within AWS, assuming Kafka is hosted there.

vasco• 3 years ago

S3 requests are paid regardless of source (tldr GETs are cheaper than PUTs). You're confused with data transfer within a VPC.

finnh• 3 years ago

There are per-API-call charges. It's just not per-byte.

morelisp5• 3 years ago

I'm also especially interested in 3) - from the arch overview it sounds like all agents are actively writing and actively compacting, how do they coordinate which topic-partitions to compact? Is the Cloud Metadata Store essentially responsible for handing out the offsets?

richieartoul36• 3 years ago

(WarpStream founder here) Yes exactly. The cloud metadata store assigns offsets, does compaction planning/scheduling, handles service discovery, etc

ryanworl13• 3 years ago

[WarpStream co-founder and CTO here]

1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our cloud control plane.

2. We think people shouldn't have to choose between reliability and costs. WarpStream gives you the reliability and availability of running in three AZs but with the cost of one.

3. We have a custom metadata database running in our cloud control plane which handles ordering.

theikkila2• 3 years ago

Related to 1. If I understood corrently the agent generates single object per each flushing interval containing all data accross all topics it has received. Does this mean that when reading the consumer needs to read multiple partition data simultaneously to access just single partition? How about scaling consumers horizontally how does WarpStream Agent handle horizontal partitioning of the stream from consuming side?

theikkila2• 3 years ago

Does WarpStream guarantee correct order inside partition only for acknowledged messages or also among the acknowledged messages (in different batches)? If so how do you keep clocks synchronized between the agents?

poorman3• 3 years ago

It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.

CyanLite42• 3 years ago

Does Azure’s Append Blob support (and AWS’ lack thereof) provide any inherent performance advantages for Azure vs. AWS?

Fiahil• 3 years ago

We have a custom metadata database running in our cloud control plane which handles ordering.

This is the secret sauce, right there. Anybody can host a bunch of topics and artefacts on S3, and have them locally-replicated if they want. However, there is no world where you can have that without a synchronisation service that ensure cursors are uniques and properly ordered.

tiew9Vii2• 3 years ago

Flushing every 100ms means you would end up with lots of tiny files (bytes) in s3 unless you have something out of process re-writing them in to larger blobs similar to Delta lakes optomize?

The lots of tiny files would be really inefficient from throughput and api call perspective in blob storage.

With the acks, you have up to 100ms waiting for the buffer to fill, + s3 put request + your metadata request/response. For high throughput that must have very high latency putting back pressure on partitions?

cbsmith9• 3 years ago

This is continuing the trend of cloud pricing driving system designs more than the underlying hardware. AWS overcharges for inter-AZ traffic between EC2 instances, but undercharges for inter-AZ traffic between EC2 instances and S3.

It makes perfect sense to design this way, and as your blog post mentions, people have made similar realizations for columnar databases, and map-reduce frameworks.

cbsmith9• 3 years ago

The way Kafka works, it naturally buffers & coalesces messages even before they get to the brokers, so yes, of course the messages are being coalesced.

There is no problem with "just" running a Kafka cluster in each AZ and only replicating data between AZs until it's time to pull it all together. It's just that when presented with a distributed system and AZs, engineers (and in fairness the business requirements) are more than likely to go with a multi-AZ solution. Same goes for regions. So the vast majority of Kafka clusters are multi-AZ but probably shouldn't be, and Kafka gets the bill for that, even though it shouldn't.

The Kafka protocol doesn't really preserve order-of-operation within a Kafka partition. It preserves the order of operations within a producer-partition pair (and even then, only if you configure it a certain way). The standard implementation does this by preserving the order-of-broker-receipt-of-messages from producers, but from an external system's vantage point, it really only means that (if configured the right way) messages with any given key, from any given producer, will be preserved in the order they are received.

cduzz2• 3 years ago

The architect mentioned a 100ms (or 10 times per second) buffer / flush rate; presumably there's some windowing settings so I can flush at 10,000 records or after 1/10th of a second, and can chose 10 records or 1/100th of a second if I don't mind the bill.

Kafka's nice because there are a lot of knobs to adjust how you prioritize availability or durability or latency. Kafka's tedious because there are a lot of knobs... I was curious as to the nature of the knobs on this product.

I probably should have also asked "and by S3 do you mean S3 or any S3 like object store?" probably that's answered elsewhere.

slantedview• 3 years ago

This article seems to be spreading some FUD, in particular this comment:

Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs.

You're no more or less forced to put Kafka replicas in different AZs than you would be with an alternative.

richieartoul36• 3 years ago

(WarpStream Founder) That's fair, but we think people shouldn't have to choose between multiple AZs and costs. All cloud providers recommend that critical workloads running in cloud environments should run in more than one AZ.

crimsonpowder• 3 years ago

Of course, as soon as an AZ goes down that had all of your Kafka nodes in it, you get eviscerated on HN for running such a setup.

EdwardDiego12• 3 years ago

3 AZ Kafka is the safest, but I've run several 2.5 clusters just fine. (cluster stretched across 2 AZs with a tiebreaker ZK in a 3rd AZ to maintain quorum in the event that you lose an AZ)

rcarmo3• 3 years ago

Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive.

I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant, AZ-redundant or globally redundant storage, most of which are available to run managed disks upon).

This is why properly designed Hadoop/Kafka cloud managed services employ storage adapters to leverage the provider's baked in redundancy. And why some cloud providers have Kafka-compatible event brokers.

The rest of what WarpStream does is just icing on the cake (although I'm curious as to the internals and how they avoid inter-AZ charges).

(full disclosure: I work at Microsoft, but built Hadoop/Spark/Kafka clusters before joining nearly a decade ago.)

hinkley• 3 years ago

In the good ol' days, we used to implement redundant systems such that there was tie breaker process that used only a fraction of the resources of the true processes. Some Raft implementations allow for nodes that are voters but cannot be quorum leaders (example: the branch office where all traffic is funneled through an asymmetric VPN tunnel should never be elected leader, but knows which candidates it can and cannot see)

So the table stakes for running a cluster was not 3x as much hardware but closer to 2.2x, which is a huge deal for solutions in the small and developer sandboxes. It also matters when 3 shards don't quite cover your load but 5 is too many. Or 6 vs 7.

The problem is that with geographic replication, this doesn't fix either of the problems articulated as part of the thesis of this article:

1. Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs.

2. Operational overhead – running your own Kafka cluster literally requires a dedicated team and sophisticated custom tooling.

Still, we need this functionality back for cloud, particularly as the pendulum swings back to self hosting, which it always has in the past.

nithril3• 3 years ago

Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system?

Azure disk replications are for the durability of the data, not the avalability of the data from a kafka perspective.

derefr5• 3 years ago

Many systems might just be using Kafka to drive async batch transaction processing (think: sending emails, charging credit cards), and therefore don't care at all about availability.

rcarmo3• 3 years ago

Managed disks reside on redundant and high availability storage.

cbsmith9• 3 years ago

Or, you know, use ephemeral data storage with your brokers like the good folks intended...

chaotic-good3• 3 years ago

Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system?

That's easy. EBS and similar solutions comes with the price. They're very expensive. Especially, when you need a lot of IOPs. You may be saving on cross-AZ traffic but you will pay ridiculous amount of money on storage. If you have replication you can use attached storage which is way cheaper.

rcarmo3• 3 years ago

Azure Managed Disks are inherently replicated. But there are other storage solutions (like redundant WebHDFS) you can take advantage of. At the time, the customers I was talking to wanted to have identical deployments in the cloud (to what they had on-premises, down to the number and size of disks).

ant6n• 3 years ago

I keep seeing these HN titles about technologies named for other concepts, thinking it's talking about those other contexts, without any indication it's about those obscure technologies. Then I'm disappointed its just about some random, badly named technology.

So, in honor of actual Zombie Kafka, here a short news blurb about Kafka international airport being Named most Alienating Airport: https://www.youtube.com/watch?v=gEyFH-a-XoQ

tekla• 3 years ago

Seriously. I ordered Windows and all I got was a piece of cardboard with some random letters and numbers on it, and not a frame with glass.

MapleWalnut• 3 years ago

It'd be great if there was an rss feed for your blog

davewritescode• 3 years ago

Is the plan to eventually open source this and provide paid support?

waynesonfire• 3 years ago

company competing with kafka says kafka is dead, is that accurate?

insanitybit4• 3 years ago

tiew9Vii2• 3 years ago

Kafka the Apache open source project is kind of dead/legacy for Confluent.

Confluent re-wrote the backend of Kafka for their SASS product and use the public Kafka wire protocol for public integration https://www.confluent.io/blog/cloud-native-data-streaming-ka... .

I don’t think the Apache Kafka distribution is going away, Confluent will still have paying customers with support contracts wanting features and the protocol will likely still need to evolve but the open source version will likely be a burden on Confluent with their own private implementation so it’ll be interesting to see how things look in a few years.

https://www.confluent.io/blog/cloud-native-data-streaming-ka...

whalesalad2• 3 years ago

Running Kafka is an exercise in sadomasochism. This on the other hand sounds pretty neat.

whatscooking• 3 years ago

What? It’s pretty easy to set up and get going

insanitybit4• 3 years ago

Getting a distributed set up plugged into your k8s/whatever framework can be a pain.

whalesalad2• 3 years ago

The last time I was running a cluster it was 2014-2015 and it was not too bad to get up and running, but it felt so heavy handed for what it was. The zookeeper requirement was frustrating. The fact that consumers needed to maintain offset state was frustrating (although... understandable). For the right use case I suppose it is 100% worthwhile but more often than not I have seen it used in places where other tools would be more appropriate.

8organicbits• 3 years ago

I've heard good things about Red Panda[1], but I haven't tried it.

[1] https://redpanda.com/

dmytton• 3 years ago

The blog post mentions "P99 of ~1s of producer-to-consumer latency". What about just producer latency i.e. message successfully received into the queue ready to be picked up? S3 writes seem be in the low 100s of ms so I assume that's part of the quoted end-to-end latency.

richieartoul36• 3 years ago

(WarpStream co-founder)

Yeah our P99 for producer latency is ~400ms right now.

otterley• 3 years ago

Sounds like you're currently optimized for throughput, not latency. (I work with customers for whom 10ms of latency for a 128kB write is too high.) What are your plans to improve latency?

nfw2• 3 years ago

The title of the article should be "Kafka is dead. Long live Warpstream." The long live part refers to the successor.

lxe• 3 years ago

You're right! I've only heard it being used as a contradictory phrase though. https://en.wikipedia.org/wiki/The_king_is_dead,_long_live_th...!

jkaplowitz• 3 years ago

As per your link, it's only "seemingly" contradictory - that is, it emphasizes that as soon as the (previous) king dies, the (new) king is automatically immediately in office to be wished a long life, without needing to wait for any coronation ceremony. This is the rule in many monarchies past and present, though not all. So, imagine that the (old) king dies at the end of the first clause and therefore the (new) king takes office at the beginning of the second.

ryanworl13• 3 years ago

(WarpStream CTO here) This is a bit subtle I will admit, but we view the Kafka protocol as a successor here because it will outlive Kafka the implementation.

pshirshov3• 3 years ago

Ok, intsead of providing commit log guarantees on their own they rely on S3.

I'm not sure if S3 can actually provide durable commit log guarantees.

There is timestamp-based last write wins policy for concurrent writes, so I'm not sure if this thingy can actually replace Kafka in all the usecases providing same guarantees.

richieartoul36• 3 years ago

(WarpStream co-founder) We use S3 for data replication, availability, and durability, but not for the log ordering guarantees. For that, we have our own custom metadata store that "imposes" ordering guarantees on top of the data that is written to S3.

We don't rely on any kind of timestamp-based last write wins policy for concurrent writes. In fact the agents will never write a file to S3 with the same name more than once or overwrite an existing file.

pshirshov3• 3 years ago

So, in fact it's not really stateless, you have your locking/ordering layer and you implement some consensus for that layer?

endisneigh• 3 years ago

This looks great but reading the accidental SRE gave me two questions:

We’ve had bare metal for a very long time now and it seems that managing your own bare metal hasn’t become much easier. If it were very easy, we’d see more of these sort of things be managed by the end user. That being said how are you managing this service? A cloud provider or bare metal?

Both you and Ryan have much experience with foundationdb which generally is managed yourself. Speaking of which, did you go with that for your metadata store again? Why or why not?

richieartoul36• 3 years ago

(WarpStream co-founder)

WarpStream's current offering is a hybrid BYOC approach. The customer runs the agents in their cloud account, and we manage the metadata store for them remotely. This keeps all the customer data in their cloud account and their S3 bucket where we can't see it or touch it. It does mean that the customer has to run the WarpStream agents themselves, but they're just stateless containers that are pretty easy to manage.

We considered using FoundationDB for our metadata store, but ended up not in the end. In order to make our free tier cost effective we really need to make our metadata store as efficient as possible for this specific use-case which required something a bit more custom. That said FoundationDB is a fantastic piece of technology. Best distributed database I've ever used, and I've used many :)

bothra902• 3 years ago

Exciting work! Some questions:

1. Any plans on open-sourcing this?

2. Why not have a tiered architecture that can provide lower latency? p99 of 1s can be too high for some use-cases.

3. Related to 2, how does WarpStream compare to tiered storage in Pulsar?

[Edit 1] Added (3)

smrtinsert• 3 years ago

Yeah really kind of surprised the free tier isn't the self hosted kind. That will keep me looking.

richieartoul36• 3 years ago

(WarpStream Cofounder)

FWIW we’re considering a version where you can host the metadata yourself for enterprise users. For the free tier though we didn’t think it made sense since for a workload that could fit into our free tier, it didn’t seem like anyone would want to be responsible for the metadata layer themselves. Would love your feedback on that.

rad_gruchalski3• 3 years ago

I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 (https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-c...) but this approach is so much better.

Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible.

Very cool, have to find some time to take this for a test drive.

richieartoul36• 3 years ago

(WarpStream co-founder)

Thanks for the kind words! I'll just add that tiered storage is not quite the same thing because it means you still have to manage local disks and replication carefully, however briefly. Please reach out with questions any time!

rad_gruchalski3• 3 years ago

Indeed. The way I was thinking about it was to implement a different file system for it.

Sphax• 3 years ago

Just fyi, tiered storage is coming at some point in Kafka: https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...

rad_gruchalski3• 3 years ago

It’s been like that for a while.

bsaul• 3 years ago

Question : does this architecture have any difference regarding ordering or durability of messages, compared to kafka ?

richieartoul36• 3 years ago

(WarpStream co-founder)

Nope, we provide the same durability and ordering messages as Kafka, albeit with higher latency. We never acknowledge produce requests until they've been durably persisted in object storage (P99 ~400ms), so its the same guarantee as running a 3AZ Kafka cluster with fsync enabled.

EamonnMR2• 3 years ago

Have you considered making a plain HTTPS frontend like Confluent's REST proxy? Or even just designating a bucket as 'incoming messages' and letting people use the S3 API to write messages?

Kafka is great but if you're targeting Amazon customers, you're competing with SQS too.

ryanworl13• 3 years ago

[WarpStream co-founder here]

Please sign up for our mailing list! We'll have some interesting things to announce related to this soon.

insanitybit4• 3 years ago

Pretty cool. I'm definitely going to check this out.

88913527• 3 years ago

It's refreshing to see cloud computing costs matter. Hopefully it'll cut down of needless cloud complexity and the proliferation of indirection in software abstractions of the last decade.

richieartoul36• 3 years ago

(WarpStream co-founder)

Cheers! We really want to drive the incremental cost (both in terms of $ and management overhead) per GiB down as low as possible so people can start using these systems for more use-cases.

skybrian• 3 years ago

As a hobbyist I probably don’t need this but I’m curious. I haven’t used any of the S3-compatible data stores. Any ideas which would be easiest for messing around with?

Also, what databases are there built on S3? I think there is sqlite replication?

rickette• 3 years ago

Will this only exist as a managed service (SaaS) or will there be an open source (or open core) version?

slotrans• 3 years ago

In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it.

(emphasis added)

Surely that's false?

Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it.

Linkedin built Kafka for massive-scale problems that 99% of us don't have. Though technologists have a well-earned reputation for using tech they don't need, my perception is that most of us are succeeding in avoiding the use of Kafka.

richieartoul36• 3 years ago

(WarpStream co-founder)

That's fair, that statement is probably significantly colored by my own personal network / work experience.

ram_rar• 3 years ago

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

dopidopHN• 3 years ago

RabbitMQ, ActiveMQ, MQtt

All of those are fine is you only need pub/sub

richdougherty• 3 years ago

They may still use some sort of event/message system. Kafka is lower level than other sorts message queue systems and requires more work to get correct (dedupe, ordering, retry logic), but has great performance. It's often easier to choose a different messaging system though.

whartung• 3 years ago

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!)

And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea.

There were two basic issues.

The first is that because of the nature of message queues, they're either empty or full. The second is that for many of the ways that queue are used, the unit putting the request on the queue in the first place may well be waiting for the response related to the request. So you've just turned an every day synchronous request into a more complicated, out of band, multi-party asynchronous request.

If your queues are not empty, they they are filling up. And queue capacity is a two fold problem. One, is that you simply run out of space. But, more likely, referring to the earlier point about waiting for a response, is that you run out of time. The response does not return fast enough to manage your response window to the unit making the request.

This is a load balancing problem. If the queue is filling you simply don't have the capacity to handle the current traffic. It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue.

The second part is that if you're throwing items onto a message queue, and you "don't care" about them, that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? If the request is Important, you certainly don't want to trust it to a message queue, a device not really designed for the storage of messages. Messages in a queue are kind of trapped in no mans land, where the easiest way to get to a message is to dig through the ones piled in front of it.

They're interesting insights and worth scratching your chin and going "Hmmm" over.

freedomben• 3 years ago

Most of my designs include a message queue, and I would not use Kafka unless there was a strong need.

Right now my preference varies a bit depending on the rest of the tech stack, but for the most part I use Redis or RabbitMQ.

If the stack is already hard dependent on AWS or another cloud, then SQS or whatever is also fine.

I also wouldn't overlook just using your existing DB (like postgres)! At low and even medium scale this can be totally fine, and also comes with lots of benefits like single-source-of-truth, normal relational DB constraints, transaction wrapping, and more. One of the highest scale apps I've worked on uses Postgres for queueing. It's take a number of optimizations over the years as performance starts to fall due to scale, but it's doable.

carlio• 3 years ago

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem.

I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

kfk• 3 years ago

The biggest issue for me is people using kafka for mqtt, mqtt is a pub/sub broker already. The other issue is thinking of Kafka as some kind of “innovative” data ingestion tool, so now instead of 50 extract jobs per day, you got to reconcile millions of events in realtime. I think message brokers make sense, but they are message brokers, nothing else, no?

nithril3• 3 years ago

Same finding.

That's why that sentence in the article "but almost every technology company uses it." should be rephrased to "but almost every technology company do not need it"

snapetom• 3 years ago

and every time it has been a solution in search of a problem.

More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

EdwardDiego12• 3 years ago

It's really good when you're producing a lot of data fast that you don't want to lose, and you want multiple consumers to read.

It's a complex tool that solves a complicated problem. But if you don't actually have that problem, then that's a whole lot of complexity for no gain.

parhamn4• 3 years ago

Kafka is a closer to a persistent WAL than a message queue. If your work doesn't need a WAL, it's almost certainly overkill and you will hate it. If your work needs a WAL then it'll be your favorite tool ever.

benjaminwootton3• 3 years ago

Why? Its quite easy to use Kafka as a messaging queue without even thinking about the write ahead log semantics. It’s there if you need it, but Kafka scales down to being a message broker fairly well in my opinion.

falcor84• 3 years ago

For those like me who aren't used to that abbreviation, it's short for Write-ahead Logging [0].

[0] https://en.wikipedia.org/wiki/Write-ahead_logging

benjaminwootton3• 3 years ago

I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics.

It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route.

I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular open source tool. If they do then it will be about some niche feature or use case.

I actually think this message reflects badly on the vendor here. Criticise or compete with Kafka on its technical merits if you like, but this is just a misrepresentation of their position in the market.

richieartoul36• 3 years ago

(WarpStream co-founder)

I'm not sure what you mean. Message persistence is a fundamental feature of Kafka that almost everyone using it relies on, its not some esoteric feature no one uses. We're each coming from our own network bias here, but in my experience a lot of people are really unhappy with the operational toil associated with running Kafka at scale in production.

deanCommie• 3 years ago

Easy: I hate how complex it is, compared to something much simpler like https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

mdaniel5• 3 years ago

I’m not sure how anyone can hate Kafka?

My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy

The whole ambiguity around whether ZK is really still needed or not <https://kafka.apache.org/documentation/#zk> makes keeping two distributed systems alive and healthy, but don't worry you can't move your production clusters off of it anyway <https://kafka.apache.org/documentation/#kraft_zk_migration>. It's a mess

twelve40• 3 years ago

i work at a place that has a whole devops department maintaining it (so not even my pain), and i still hate how overengineered the overall system ended up, it's more busywork programming for it, debugging it, etc. No reason not to use sqs or rabbit or whatever unless you have a very special use case, or begin to hit X messages/second. Or just like to spend lots of time writing boilerplate and configuring.

diarrhea• 3 years ago

Ironically, I have a fundamental hatred for LinkedIn and its sluggishness. It’s one of the slowest websites I frequent. I have few connections all things considered. Putting my feed together cannot be rocket science (its contents stay quite static for sometimes weeks at a time).

badrabbit• 3 years ago

A lot of companies use a product that uses kafka under the hoods. I was running graylog a few years ago for months before I knew kafka lay under the hoods.

They have a list: https://kafka.apache.org/powered-by

yakkomajuri• 3 years ago

I disagree. I certainly think it's possible people might be looking to fit Kafka into things that simply don't need it (perhaps driven by the system design theory focus in hiring), but for the applications where you have event streaming, Kafka is still the top choice. Analytics, messaging, sensors, etc.

From my side, I agree with the author about the "Accidental SRE" points. But Kafka is a solid technology, so much so that there's no shortage of "Kafka but better" tools out there (e.g. Redpanda).

Also you kind of drift off the point there at the end - even if it wasn't used extensively (a point of contention), that has nothing to do with whether it is polarizing or not? The statement about it being loved or hated is still relevant to those solving the 1% scaling problems you mentioned, even if 99% aren't.

It's like saying that the statement "lamborghinis are polarizing" is false because most of us don't have one? The author explicitly says "in the data space" too, effectively restricting the people he's talking about.

agallego• 3 years ago

that's a littlebit of a stretch. when you say "no shortage" - outside of redpanda what product exists that actually compete in all deployment modes?

it's a misconception that redpanda is simply a better kafka. the way to think about it is that is a new storage engine, from scratch, that speaks the kafka protocol. similar to all of the pgsql companies in a different space, i.e.: big table pgsql support is not a better postgres, fundamentally different tech. you can read the src and design here: https://github.com/redpanda-data/redpanda. or an electric car is not the same as a combustion engine, but only similar in that they are cars that take you from point a to point b.

jshen2• 3 years ago

What do you use instead? Polling APIs? A queue instead of an event stream?

Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).

bob10292• 3 years ago

We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there. Oh - it is also magically included in our DR strategy. No extra infra. We don't have to have a totally separate procedure to recover this thing.

We primarily use it as a backhaul between parts of our infrastructure in order to perform RPC. The approach is for the users of the broker (our services) to poll it at whatever rate is required. This is actually a little bit clever if you think about it - Users that don't really care about liveliness can poll for their messages every minute or so. Users that are in the hot path of a web UI could poll every 50~100ms.

Polling sounds kinda shitty (at least to me) but I argue it's the best default engineering solution until proven otherwise (assuming its not somehow harder than the other magic async event bubbling things). We don't have a lot of services doing this so contention isn't really a problem for us. Even if it did get to that point, I would reach for a read replica before I refactored how all of messaging worked. Most of polling is just a read operation that does nothing, so we can go horizontal on that part pretty easily.

lmm5• 3 years ago

To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option.

What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to built proper master-master HA on MySQL or PostgreSQL or similar tend to be flaky, expensive, and vendor-lockined. BigTable can work if you're all-in on Google Cloud, but that's quite a risk.

As far as I can tell there are mostly companies that use Kafka and companies that have a SPOF PostgreSQL/MySQL database (with some read replicas, and maybe some untested perl scripts that are supposed to be able to promote a replica to master) and stick their fingers in their ears.

otabdeveloper4• 3 years ago

We standardized on Clickhouse for everything. (With its own set of surprising and/or horrifying ops issues.) But at least it is a proper high-load, high-availablity solution, unlike Kafka, Cassandra, et al.

fulafel2• 3 years ago

I think if you dismiss HA setups of SQL dbs as "you won't get around to operating it properly" the same ops culture will also end up getting many less 9's availability than aspired to with Kafka.

(But also of course lots of applications are also fine with the availability that you get from fate-sharing with a single db server)

zbentley6• 3 years ago

As far as I can tell there are mostly companies that use Kafka and companies that have a SPOF PostgreSQL/MySQL database

I haven't seen that at all, across the many companies I've worked at, consulted with, and talked with others about.

Kafka is usually an ancillary system added to companies with a strong culture around one or more pre-existing datastores (from PG/MySQL to Dynamo/Cassandra to Mongo/Elastic). When Kafka's actually needed, it handles things those pre-existing stores can't do efficiently at high volumes.

Are you really seeing companies use Kafka for their main persistence layer? As in, like, KQL or the equivalent for all/most business operations?

Even the CQRS/ES zealots are still consuming from Kafka topics into (usually relational) databases for reads.

raxxorraxor• 3 years ago

I think many that use it might want use something like ActiveMQ instead. I think such middleware can also be interesting for smaller applications. Especially if they manage messages and data streams between two different companies that like to communicate directly but have arcane software components on each side that fit together like fire and water.

cbsmith9• 3 years ago

"Linkedin built Kafka for massive-scale problems that 99% of us don't have."

What a tool is built for is not the same as what it is good for is not the same as what it is used for... and just like people spend an inordinate amount of time worrying about what happens if they get rich, companies spend a lot of time future proofing for scenarios where they are hugely successful. If nothing else is true about the tech industry, it's certainly true that people misjudge tools and misapply them with alarming regularity, to the point where it is at least as likely that the tool being used is a bad fit for the problem as it is a good fit.

esafak• 3 years ago

How do Pulsar and Kafka compare these days?

sgt2• 3 years ago

Can this be used without S3?

richieartoul36• 3 years ago

(WarpStream co-founder)

It requires some form of object storage (S3, GCS, Azure blob storage, etc). You could use minio if you want to host the storage yourself, although I think I need to make a few changes to make that configurable in the agent.

For local development you can use an in-memory or local file storage, but not for "real" usage.

sgt2• 3 years ago

Okay - yes, it would make a lot of sense as a Kafka alternative to those of us who prefer to stay away from the public clouds.

computershit• 3 years ago

Can I use minio?

richieartoul36• 3 years ago

(WarpStream co-founder)

That does not work currently because there is no way to configure the agent with a Minio URL. That's just an oversight on our part though, I'll get that working within a few days.

derefr5• 3 years ago

As someone who's been doing async-fanout physical-replication of Postgres instances for the longest time using pgBackRest's S3 repo support (the primary writes to S3; the replicas read from S3; the primary's uplink doesn't get saturated serving the writes) I've always wondered where the equivalent for message-queue systems was. Glad to see it :)

jolux• 3 years ago

Haven’t seen it mentioned yet so I’ll just say we use Kinesis Data Streams with enhanced fanout at work and it seems very minimal in terms of operational overhead if it fits your needs.

richieartoul36• 3 years ago

(WarpStream co-founder) Yeah if you can fit your workload into it, it's pretty solid from an operational perspective since AWS manages the whole thing (except scaling up shards and stuff like that).

It gets pretty pricey though for large volumes of Data.

nathas• 3 years ago

Is WarpStream considering hiring Aphyr to do a Jepsen test?

guywithabike• 3 years ago

WarpStream relies on a proprietary metadata store hosted within their internal network to operate, so it's pretty unlikely that Jepsen tests could cover that.

If you're ok with the externally hosted metadata stores as well as the high per-request latencies (p99 of 400ms, according to WarpStream), it's highly likely that things like liveness and safety properties are pretty far from your mind. So, I wouldn't bank on them submitting to a Jepsen test. :)

abhinav-t• 3 years ago

I think that WarpStream relying on a proprietary metadata store isn't an issue for Jepsen tests. If I understand correctly, Jepsen tests treat the distributed databases (or logs like Kafka) pretty much as a black box. Jepsen tests introduce partitions and look for missing/unexpected items against the items that were acknowledged as written successfully by the system.

If you look at Kyle's blog post, https://aphyr.com/posts/293-jepsen-kafka, there is no mention of looking into a broker's storage or any storage for that matter.

richieartoul36• 3 years ago

(WarpStream co-founder)

FWIW we subject WarpStream to continuous chaos/fault injection in our integration tests and staging environment to verify correctness and liveness properties. I wouldn't say they're far from our mind, we've just made a big trade off around latency that we think will make sense for a lot of people.

Hixon10• 3 years ago

I wonder, will we have any solution for local development? Or we always will depend on closed control plane?

richieartoul36• 3 years ago

(WarpStream co-founder) We're thinking about making the `warpstream playground` command run an in-memory local version of control plane. Come chat with us in our Slack or Discord if you want to discuss more!

onetimeuse923042• 3 years ago

I think the focus on the $/GB is wrong. The main goal of Kafka is providing certain processing guarantees and connecting consumers with producers of events.

What Kafka is not for is dealing with enormous number of events or enormous amount of data. Not that it is particularly slow at it (actually, quite fast) but I see a lot of people needlessly try to push more data and events than they need and then complain they have problems.

One easy trick to deal with enormous numbers of events and amounts of data is batching. Just batch some number of events into a single event so that a consumer can pick it up in one go. Removes a lot of cost of transferring the event.

Too much data to transfer? Don't be stupid and try to transfer those huge documents through the Kafka topic -- there is no need for this. Just upload them to S3 and pass through a reference. Or even better, take 10k events, extract data, zip it up, push to S3, and THEN pass a single event that describes 10k large events.

ec109685• 3 years ago

You’ve described the tricks WarpSream does in their implementation, buffering data until a large enough chunk can be written to S3.

onetimeuse923042• 3 years ago

I didn't know that. I worked with some pretty large scale projects and solutions like this are my daily bread and butter to the point where it seems obvious to me but I have no idea what is/isn't obvious to general population of developers.

purpleblue• 3 years ago

What's to stop Confluent/Kafka from simply implementing a feature like this themselves?

mdaniel5• 3 years ago

their recent blog post doesn't sound that far off from the OP's architecture: https://www.confluent.io/blog/cloud-native-data-streaming-ka... (discussed: https://news.ycombinator.com/item?id=36870278 )

poorman3• 3 years ago

Oh man, I can't wait until these people get their first S3 bill for the number or GET requests.

ryanworl13• 3 years ago

[WarpStream co-founder here]

My co-founder and I worked at Datadog for over 3 years where we built Husky, an event storage and query system built directly on top of S3 as well. We know what we're doing here, I promise ;)

https://www.datadoghq.com/blog/engineering/introducing-husky...

poorman3• 3 years ago

What happens if I pull the plug on a server that hasn't flushed to S3 in the configured time? Do I loose that event?

madmax108• 3 years ago

"My co-founder and I worked at Datadog"

If there's one company that's known for blowing up budgets as you scale, it's Datadog. Hoping WarpStream doesn't follow that playbook.

Best wishes!

richieartoul36• 3 years ago

(WarpStream co-founder) Check the end of our blog post, we brought receipts :)

liminal• 3 years ago

Can this be installed locally and run on Minio?

richieartoul36• 3 years ago

Will make this possible soon, oversight on our part that the agent doesn't expose a way to let you do that right now

yingjunwu• 3 years ago

I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion:

* Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol.

* The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support:

-- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/;

-- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...);

-- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco);

-- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/);

-- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/).

* WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable.

* BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security.

I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!

mdaniel5• 3 years ago

If you're still within the edit window for your comment, the ");" in them are attached to the URLs, making them problematic to click

-- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/

-- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...

-- Redpanda Data, which cuts your TCO by 6X https://redpanda.com/platform-tco

-- DataStax, which commercializes Apache Pulsar https://pulsar.apache.org/

-- StreamNative, which commercializes Apache Pulsar https://pulsar.apache.org/

hintymad• 3 years ago

So the idea is that S3 offers practically unlimited throughput, so we can certainly use S3 for a scalable queue? I wish the open-source community could have something competing with S3, but then S3 has made it really hard for companies to have a convincing justification to build their own alternative[1].

[1] There are a few alternatives, such as Apache Ozone, Ceph, and Good'ol HDFS, but none seems have an accelerating momentum for S3's workload.

Exuma• 3 years ago

Care to summarize the 'contact us' pricing?

ryanworl13• 3 years ago

We're aiming for per-GB usage-based pricing that is significantly cheaper than the alternatives, but the BYOC model combined with our extremely efficient cloud control plane gives us a lot of flexibility here. That's why we can offer a free tier at all.

We're mostly just not sure yet, so your input would be appreciated.

pfannkuchen• 3 years ago

I feel like people misuse the X is dead, long live X construction, but maybe I’m missing something.

I’ve always taken it to be a reference to the stability and near-programmatic nature of British royal succession. So while “the king is dead” in some places historically would be a disaster with conflict surely following, in Britain it automatically and instantly becomes “long live the king” for the next king and there is no panic.

EdwardDiego12• 3 years ago

Dear richieartoul, your blog post is a tad oversalted.

Kafka doesn't inherently require dedicated teams of experts and millions of dollars until you're running very large clusters.

But fully agree that 3-AZ stretch clusters suck money through those inter-AZ xfer fees. Which is how AWS sells MSK, the inter-AZ xfer is "free"! That is, already priced in...

jjwiseman• 3 years ago

This comment from 4 months ago helped me understand when you might want to use kafka: https://news.ycombinator.com/item?id=35160555

(The comment is by rad_gruchalski, who I see is also active in this thread. Thanks, rad_gruchalski!)

siliconc0w• 3 years ago

It's a neat idea for a SaaS to just host the control-plane/metadata and allow customers to write to their own data store via S3. I can't think of too many other systems work that way - it certainly is operationally simpler but I wonder if it also gets you some compliance wins.

I wonder if we'll eventually standardize on a cloud-agnostic S3-like protocol but for locking and fast transactional writes so we can just have companies building pure software again but using standardized Cloud instead of POSIX APIs.

edude03• 3 years ago

Okay, I hate to be that guy but I'm given two options for pricing

Free but with only 24 Hours of Retention

Contact us.

Pricing is hard for sure, but if I'm being honest neither of these options make me want to try the product.

richieartoul36• 3 years ago

(WarpStream co-founder)

I know that is super frustrating, I’m sorry. We’re just really early and unsure how to begin even discussing public pricing.

We are 100% committed to that free tier. We designed our entire architecture around making that free tier so cost effective that we could offer it for free.

We want to get to a point where we can have transparent pricing on our website, it will just take us some time to figure that out after speaking with initial customers. We would love to hear your feedback though. You won’t get handed off to a sales guy or anything, it’ll just be a direct conversation with me (CEO) and my cofounder Ryan (CTO)

topaz0• 3 years ago

Kafka's been dead almost 100 years.

backendanon• 3 years ago

I didn't check the site. I have nightmares debugging kafka from years ago and it's only huger now. No thank you, give me Apache MQ or RabbitMQ, even IBM's MQ and it's all good and don't try to tell me I need a streaming thingamajig.

rupurt• 3 years ago

This is super interesting @richieartoul. I specced out something similar myself and was going to implement it in Zig https://github.com/fremantle-industries/transit.

FWIW I came to a similar conclusion that a lot of the power in Kafka comes from the API and that eventually much of the complexity of managing the cluster will eventually be abstracted away with multiple implementations. I also felt that if I could implement Kafka persistent over the S3 keyspace then I could start with persistence direct to S3 like you've done with warpstream and then layer on a faster hot disk and in memory tiering mechanism to eventually lower end to end latencies.

I love where you're going with this so hit me up on twitter if you ever want to chat more in-depth https://twitter.com/rupurt.

richieartoul36• 3 years ago

DMd you on twitter

d_t_w• 3 years ago

Hi @richieartoul, does warpstream support the standard Kafka Admin API?

We build an admin console / dev tooling for Kafka (https://kpow.io) that supports Kafka 1.0+ including Redpanda due to their fairly strict adherence to those API.

Warpstream seems like a cool idea, I'd like to see what happens if we plug Kpow on top of it, if that's possible.

richieartoul36• 3 years ago

Our kafka protocol support is still incomplete. We’ve documented our progress here: https://docs.warpstream.com/warpstream/reference/kafka-proto...

That said we’d be stoked to get this working as an additional tool for people. Do you want to shoot me an email or join our slack so we can discuss further? I can probably prioritize whatever protocol features were missing to get it working.

founders@warpstreamlabs.com

kabochya2• 3 years ago

We've built something probably very similar to this product at my previous job. We had double digit TB daily ML traffic that didn't require realtime latency, so we moved those all onto S3 and also saw approx ~90% cost savings. This was built on top of JVM and still used a 6 broker Kafka cluster to keep metadata (vs probably 300 when it was originally all on Kafka). Kafka's compute and storage model doesn't scale too well for the extreme use cases that can tolerate latency, and the Apache Pulsar model sort of worked better (though at that time Pulsar wasn't too stable for us to use in prod). One of the keys to cost efficiency for us was that the size of the data was large enough that we didn't need to wait too long before hitting an economic file size to upload. Trying to imagine how a pipeline with less than 10 MB/s would work with this efficiently.

richieartoul36• 3 years ago

(WarpStream co-founder)

Yeah we’ve run into a number of people who’ve rolled their own solution in this space. The “push pointers to S3 through traditional Kafka” approach is a very practical one.

Was this memq at Pinterest, or something else?

kabochya2• 3 years ago

Yeah it is, sort of glad that it's actually known. I've since left the ingestion side of data infra so not very familiar of the landscape after one year.

Spiwux• 3 years ago

how many partitions do I need? Unclear, but better get it right because you can never change it!

That is.. simply wrong. You can change the number of partitions.

Plus I really don't get the "you need an entire team of engineers to operate Kafka" claim you keep repeating. That is simply not true, speaking from experience. It is certainly expensive to run, but doesn't require a lot of engineering hours in our team.

FridgeSeal• 3 years ago

requires teams to operate it

I mean, managed Kafka has been a thing for a long while now. I’ve used it plenty and haven’t had any issues…

jooz• 3 years ago

The criticism I have through kafka is the "at-least-once" semantics.

How do you manage exactly-once semantics? If kafka performance is based on reading small batches of 50 messages, in case of crash of the consumer, some of them will be processed twice. Depending on your business logic this may be ok, or may be create a new problem that must be solved farther in the process by adding an external data store.

kilotaras• 3 years ago

You can't have exactly-once in distributed system. Crashes happen, network or power goes down and your ack get's lost[0].

The best one can do is accept that and make your processing idempotent.

[0] The most interesting case I've witnessed was power going down because a hot air balloon crashed into power line.

benbuick• 3 years ago

Hey Richie, As you seem to support the Kafka + Kinesis(?) API: We have developed Kadeck (kadeck.com), a control plane and collaboration layer for Kafka & Kinesis teams. I have looked at the documentation and reckon we should be supporting a lot of Warpstream already. Is there any quick way to test this out? Would love to share this with our users!

Whether dead or alive, there's no denying that Kafka, or data streaming in general, is one of the most exciting fields today. Happy to welcome a new member!

richieartoul36• 3 years ago

(WarpStream Cofounder)

https://docs.WarpStream.com is the best document we have right now. This sounds interesting though, can you jump in our slack or shoot me an email at founders@warpstreamlabs.com ? Happy to support you any way we can!

substation13• 3 years ago

Is there any source available for this?

throw4847285• 3 years ago

I was hoping this would be about Franz.

aftermodern• 3 years ago

The object storage backend is interesting and I’m wondering if Kafka will have a native option for that, eventually.

Confluent hasn’t outright advertised it, but one of their engineers mentioned in a q&a that their Cloud offering is using object storage for their own backend now.

relay232• 3 years ago

They always have.. you get it but you don't know it.

dtjohnnymonkey• 3 years ago

I've seen both your FoundationDB talks, curious if you are using FoundationDB under the hood or how much of the metadata store is homegrown?

Also, how did you implement the parsing of the Kafka protocol? From scratch or pieced together with open source Go bits? Having a nice programmable API for building Kafka-compatible servers could be a huge boon to the Kafka community (wink wink).

michaelcampbell• 3 years ago

WarpStream is 5-10x cheaper than Kafka

<sigh> I'm guessing they mean 1/10 to 1/5 the cost? "1x cheaper" would be free.

Kafka is dead, long live Kafka

Related Stories