Splunk IP suit against Cribl

https://www.splunk.com/en_us/blog/bulletins/splunk-files-intellectual-property-complaint-against-cribl.html

110 101 4 years ago

Dang! Back in early days of AppDynamics, the founder who started AppDynamics after working at CA got hit by CA lawsuit, which lasted for a while but eventually got settled. Similar allegations. it was highly unpleasant and detrimental to the IPO preps. Some of my colleagues from there went to Cribl and sure hope they aren't going to be impacted, but they likely will will.

debacle• 4 years ago

Splunk is great software.

With no details, hard to read this suit. Would need to know what evidence Splunk has that Clint Sharp stole source code. All the rest seems superfluous.

skipkey• 4 years ago

The lawsuit is linked in the post. I didn’t read the whole thing, but skimming, the smoking gun seems to be that an implementation of Splunk’s S2S protocol was posted to his personal GitHub while he was still an employee. They claim that the header files just had the Splunk copyright notices removed, but just being a re-implementation of the objects wouldn’t surprise me. Depending on the jurisdiction that might matter.

There are also various copyright claims on things in manuals, plus claims that they infringed numerous patents.

All in all, it sounds pretty bad, but lawsuits almost always do. I would wait to read the responses before coming to any conclusions.

ec1096855• 4 years ago

Pretty poor that S2S is a proprietary protocol to begin with.

badrabbit2• 4 years ago

Splunk is the best at what it does with no close competition.

I've been looking into Cribl and it seems their product has surpassed their competition as well but not in search, more in data summarization and log reduction, possibly before you ship it off to a more proper place like Splunk.

Splunk's cost makes it inaccessible to most people or companies. I mean, I work in infosec and I highly caution against Splunk because it is so amazing you will hate anything else but in security you need tons of otherwise rubbish data collected centrally sometimes and it will force you into a corner where you will say you can't afford to store that log you really should be storing. Better a crappy tool that can be used to find the logs you need than a nice tool that can only retain so much.

Cribl is supposed to help people reduce what they put i Splunk so they can keep using Splunk, it would have been nice if they partnered instead.

Graylog is another nice tool I like that is somewhat but only slightly similar to Cribl that was founded by a former Splunker out of frustration.

chillfox7• 4 years ago

Is Splunk fast now?

Last time I used it was almost a decade ago and it was rubbish, queries took 10-40 minutes to complete.

jitl• 4 years ago

We recently transitioned to it at Notion and it’s been very fast, outperforming the previous log vendor substantially while offering better search and UX. If you used the on-prem version, the cloud version is quite a different experience.

flounder311• 4 years ago

It has always been the Cadillac of search, and moreso with unstructured indexing (e.g. key collisions with different data structures. Foo = string vs foo = integer vs foo = array).

Your queries or infrastructure were not optimized. It’s very fast when optimized.

mattnewton3• 4 years ago

40 minutes sounds exceptionally bad, but 5-10 minutes with splunk was totally common when I worked at Apple almost a decade ago, and I could never figure out why because I only ever used it for O(grep on a log file on disk) level operations. I was probably holding it wrong or maybe the infra team had misconfigured it, idk.

greggsy• 4 years ago

No offence, but that sounds like lazy query design, poor architecture, or both.

badrabbit2• 4 years ago

Very very fast, I can do an all time search on terabytes of data in seconds.

But you have to learn to use it, if you don't give it an index and a sourcetype that will slow it down, and like ES leading wildcards slow things down. The fastest searches are simple terms like a word or an IP.

HyperSane• 4 years ago

On modern NVMe storage it is INCREDIBLY fast.

sandy_coyote• 4 years ago

Yes BUT it needs tuning. Splunk is complicated and takes continuous maintenance to optimize speed.

I work as a Splunk integrator and here's what I often see:

1. Customer installs Splunk with a qualified Splunk or third-party architect team. The deployment works well.

2. Customer adds infrastructure to the deployment. Splunk slows down. License costs go up.

3. Customer chooses between outside help or DIY. DIY rarely works.

4. Customer now needs outside help. Now Splunk is very slow and expensive, and now it will cost a lot to tune it.

Splunk, the company, is in a tough spot for several reasons: rotating c-level cast, unpopular changes to license model, bad acquisitions. The product is still best in class but tough to keep optimized.

gorbachev• 4 years ago

Humio is faster and cheaper than Splunk and while I haven't compared the two products feature by feature, does pretty much the same.

As an end user having used both to manage logs on a few dozen distributed applications I would never choose Splunk over Humio.

achillean• 4 years ago

There's also Gravwell (https://www.gravwell.io) that competes head-on with Splunk and doesn't punish you for storing/ indexing more data. I'm on their board and knew the founders before I joined so I'm a bit biased but it's basically what if you wrote Splunk from scratch using modern tech.

devchix2• 4 years ago

Splunk is the best at what it does with no close competition.

I'm with you. Splunk core - the indexing, automatic parsing, HA architecture, is unsurpassed. You can rebuild/duplicate parts of it but it's not going to come close to what Splunk can do, effortlessly, out of the box. I'm frustrated at the crud that Splunk has acquired which doesn't solve their customer's core problems. Splunk isn't well-rep in the network space. In my past I've worked for a huge tech company that was the darling of its day and Splunk business trajectory reminds me of that; we're within the start of the descent.

I read through the complaints in this thread, how it's slow, behemoth, hard to manage, copmlexities grow ... I've never experienced this problem. I've built and managed 3 Splunk clustered installations, in the 10sTB/day, and I will never use anything else. Sadly, that makes me only able to work for people able to afford the license :nervous laugh: So if you're made of money and want black car white glove data service, buy Splunk and hire people like me.

npalli• 4 years ago

You can't start Cribl without some Crib.

teraflop• 4 years ago

On the one hand, taking code from your employer and posting it to GitHub with the copyright notices removed is about as clear-cut a case of copyright infringement as you can get -- if they have evidence. Should be easy to confirm or deny by looking at version control history.

(This seems to be the repository in question, but it's been taken down: https://web.archive.org/web/20210104032001/https://github.co...)

On the other hand, the patent claims referenced in the lawsuit seem to me like great examples of software patents that ought to be struck down for being uselessly over-broad. For example, I would love to hear an argument as to how the "'433 Patent" wouldn't be infringed by running Wireshark in a Kubernetes pod. That meets every single one of the claimed elements that Splunk is claiming Cribl is infringing.

EdwardDiego• 4 years ago

I'm not sure I understand the business impact of a protocol?

Presumably anyone with Wireshark could reverse it, so does it impart a significant advantage? Or is it just about control?

adinisom• 4 years ago

A more complete archive: https://archive.ph/HKAF2

flounder311• 4 years ago

Splunk, as a company, is a shell of its former self. All they care about is pimping themselves out to maximize profits to an extreme that only Dilbert can relate to, even at the expense of destroying a long term professional relationship over trivial matters. They are more than happy to kill a deal over a 5% disagreement rather than understand the needs of a Fortune 500 customer and negotiate.

They are mad because Cribl is good at transforming data before it ingested by Splunk, so as to reduce the amount of data that is indexed. Period.

Splunk ONLY RECENTLY released “Ingest Actions” to filter data post-ingest (to avoid indexing) for their SaaS product — something that has always been a mainstay of their on-premise “Enterprise” product. Their ONLY suggestion to filter data that we didn’t care to index in early 2021? Cribl. There’s literally no other reason for us to use Cribl.

I’ve been paying for Splunk since 2008 and can’t wait to get away from them. Their sales teams have decayed into unethical slimebags and I am trying everything in my power to not renew our contracts with them. This just sealed the deal.

Source: I cut checks to Splunk for $x,xxx,xxx yearly

ec1096855• 4 years ago

What are you planning to move to?

flounder311• 4 years ago

Sounds crazy, but Datadog. I’ve been hammering their product teams for years with specific use cases for the sole purpose of replacing Splunk. They recently migrated search technologies and are rapidly closing the gap. Plus, their exclusion features are instant and fantastic, and their C-suite replies to me when I escalate.

Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.

jollofricepeas• 4 years ago

Exactly.

This is the question. If you’re looking for APM well you’ve got great options but for those using Splunk in the security space (SIEM & SOAR) you’re screwed.

There’s no better SIEM alternative that deals with logs at scale.

Splunk recently screwed a friends Fortune 50 company. They didn't pay a bill on time (renewal negotiations) and Splunk without even contacting them just left all the logs from one of their instances on the floor. They lost everything for literally an entire country.

I mean EVERYTHING.

kordlessagain• 4 years ago

Ex-splunker here. I just started working at FeatureBase and would say, if your data is in Kafka, FeatureBase might be something to consider. It’s a crazy fast binary index built on Roaring bitmaps.

windexh8er• 4 years ago

I can add from the other side of the fence. I worked for a startup that was acquired by Splunk. They are everything listed here and worse on the inside.

My first few weeks at Splunk were very odd. They try to indoctrinate new hires with a barrage of "A-players" that continuously talked about how awesome Splunk was. Except... When I started Splunk was getting their ass kicked by cloud-first players that had recently come to market. Splunk's monolithic architecture wasn't well suited to be run as SaaS at the time and Splunk was burning cash and losing money on every customer that they suckered into moving away from their perpetual licenses into subscription hell. I left money on the table when I ran out the door less than 6 months later.

I'm curious what Splunk's long game is with this because they just told every F2000 that their bottom line is being chipped away by Cribl and friends. So if I'm an enterprising procurement department I'd be tossing Cribl or Rudderstack or whatever other data transformation preprocessor on the table alongside my renewal. Expand opportunity? If you put your ear to the tracks you can almost hear all of the account managers digging out missed quota excuses.

Splunk isn't innovative and hasn't been for a long time. Most of the employees saw the writing on the wall and went to Snowflake as soon as the opportunity presented itself. Splunk tried to capitalize on the security market by, basically, double charging customers for ES. Instead of delivering value it seems to be Splunk is just looking for ways to squeeze a few last drops of lemonade.

keyle2• 4 years ago

Sounds like a good move... From their about page:

    Splunkers have received over 1,020 patents to date

that tells me everything I need to know.

dec0dedab0de• 4 years ago

Interesting, I helped manage a splunk install at a fortune 200 about a decade ago. At the time the recommendation was to use syslog-ng to filter incoming logs before indexing. I just heard of cribl 2 weeks ago because the fortune 20 I currently work for is planning on switching to it. I didn't realize it was a massive shift like that, I just thought it was the corporation switching things just because they do that sometimes.

psaux2• 4 years ago

No comment about the company, but want to make clear as a buyer you understand the procurement and legal parts i.e. MFN or MFC.

If they do discount, even 5%, then it ripples across their accounts as a legal matter, esp at your scale. I was a buyer for some big companies, 8 digit, and the procurement office would only do a deal with MFN/MFC clause. They would also audit the supplier from time to time.

flounder311• 4 years ago

I totally understand that ripple effect and am very familiar with Most Favored X when it comes to unit pricing of a tangible good (e.g. xx,xxx physical servers with a particular SKU), but in this case we were talking about a SaaS product where overages were disputed. Nearly every vendor would jump at the chance to discount additional commitments or support at the ‘expense’ of waiving some past overages.

totetsu• 4 years ago

Last used splunk around 2010, and we ran a bunch of scripts to truncate and reformat logs before they got anywhere near our splunk data load license

sn0w_crash• 4 years ago

PMs leave and start competing products. This stuff happens all the time.

Unless splunk has a smoking gun it’s hard to really take their side here.

throwawanginee2• 4 years ago

Splunk is a great tool but expensive. I like splunk's aggregation feature very much. If it is server logs, it can aggregate and tell me how many http 500 errors I have, how many requests resulted in 404 etc. It can tell me top IP addresses where I am getting requests from, etc.

I want to take a CSV file and provide same functionality. Eg. Give user information on how many times each field occurs. For example, if it is a CSV file with cities, countries, continents, I want to aggregate and tell how many cities are in each country and how many countries are in each continent.

Is there an open source version of splunk I can modify? I tried logstash but it is not straight forward to work with. It still needs me to define schema everytime.

Thx!

stevewatson301• 4 years ago

Is there an open source version of splunk I can modify?

https://github.com/grafana/loki might work for you. It’s not a drop in replacement for Splunk, FWIW.

thayne• 4 years ago

Is there any way to do subqueries (or some kind of join) with loki? That is one feature of splunk I haven't seen elsewhere, open source or not.

sofixa• 4 years ago

What you're describing sounds like Loki (Grafana's Prometheus inspired logging tool, which is super fast and cheap/easy, even though it sacrifices some flexibility to get there) Metric Queries: https://grafana.com/docs/loki/latest/logql/metric_queries/

wizwit9992• 4 years ago

We're building Matano (https://github.com/matanolabs/matano), an open source security lake platform. It's a different approach since we normalize logs from JSON, csv, etc, and ingest them into Apache Iceberg tables, but it allows for massive scale and joins, aggregations, etc using SQL.

rmdoss• 4 years ago

Hope Splunk loses. Trying to kill a good player that makes Splunk less expensive.

doorsopen2• 4 years ago

From the lawsuit looks like the most clear cut evidence they have is:

- Founder publishing a private protocol definition to help in building for it

- Sales staff sending account and prospect info to their new cribl email addresses before leaving Splunk

- Engineers leaving Splunk with technical specifications, such as their newer S2S protocol versions

The patent stuff is kind of whatever, but all three of those items would be enough to establish some very clear damages. Cribls an exciting new player but they can't take shortcuts like this, if the allegations are founded.

ZephyrBlu• 4 years ago

Honest question, where is the line here? Obviously we all retain knowledge from previous jobs so what's the line between that and exactly copying a spec?

compsciphd• 4 years ago

honestly, I've never moved from a role in one company to a role in a new company that directly competes with the role I had in my old company.

While I have jumped to competitors, I moved to roles that weren't in any form competition to my former team/role. That makes it easy, even if I would accidentally take things with me, I wouldn't be tempted to look at it, as there would be no point.

So yes, I take all my growth, knowledge and experience, but nothing that is really unique (say trade secrets) to old company would directly apply to my new role, so there has never been any problem. Once one is willing to jump to a competitor in a manner where you trade secret knowledge would benefit your role directly, one is creating a problem.

aaa_aaa• 4 years ago

To me, unless there is a legal document you signed with your employer, there is no line. Even, IMO, IP is not `property` so that it cannot be used against. But that is another discussion.

finnh• 4 years ago

I think the line here is pretty straightforward: the contents of your mind are all yours. Anything beyond that (documents, source code, lists of prospects) is not.

Non-compete clauses will try to limit the usefulness of the "in your mind" knowledge by restricting the domains in which you can work post-departure. It's my understanding that such clauses are generally held to be unenforceable except in an acquisition scenario.

doorsopen2• 4 years ago

I think the line is drawn at actual stealing. In this case, they're not redesigning the protocol from memory or black box testing. Allegedly they took several specs of the protocol from former employees. Even then, apparently the founders were involved in some of the Patent filings from splunk that they are accused of violating. You cant claim IP for a company in the form of a patent and then turn around and re-implement that IP. You clearly believed it was patent-able since you patented it. There's ways of doing this (clean room dev) if you wanted to do that without infringement. (I do feel a lot of the patent claims in the lawsuit are typical generic weak software patents)

Really egregious is taking the sales data. Business analytics around leads, customer satisfaction, pricing, etc are not the same as retaining general knowledge. If you left and remember the point of contact you had at a customer, that's allowed (barring non-solicitation agreements). If you leave and you take a list of customers, data that the business has generated about them, etc, that was never yours and it's not your knowledge. It's clearly the business's and there's usually dozens of people involved in the creation. That's clearly theft, especially since it was never yours to begin with.

pharmakom• 4 years ago

can anyone explain to a developer what splunk does?

tannhaeuser• 4 years ago

Conveniently (and expensively) provide a destination for log data streams and enable realtime ad-hoc querying against that data at scale, with time series features and a Unix shell pipeline-like query syntax. As the comments tell, though, you must tweak it to actually see performance. And, having seen its use in ecommerce with eg logged transaction details such as credit card numbers and other PII, the prospect of easily and loosely logging all the things is limited by data security and privacy concerns, with some practices a recipe for big-time breaches and lawsuits. Aand, per seat licensing is expensive, such that more often than not IME you have a Splunk guy/gal to whom you must address your data reporting needs, questioning any benefits that ad-hoc querying your logs may have.

bak3y3• 4 years ago

It does Enterprise level logging better than any other tool I've ever used.

And it drains your bank account.

PanosJee5• 4 years ago

Splunk now an IP bully?

Go Clint & Ledio!

wdb• 4 years ago

Pretty happy with opentelemetry collector that allows to receive traces/metrics/logs etc in different formats and then cleanup the data and push it to aggregators like Splunk or Datadog. Makes it easy to switch when the tool I am using now gets a bit expensive for tracing

kenm47• 4 years ago

I’m currently working with a company in this space (axiom.co) and this shit scares me because it’s splunk scared. Maybe cribl did this? But the press release reads like a self-Pat on the back.

PanosJee5• 4 years ago

Another funny tidbit:

On March 24, 2017, a few months after his initial copying of Splunk’s source code, Mr. Sharp resigned from Splunk to co-found Cribl with Dritan Bitincka and Ledion Bitincka— both former software architects at Splunk.

Except that they didn't because initially the had created a company called diag.io that was focused on troubleshooting fault configurations.

smallerfish• 4 years ago

Our alerting solution, "OpterVics", was bought by Splunk. Since then it's been a shitshow - the service is running, but it's almost impossible to get a response from support.

They sent us an invoice for renewal in early August. I replied back (5 separate times) asking for the original contract (our ops department is tightening up on vendor management, didn't have it on file already); and we've heard nothing. Our service has continued to work despite not having paid (or signed a renewal), but we're switching to opsgenie.

glonq• 4 years ago

Splunk is/was a damn fine tool, but I had to stop using it 5+ years ago because they priced themselves out of the stratosphere.

_fat_santa• 4 years ago

I would love to use splunk on some of my side projects. Does anyone know of a decent alternative for non-enterprise customers?

alloutblitz1• 4 years ago

OpsVerse allows you to bring up Grafana Loki for logs.

There's also Logz.io or you can use Elastic for an ES backend.

rsdbdr2032• 4 years ago

I posted above, but starting log-store.com so I'm trying to promote it in threads without being too pushy :-)

It's in beta and free. My plan is honestly to have my pricing be free for small amounts of data, and then 50% the price of Splunk for larger data sets. Just show me an invoice, and you'll pay half!

njpatel• 4 years ago

(co-founder here) Try axiom.co - we support splunk-like query syntax, dashboards, monitors, unlimited sources/hosts/etc, and you get 500GB/mo ingest + 30 days retention on the free plan.

blarneyrocks• 4 years ago

After reading the full lawsuit, I think Cribl has a real threat on their hands. They've been playing fast and loose with the rules for a long time. Exports of leads from departing Splunkers, using licenses they're not entitled to use, and yes, using proprietary code that was gathered through less than fully kosher means. While this doesn't look great for Splunk, they wouldn't have filed the suit if they thought they would lose.

Splunk IP suit against Cribl

Related Stories