Final thoughts on Ubiquiti

https://krebsonsecurity.com/2022/08/final-thoughts-on-ubiquiti/

by todsacerdoti • 4 years ago

539 228 4 years ago

Operyl• 4 years ago

Jeez, I was certain at this point he would never retract his articles. I feel like it's too little too late, imo, though.

cronofdoom• 4 years ago

Funny, I was feeling the exact opposite. They could have just taken down the articles and issued no statement. It is hard to publicly admit that you're wrong and it's good to see they took that step.

blitzar4• 4 years ago

They could have just taken down the articles and issued no statement.

There are people that came bottom of their class and barely qualified yesterday to practice law who easily would have been able to make this not an option.

selectodude• 4 years ago

I imagine a public retraction was part of a settlement with Ubiquiti.

riffic• 4 years ago

this reads like he got out-lawyered here.

context for the unaware:

https://arstechnica.com/tech-policy/2022/03/ubiquiti-sues-jo...

duxup3• 4 years ago

Out lawyered, because he was wrong?

bombcar8• 4 years ago

A good journalist knows how to verify sources and check before writing.

A great journalist finds great sources that are unimpeachable.

But a wealthy journalist knows how to write articles from horrible or no sources at all that are not technical defamation.

kspacewalk22• 4 years ago

Out-lawyered by life and facts.

mewse-hn• 4 years ago

Next story will probably be that the suit was settled out-of-court :P

nerdponx• 4 years ago

Do I understand this correctly?

There was a minor data breach at Ubiquiti. An employee named Sharp was using this as an opportunity to extort his employer and exfiltrate data. Sharp was telling Krebs some yarn about the data breach being bigger than reported, which Krebs then repeated on his blog, accusing Ubiquiti of covering up a more significant breach. And Ubiquiti is claiming that Krebs knew the truth all along.

This sounds like a weird and complicated story, so I feel like I'm probably misunderstanding.

InTheArena15• 4 years ago

Sharp did the breach and then extorted Ubiquiti. Ubiquiti got the FBI involved and declined to pay off Sharp. Sharp followed through on his threat and disclosed everything to Kerbs, who wrote an article about it. The FBI and Ubiquiti were on to Sharp, but since Sharp was Kreb's only source, Krebs doubled down on the allegation with a series of articles, and then never retracted it (until now)

mzs• 4 years ago

That's pretty much it, yes:

https://www.justice.gov/usao-sdny/pr/former-employee-technol...

https://www.cyber.nj.gov/public-data-breaches/ubiquiti

mardifoufs• 4 years ago

I wonder what has changed? The original thread discussing the lawsuit was filled with super dismissive comments, arguing that ubiquity lawyers were incompetent and had no actual way to win the case. Some of the commenters were supposedly actual lawyers too, so it's not even just the normal "terrible armchair law advice" we are used to from HN.

encryptluks23• 4 years ago

Everyone loves to play the armchair lawyer. Bias quickly fuels whatever side of a case you're on. However discovery and a few court conferences can quickly put things into perspective. Almost everyone's lawyers start off with some encouraging words but eventually they are telling you to settle for X and it is clear that they wouldn't have got paid if they told you that you had a losing case from the get go.

james_in_the_uk• 4 years ago

Perhaps Krebs has chosen to move on with his life. Defending litigation is often expensive, distracting and stressful, even if you think you have a strong case. The law isn't necessarily always as pugilistic as Hollywood might lead you to believe. Perfectly fine to think you are right, say you were wrong, settle the case and move on. Of course, we have no way to know what Krebs really thinks or what actually went on behind closed doors here. We should take the written statement at face value, exactly as written. No more no less.

JacobThreeThree• 4 years ago

I guess the Krebs naysayers were right?

The wording of this apology makes it pretty clear in my opinion that he's reacting to Ubiquiti lawyers.

acoard4• 4 years ago

An accurate but pretty lacklustre "mea culpa" and retraction. I don't mind people making mistakes, everyone does, but seeing how Krebs has handled this whole episode has not inspired optimism in how he'll handle future mistakes.

He was essentially used as an unwitting party in a cyber blackmail scheme, and he doesn't touch on that at all. There will continue to be nefarious parties trying to misuse his reputation, so long as he remains a popular cyber researcher. I wish he would show consciousness of that rather than simply saying "I was wrong."

blantonl• 4 years ago

This is about of straightforward as a "I screwed up, I own it, I apologize"

Everyone makes mistakes. Some of the good work Krebs has done seems to be completely overshadowed by a mistake here.

Granted, this is probably in response to some legal action either in progress or already settled, but what more do you want from the guy?

kspacewalk22• 4 years ago

This is about of straightforward as a "I screwed up, I own it, I apologize"

"A source provided info. Source is now discredited. I thus no longer trust the info." That's the gist of the apology. But that's neither here nor there, it does not show understanding of the fact that his reputation was deliberately used for criminal purposes.

cthalupa2• 4 years ago

but what more do you want from the guy?

By the time the December story was published, it seems that Krebs knew full well that his source was the person implicated in the crime to begin with. I would like to understand why he thought it was responsible to press forward while obfuscating this fact, and how he will handle similar situations moving forward. His thought process there will help inform me as to whether or not I can personally take him seriously on things of this nature in the future.

As it stands, I don't know if he learned anything from this, or if he still thinks that people that very well might have perpetrated the crime he's reporting on are reputable sources that he should post information from without question or disclaimer and the only reason this is posted is because he settled in court.

acoard4• 4 years ago

Everyone makes mistakes.

Fully agree, which is why I said the same thing. :)

Granted, this is probably in response to some legal action either in progress or already settled, but what more do you want from the guy?

As I said in my post, a stated awareness that he was used in a cyber blackmail scheme, and at least some nominal promise to try and be aware of that in the future. The difference here is between "I made an honest mistake" and "I was taken advantage of and used unwittingly in a scheme." I'm not interested in him self-flagellating and begging for forgiveness, as my concern is totally forward-looking. I believe this type of problem will come again, of people trying to unwittingly use his reputation to push certain agendas. If he isn't aware of that dimension of the problem then it's likely it will re-occur.

That being said, your point about this post made in a legal context is totally fair and had slipped my mind. I can imagine any apology/statement/etc getting neutered by lawyers for perfectly rational reasons.

stickfigure• 4 years ago

what more do you want from the guy?

He's a guy who writes about hacks. He got "hacked". At the very least I am curious to know more of the story.

ineptech• 4 years ago

what more do you want from the guy?

"This has taught me that my platform can be weaponized by any bad actor who can fool or manipulate me. One column from me could get a CISO fired or move a Fortune 500 company's stock price. That's a heavy responsibility that I wasn't really accounting for, but now that I understand it, I've put some thought in to it and I have made some changes that I hope will harden me and my platform against this kind of social engineering attack."

fnordpiglet• 4 years ago

If I were ubiquiti management or shareholder I would want a pound of flesh, and I expect their lawyers will be pursuing that.

ajross• 4 years ago

The timing doesn't support that take, though. Nikolas Sharp (the sole source for these stories) was arrested almost a year ago. Krebs knew then that his source was tainted, and he did nothing. Instead he waited until he was months into litigation with Ubiquity (which he's almost sure to lose) to try to backpedal.

That's just a straight up violation of journalistic ethics. I think it's very reasonable to demand that our reporters in the security community be clear about their sourcing and prompt about corrections.

A "what more do you want from the guy" implies that we shouldn't hold his past actions to account. And... we should. We absolutely should.

danso• 4 years ago

I would've like to see some explanation for how Krebs fell for this ruse, such as why this single-sourced claim was convincing enough to him to do a series of articles that apparently did serious material harm to Ubiquiti. And at least a few specifics of the key information that Krebs now believes to be faked. Just because his source has been indicted for alleged false info to the press doesn't mean that everything this source gave Krebs is automatically fake. In other words, what claims in the indictment, relating to which evidence the source gave to Krebs, leads Krebs to believe that that evidence is completely unreliable -- and how much, if any, doubt/scrutiny did Krebs give that evidence before this indictment?

It doesn't have to be written in the tone of CYA excuses. The angle is: this is how I got fooled, and these are the lessons I've learned going forward.

As Krebs writes in his mea culpa: "I always endeavor to ensure that my articles are properly sourced and factual." Okay, so why didn't that happen here? Is it one-time bespoke situation, i.e. a perfect storm of mistakes? Or was it because of standard practices that he now sees as insufficient for these kinds of stories going forward?

Hamuko• 4 years ago

I'm not really sure if he's owning it. This post has not made it to his Twitter feed, unlike most of the other recent stories. They're probably not automated, so I wouldn't expect it to be there immediately, but I kinda feel like he wants it to just quietly go away if he doesn't mention it there as well.

https://twitter.com/briankrebs

initplus• 4 years ago

Brian is a journalist more than he is a security researcher, anything he publishes as a journalist should be held to held to a higher standard than an random person just speaking their mind. He had ample opportunity to get out ahead and issue a retraction of the story when it was known to be false, well before the Ubiquiti lawsuit.

duxup3• 4 years ago

For a site that generally is there to give you the inside scoop on what is really going on / happened, interesting / disappointing that the choice is to not do so here.

To me "sorry I was wrong" isn't enough.

Jenk2• 4 years ago

That's a non-answer. What more do you want? What would be "enough"?

ulrashida• 4 years ago

I think you're right, but it took me a few minutes to put my finger on what was missing.

There should have been a "going forward I will..." segment.

criddell• 4 years ago

What should he do going forward?

stronglikedan• 4 years ago

This is why he should have never apologized in the first place, but rather just admit being wrong an move on. Apologies are never enough for some people, and often even weaponized.

Jenk2• 4 years ago

Yep. Apologies are blood to lynch mobs.

karaterobot• 4 years ago

As a third party unaffected by the events in any direct way, I don't feel it's appropriate to give an opinion on whether the apology is satisfactory or not. If Ubiquiti has one, I suppose that's for them to express, or not, as they choose.

lolc• 4 years ago

As a reporter, Krebs failed to promptly disclose or retract when the source turned out to be the leak. That means my understanding of events was left incomplete for longer than it should have been.

So he wronged Ubiquity, which I don't particularily care for. He also wronged his readers, which I am party to. This late retraction is underwhelming and doesn't give me trust. As he seems to only have retracted after being forced to by Ubiquity, now what do I make of his stories where his targets don't have a legal team in his jurisdiction?

bedhead2• 4 years ago

I’m still bitter about this. The story absolutely reeked from the beginning and Krebs did nothing but unnecessarily sensationalize it like a tabloid journalist. I got downvoted a million times over when I pointed this out at the time, why I don’t know, it was obvious to anyone who wasn’t foaming at the mouth to pounce on Ubiquiti. In decades past this was a career-ending error…I wish it still was, I’ll never take a single word Krebs says seriously ever again.

oaiey4• 4 years ago

One upvote returned ;)

BonoboIO• 4 years ago

Sometimes HN doesn’t want to see the obvious, because he is their hero

neilv• 4 years ago

I'd guess he's been advised not to say too much, and the specific way to say what he did.

Besides the ongoing criminal case for which he may be a witness, I'd guess there may be potential liability wrt the company, and I'd guess that he's being careful not to create new potential liability wrt the indicted person (see several different nuances in his "My sole source" sentence).

And this sounds like a lawyer-approved away to convey that he recognizes the importance, without saying any specific possible mistake of his that could be fodder, nor prejudging the case:

I always endeavor to ensure that my articles are properly sourced and factual.

dotBen2• 4 years ago

Yes, this is exactly right - his post is clearly the product of legal negotiations with Ubiquity and probably cleared by both them and his own counsel. He's well advised not to say more than he needs to, even if people in this community would like him to fall on his sword more, that's just not how this stuff works.

skullone• 4 years ago

I think he had to limit what he said, because he potentially has some liability on what he had reported on in the past. Crazy situation

pdntspa• 4 years ago

Given that the target he "hurt" is a massive company that can absorb losses, I think this retraction is quite enough.

bombcar8• 4 years ago

He also "hurt" his readers and those who had trust in him, damaging his own reputation.

VogonPoetry• 4 years ago

Unfortunately I don't think this is the first time he has been socially manipulated in this way. Mr Krebs does seem to have a habit of only getting the details from one side of things and only writing things from that side of the story. Perhaps due to the nature of some of his investigations.

Everyone has weaknesses to being socially manipulated. One way to mitigate this is to open a dialog with "the other side" to check and seek out inconsistencies. Perhaps not revealing everything in your exposé story or leaving the veracity of it somewhat ambiguous until things develop further. This could weaken the impact of your initial story. Dialog is probably not easy when the other party is undoubtably criminal and you can get blocked from reaching the right people. In this case, the accusations were against a corporation. They can be good or bad, but ultimately legal processes will reveal things.

I do think Mr Krebs has upped his game in recent years and enjoy reading his stories, but I read them like fiction rather than actual verified facts.

jpgvm• 4 years ago

I read this as a post probably vetted by his legal team and probably not issued earlier because of the ongoing legal action (and then probably subsequent negotiations with Ubiquiti).

He absolutely fucked up here but he probably can't say so and likely wasn't able to retract sooner less he open himself up to legal culpability for his part in the blackmail scheme (unwitting or not).

Unfortunately this is just how the world works. I hope he has learnt his lesson and will be more through in his vetting of his sources and how his reputation can be misappropriated by malicious actors to do very serious harm.

doctor_eval• 4 years ago

It does read, to me, much like it’s something that was workshopped between Krebs, his lawyers and Ubiquiti’s.

This apology should be about Ubiquiti, not Krebs, and maybe the terseness is just a reflection of that.

Frankly I was on the fence about buying new Ubiquiti gear because of what I’d heard, and seeing this lets me reevaluate my position - which I think is the point.

CaliforniaKarl3• 4 years ago

I don’t see this post when I go to https://krebsonsecurity.com/, at least on iOS Safari. Also, on the home page, when I scroll down to the list of all posts, I don’t see this one.

Edit: It’s there now! Thanks to u/Pharaoh2 for the heads-up.

Pharaoh2• 4 years ago

I see it there

CaliforniaKarl3• 4 years ago

Ah, indeed! I now also see it. I guess parts of the site are cached and take time to refresh?

Regardless, thanks for the correction!

alexk307• 4 years ago

Wow, I wonder how much they were suing him for. Probably tons of money in damages from his report

ryneandal• 4 years ago

Ubiquiti is asking for $425,000 in damages.

- https://arstechnica.com/tech-policy/2022/03/ubiquiti-sues-jo...

borski• 4 years ago

From [1], it would appear to be only $425k.

[1] https://arstechnica.com/tech-policy/2022/03/ubiquiti-sues-jo...

InTheArena15• 4 years ago

Mad props to Brian on this. It's way overdue - and frankly, the Ubiquiti lawsuit was poor PR management - but it's good to put things right given some very poor journalistic choices. Journalists admitting when they are wrong is a key step in rebuilding trust in our institutions - not only news but many aspects of civil society here.

Ubiquiti - as a fan of your products - please drop the lawsuit now. I get that this did a ton of damage to the company, but I don't think anyone wins by dragging this out. The product lineup has improved dramatically over the last year, and it would be good to focus there.

jnwatson• 4 years ago

I’m not aware of any new information in the last 6 months about the matter.

I guess better late than never.

Semaphor• 4 years ago

Mad props to him for finally posting a retraction and half-assed apology after Ubiquity forced him to with the lawsuit they never should have done? What?

vel0city2• 4 years ago

Is it really props when you're being sued nearly half a million in damages to continue hosting the articles?

If he wanted to take down the articles because he felt he was wrong, he had months to do so.

He gets zero props from me for only taking down the articles after being sued.

bombcar8• 4 years ago

Once sued his lawyer probably told him to shut the hell up and not touch anything.

The time to do the right thing was before the lawsuit was filed.

jonpurdy• 4 years ago

There was a lot of discussion from ex-Ubnt employees in a January 2021 thread※ related to outsourcing and incompetent management. From what I've read, they still show ads for their products in newer Unifi Controller web interfaces and don't have a way of disabling tracking.

But now that the Krebs retraction has occurred, my brain doesn't know how bad/incompetent Ubiquiti is these days.

Is there an updated-for-2022 source of info on Ubiquiti's problems? ie. what complaints are still valid, and which ones are not valid due to the cyber blackmail incident?

I was a big supporter from 2015-2019 and I still run their AC Lite AP + EdgeRouterX, but haven't updated them beyond 2019 firmware.

※ - https://news.ycombinator.com/item?id=25735032

InTheArena15• 4 years ago

It's solidified incredibly over the last year. It no longer requires centralized login, no longer shows ads (or you can opt out of them. I don't see it one way or another).

More importantly, the network infrastructure has gotten much much better. I haven't had any stability issues other then testing our new early adopter firmware, and the first versions of policy based load balancing have landed.

mattgreenrocks• 4 years ago

Recent updates to the base UDM have been noticeably better than before. I don’t know what they’re doing differently but I hope they continue along this trajectory.

flyinghamster• 4 years ago

It no longer requires centralized login

Thanks for the update. I have a couple of UAP-AC-Lites and an EdgeRouter PoE, but the recent "cloudiness" began to set my teeth on edge, and I've been loath to upgrade my controller.

LilBytes2• 4 years ago

Also see [1] from HN user `_pplp` who was an ex-employee of Ubiqitui, and my own comments when this all first came out.

https://news.ycombinator.com/item?id=29455730

Arcanum-XIII• 4 years ago

I was a bit disappointed with my Dream Machine Pro at the start, but now it's getting there - they now even provide a simple VPN solution that work on top of Wireguard.

What's still missing is more advance routing exposed in the interface (like dnat) and it would be perfect. I have another one deployed in a shop and it's working fine, even with the harsh hardware condition it's put in (lot of dirt/dust from auto mechanics, even if it's protected in a rack high in the room)

salmon• 4 years ago

Do you have a link to the UDM Wireguard docs? I want to upgrade to one, but not until there is official support for wg tunnels

user3939382• 4 years ago

Ubiquiti is so worried about suing Krebs meanwhile their brand reputation has turned to mud due to the quality of their products, both from my own experience and the general consensus I've heard online. If this incident had never occurred I still would have stopped recommending and using their equipment.

xxpor2• 4 years ago

The problem from my end though is, who really competes with them? No one else offers the same level of control at the same (or even close) price point.

beams_of_light• 4 years ago

The old adage "you get what you pay for" applies here. Yeah, it's cheaper than buying Meraki, Aruba, Fortinet, etc, but the IDS/IPS on their Dream Machine is awful, logging is awful, reliability of anything but wireless gear is awful, Protect storage equipment is awful...

msh• 4 years ago

Mikrotik is in the same price range but not as polished.

kevinmgranger• 4 years ago

I've heard good things about MikroTik?

corndoge• 4 years ago

MikroTik

MartinCron• 4 years ago

I'm using the TP-Link Omada access points and router along with the software controller and it's pretty great.

OJFord2• 4 years ago

Nobody competes with them as 'Apple for networking', but MikroTik is if anything a bit cheaper and better on the actual specs etc. - just without the snazzy UI and easy GUI (highly-G) config.

There's probably a lot of people who'd love Ubiquiti gear ('gadget nerds', Linus Tech Tips viewers, gamers, etc.) to whom I wouldn't recommend MikroTik, but to anyone who's.. idk, heard of iptables, I would.

All the gamer-marketed WAP/routers with a million antennae are somewhat competitors in the former category too I suppose.

beermonster• 4 years ago

No one. Sadly!

Covzire• 4 years ago

Personally I think Ubiquiti fared better than Krebs did in the reputation department. They were both victims in their way but Krebs should have retracted several months ago.

InTheArena15• 4 years ago

Give it a try again. I have a ton of equipment, and it's just working well for me now. I think the consensus among ubiquiti users has shifted considerably over the last year.

oaiey4• 4 years ago

True. The new UI finally got better ... and that was what everyone was complaining about ;). And let us be honest, the UI matters with Unifi.

treesknees• 4 years ago

Is that so true? I've seen that as they've become more popular, some of the rougher edges and complaints come out. Personally I've had no reliability/quality problems with my Ubiquiti equipment. But I do know several people who have moved away from their gear due to the breach and fallout from these Krebs articles.

NoPicklez• 4 years ago

I've used Ubiquiti in my home for roughly 7 years now, with 2 UAP-PROs, 8 Port PoE Switch and the USG.

Apart from the USG getting a little old and struggling to cope with the latest features added (A positive in itself). I really can't think of many other solutions that tie everything together so well. Yes I've had the odd problem, or I've needed to change the config of something. But apart from that, it works fantastically and solved WiFi woes I had many years ago.

They release regular software updates to all devices, including the controller. My only issue is I think the design of the landing page on the controller has gone downhill.

UberFly• 4 years ago

"their brand reputation has turned to mud"

Meh. You're paying attention only one small side of the story. Most by far are pretty happy with the product for the price.

CaliforniaKarl3• 4 years ago

Sitting on the bus, I’ve already changed my mind about the decision to take down the articles, instead of posting a retraction notice. At first I thought Mr. Krebs was being scummy by pulling the posts.

The Ars Technica article linked by u/riffic mentions that there was an earlier, denied takedown request. So, now I think the posts were likely taken down as part of a settlement.

We’ll probably never know—I expect an NDA to be part of the terms—but I wonder if, from Ubiquiti’s side, it might have been better to leave the posts up, but with a retraction notice.

obblekk• 4 years ago

Before people jump on this with super negativity... mistakes happen.

What is Krebs' false positive rate? I think low enough that a simple, clear explanation of why it happened is sufficient.

There's no weasel words or evasion here - he owns up to the error, apologizes to affected parties, and retracts all original posts.

It's true that his reporting probably caused stress for Ubiquity. I'm curious what people think is a fair system to compensate for that, without wiping out independent, generally high accuracy reporters like Krebs

curiousgal• 4 years ago

He has a history of doxxing people who have nothing to do with his pieces so yeah excuse my negativity.

GordonS• 4 years ago

IIRC, he never even apologised for it - just straight up said nothing, like he was pretending it never happened. I haven't followed Krebs' articles since then, he totally lost my trust.

PragmaticPulp2• 4 years ago

What is Krebs' false positive rate?

What's more important is how those false positives are handled.

In this case, it feels like it was swept under the rug and he avoided addressing for as long as possible. If he had simply addressed the problem head-on as the news came out and the FBI information became public, it would have been a different story.

The way he rushed to report accusations from an anonymous source (who was actually the perpetrator) felt asymmetric relative to the minimal reporting on the extortion scandal and ensuing FBI investigation. IMO, the story about someone extorting their employer and then abusing security reporters as leverage was more interesting than the original story. Yet Krebs did very little reporting on the latter, likely because he knew he was central to making it all happen in the first place.

dewey• 4 years ago

What's more important is how those false positives are handled.

Is it really though? If there's a company that has to defend / apologize often (Facebook/Meta maybe) I'd be way more critical of their apology than if one guy who didn't have a case like that before apologying a bit too late for some people or not in the way they wish he would. There's also a lot of information we don't know yet, we don't know what happend behind the scenes and when he was provided with the final verdict and facts.

rovr1382• 4 years ago

If he had simply addressed the problem head-on as the news came out and the FBI information became public, it would have been a different story.

Isn’t that what caused the issue in the first place? He talked before all the info was out and he could verify all of it.

wlesieutre• 4 years ago

I wonder if the delay in addressing it was at the advice of his lawyer so that they could negotiate with the "oops" blog post as part of a settlement

washadjeffmad• 4 years ago

Well, we don't know. How many times has he been either the willing accomplice or unwitting patsy in stock manipulation or corporate sabotage? Does he even know?

de6u99er2• 4 years ago

Yeah, but publishing information as quickly as possible to surf on the first big clicks-wave can cost people their jobs. Because it can result in someone deciding to go with an other company.

A friend who is looking for a easier to manage network for his wife's doctors office let me know that there's reports about security issues after I recommended to him to evaluate if Ubiquity could be a good option. Not sure what exactly he was referring to. Nevertheless, I sent him now the link to this article.

thesausageking2• 4 years ago

Ubiquiti lost $4B in market cap based on this one, poorly sourced post Krebs wrote. He then waited 9 months after he knew it was lies to correct and only does it in the most muted, begrudging way possible. This is completely unethical behavior for a writer.

ganoushoreilly• 4 years ago

I don't think it's fair to attribute their losses directly to Krebs. While this instance is in their favor, Ubiquiti have been doing plenty on their own to alienate their client base. Half backed software updates, pushing products in new verticals without delivering on existing prodcuts. It's clear that there are still issues within Ubiquiti that aren't washed away by this "breach". They're attempting to be enterprise and barely delivering in the Prosumer market.

phoboslab• 4 years ago

What is Krebs' false positive rate?

You only ever hear about it when he gets high profile cases wrong.

When my project was targeted by him, he ended up going down some conspiracy rabbit hole and doxed all the wrong people. This forced me to issue a correction - mission accomplished, I guess.

During his "investigation" he accidentally sent an email that was meant for his business partner to some of my friends. It offered a glimpse into his sensationalist mindset. I don't have much respect for that guy.

modriano• 4 years ago

I had been looking to improve my home network and was about to buy a Unifi Dream Machine Pro-SE and some access points when I read Krebs initial article on Ubiquiti being compromised (coupled with HN commenters really dogpiling on Ubiquiti). Since then, I must have spent 40 hours trying to find a comparable alternative before my discomfort with the opacity of my prior setup drove me to surrender and buy what I had planned out a year ago.

I still think highly of Krebs, but I'm pretty irritated that he wasted so much of my time by reporting something so poorly sourced.

BeefWellington• 4 years ago

What is Krebs' false positive rate? I think low enough that a simple, clear explanation of why it happened is sufficient.

Pretty damned high. From doxxing completely incorrect people over Twitter beef to over-trusting sources like this, he's had a number of controversies over the years and has demonstrated his unreliability as a source of information.

There's no weasel words or evasion here - he owns up to the error, apologizes to affected parties, and retracts all original posts.

Actually he carefully does not own up to the error:

Last year, I posted a series of articles about a purported “breach” at Ubiquiti. My sole source for that reporting was the person who has since been indicted by federal prosecutors for his alleged wrongdoing – which includes providing false information to the press.

By including "which includes providing false information to the press" he's effectively saying it wasn't his fault. Did other press report on it? Did other press ignore the follow-up for nearly a year after it became apparent the source of this information was the person committing the crimes? It would seem to me that owning up to the error wouldn't look so devoid of any responsibility.

He then writes:

As a result of the new information that has been provided to me, I no longer have faith in the veracity of my source or the information he provided to me. I always endeavor to ensure that my articles are properly sourced and factual.
This time, I missed the mark and, as a result, I would like to extend my sincerest apologies to Ubiquiti, and I have decided to remove those articles from my website.

"This time, I [failed to ensure my articles are properly sourced and factual], and as a result, I would like to extend my sincerest apologies to Ubiquiti, and have decided to remove those articles from my website."

The paraphrasing I've done is a more direct way of taking responsibility than Krebs has done. Note in his version the complete lack of ownership, passive voice, and key -- the separation of the problem from where the failure occurred by a paragraph. That's all designed to essentially shirk responsibility here.

Given Krebs' other tendency to get things really wrong, I think it's time people actually viewed him as an unreliable reporter. Especially given he seemingly ignored/dug his heels in about being taken for a ride on this and that then begs the question: How many other instances like this are there that we just haven't heard of or seen?

physhster• 4 years ago

I trust Krebs so I tossed every piece of my Ubiquiti gear I owned as a result. Ended up with a lesser solution since there isn't a good alternative on the market that would do for me everything Ubiquiti did.

NelsonMinar• 4 years ago

It's great that he retracted his story but the way he did it isn't so great. In particular he's removed his older incorrect stories and replaced them with a redirect to the retraction. Thankfully the Wayback Machine has archives

https://web.archive.org/web/20220223015405/https://krebsonse...

https://web.archive.org/web/20220711220855/https://krebsonse...

filmgirlcw3• 4 years ago

Speaking from experience with these things (although in my case, the articles we were forced to remove were absolutely and completely 100% accurate -- but the company that acquired us wanted to settle all outstanding lawsuits and ended up caving so that the transaction could close), this might have been terms of the settlement or whatever it was he came to with Ubiquiti.

In our case, because our articles were in fact, factual, we were able to re-iterate and even quote, as part of the legal filings, aspects of the original reporting as part of a story that was in response to the removals themselves, but the content at those original URLs was replaced with a notice that the articles had been removed because of litigation with our former parent company.

The fact that he didn't (or hasn't) scrubbed the stories themselves from the Internet Archive is a good sign (I think we had to remove our stories from the Internet Archive, though I do know that individuals did make archives other ways).

I'm pretty opposed to suing journalists for the act of doing journalism and even though I'm a big fan of Ubiquiti products, I still don't love this sort of tactic. That said, it does seem clear that these stories were not correct and at the very least, flawed because of the single-source who was not a reliable narrator (and admitted to lying to the press), so in an ideal world, these stories would have been retracted anyway.

ifsothen• 4 years ago

Just because you hang a press pass around your neck doesn't mean you should be allowed to recklessly spread highly damaging lies about people with impunity.

cptskippy• 4 years ago

What would have been a better way to handle it? From personal experience, I've overlooked header/footer retractions on material before and referenced things only to have the retraction point out to me later.

Complete removal of the article isn't ideal, and it's less error prone.

I appreciate that the old articles aren't 404'd, they redirect to the retraction so any other sites linking continue to work.

hot_gril6• 4 years ago

Eh, this is what web archives are for. Krebs doesn't want to show wrong info, and he may even be legally obligated not to.

wlonkly• 4 years ago

Both are also available on archive.today, which I believe is less publisher-friendly for removal requests, should that situation arise.

"All but confirms": https://archive.ph/QycQv

"Charged with extortion": https://archive.ph/V4dUu

PragmaticPulp2• 4 years ago

As a result of the new information that has been provided to me, I no longer have faith in the veracity of my source or the information he provided to me. I always endeavor to ensure that my articles are properly sourced and factual.

This is a strange statement given how the details of the FBI investigation have been public for a very long time.

Krebs was fast to report on the initial accusations, but seems to have waited as long as possible to write about the revelations that his source was actually the perpetrator.

This time, I missed the mark and, as a result, I would like to extend my sincerest apologies to Ubiquiti, and I have decided to remove those articles from my website.

Given that Krebs is a reporter who has historically built a reputation on exposing things and bringing information to light, the brevity and vagueness of this article feels much more like a compromise to settle a lawsuit than typical reporting.

bombcar8• 4 years ago

This is a failure across many journalists; the inability to view what they're involved with objectively. The amount of scrutiny applied to a source is inversely proportional with how much I want to believe the source.

IncRnd• 4 years ago

This is a failure across many journalists

Only one journalist was involved here. I think you meant "failing of many journalists" not "failure across many".

The amount of scrutiny applied to a source is inversely proportional with how much I want to believe the source.

Why do you disbelieve a source that has been greatly scrutinized? Did you mean "directly proportional"?

tcgv• 4 years ago

The amount of scrutiny applied to a source is inversely proportional with how much I want to believe the source

Aka "confirmation bias"

oaiey4• 4 years ago

Maybe part of an informal settlement ;)

blitzar4• 4 years ago

The lawsuit was close to being settled ... my guess is this is part of the formal settlement.

photochemsyn• 4 years ago

This seems to be the basics of the case:

Initial report: https://web.archive.org/web/20211202143043/https://krebsonse...

Indictment of source: https://web.archive.org/web/20211202161703/https://krebsonse...

In cases like this it's probably better to leave the article up but plaster a big red 'retracted' banner across it, with a link to a complete explanation as to why it was retracted.

As far as defamation, isn't the legal bar on that pretty high in the USA? Maybe there's a negligence issue, i.e. relying on a single source, not doing enough background, etc. that overrides the normal 'good faith' reporting norms?

filmgirlcw3• 4 years ago

As I said in another comment, I feel certain (based on my own direct experience working for a publication that faced numerous lawsuits over what in those cases were factual articles) that this was a condition of a legal settlement.

And the thing is, you settle in this case because even though the defamation bar is really high, if your sourcing was wrong (and you maybe didn't do the best job of vetting that sourcing) and the more complicated aspect is that your source was later indicted in relation to a crime directly connected to the information they shared as the basis of that article, this seems like a pretty straightforward "settle it and move on" scenario, rather than trying to fight it in the courts. Barring the largesse of a large news organization (who also might choose to settle, as the Washington Post did with that kid in DC, even though the New York Times and others were years later found to not have defamed him), this is probably not the sort of thing you want to spend the potentially hundreds of thousands of dollars fighting. Because at the end of the day, the reporting was still flawed.

blitzar4• 4 years ago

Date Filed: August 25th, 2022 "Defendants Brian Krebs and Krebs on Security, LLC respectfully request that the Court extend the deadline for Defendants to respond to the Complaint by an additional thirty days in light of extraordinary circumstances that have delayed the finalization of the parties’ settlement"

https://www.courtlistener.com/docket/63197557/21/ubiquiti-in...

They have been finalizing the settlement for some time. I would guess it is now settled.

jaclaz• 4 years ago

In cases like this it's probably better to leave the article up but plaster a big red 'retracted' banner across it, with a link to a complete explanation as to why it was retracted.

Yes, that would have been IMHO much more appropriate, though there is the Wayback Machine (thanks for providing the link), allowing everyone to see with their own eyes what the matter was.

Fnoord• 4 years ago

There used to be a rule in journalism: one source is no source.

BLO716• 4 years ago

A bit of extreme ownership in the same vein as Jocko Willink is inspirational. It's not a reward or ego contest, when you have to open up and be humble about leadership and admission as such - so, critics will be on both sides of the judgement and the reporting.

I myself believe in being humble and honest to a fault, so I'm more sympathetic in this case.

Either way, strive to be better and hey .. humanity is a b*tch sometimes.

nibbleshifter4• 4 years ago

and I have decided to remove those articles from my website.

Updating them with a link to this for context would have been the better move.

nottorp4• 4 years ago

Ok no security breach. Can I set up a new ubiquity device without registering with them at all now?

Been told that you can delete the cloud registration after the set up, but that’s still unacceptable.

InTheArena15• 4 years ago

yes.

You do not need any cloud login at this point.

nottorp4• 4 years ago

They saw the light :)

I have an older device (first in 5 Ghz I think) and I was beginning to think I should upgrade it to <whatever the latest Wi-Fi standard is>.

balentio3• 4 years ago

You guys are thinking about this in a very cloudy kind of way. Assuming that Ubiquiti was being blackmailed, they have a security problem in who they hire (Who held user data for ransom). Assuming they were not being blackmailed, but had a security hole in their software, Ubiquiti has a security problem.

Krebs reporting comes from a potential conflict of interest in that the person who might have been trying to blackmail was also the source. Defamation is not really the issue then because the source was pointing at a security problem which they happened to also be the cause of. The entity that hired this person was...Ubiquiti! Hence, it is not really defamation AS SUCH. Rather, if anything, it was true but maybe blown out of proportion to get a larger sum of money from Ubiquiti. We don't know how much info the person got their hands on, because Ubiquiti would be to blame for that, wouldn't they?

So, ultimately I think taking down the articles is a mistake in the sense that they reported on a problem either way with Ubiquiti and security. Take off the ad revenue from those articles, and issue a modified retraction on the conflicted interest the source held as a correction. Use it as a cautionary tale on "Sensationalism" and "not always knowing what the hell someone is doing when they report a leak" and move on.

InTheArena15• 4 years ago

Kreb's article specifically alleged malfeasance on Ubiquiti's part - that they were deliberately covering up a huge data breach.

This turned out to be untrue on three levels: 1) There was no cover-up. Ubiquiti disclosed the attack, and was working with the FBI, working to identify what had happened, and in fact where already onto Sharp as a insider attack. 2) There was no large scale data breach. 3) The claim that there was a huge cover up was part of a extortion scheme, that Krebs was (unwittingly) assisting in.

Yes, this is a standard insider attack - and Ubiquti's security needed to be significantly better - but it doesn't change the fact that Brian Krebs reported false information - including information that he should have been in a position to know was untrue at the very least in the second article, if not the first.

Ironically enough, the person at Ubiquiti that introduced the wider GITHUB access to production secrets and new policies that allowed Nick Sharp to get production access was - according to former Ubiquiti employees - Nick Sharp.

Who watches the watchers?

balentio3• 4 years ago

> 2) There was no large scale data breach

Says who? The FBI? Says Ubiquiti? I bet BOTH of those places have a reason to say that, and it is green and smells of dead presidents.

gonehome2• 4 years ago

The HN comments here at the time weren't great either: https://news.ycombinator.com/item?id=30850983

Despite it being clear that Krebs was wrong on this issue for some time, it showed the extent of his influence and the attacker's success in leveraging it to manipulate the public (including HN users).

Hopefully his retraction at least helps with that.

itsoktocry• 4 years ago

The HN comments here at the time weren't great either

Amazing, people accusing you of "astroturfing" (an HN favourite, doesn't even make sense in this context).

These are the comment threads that cross my mind when people claim this forum is atypically "smart".

badrabbit• 4 years ago

Someone correct me, but, isn't a journalist supposed to have independent corroborating source/evidence no matter how solid one sole source is? Is that basically where he missed the "mark"?

Werewolf2552• 4 years ago

Yeah, at this point I'm taking Krebs off of any alert or recommendation lists. Appreciate the Mea Culpa, but it's not like he's been making stellar decisions before this problem.

It also doesn't read as an apology, but an acknowledgment that he was given bad info from a source.

bikezen• 4 years ago

Reads like a lawyer wrote this for him, and is still way too late. It was pretty clear early on that his source was a bad actor.

Even if he actually wrote this post himself, it feels like its a result of the ubnt defamation suit against him.

legitster• 4 years ago

"A lie gets halfway around the world before truth puts on its boots."

Another good reminder to take whistleblower claims with a grain of salt. Even someone as professional as Krebs still wants to get the scoop.

I still don't understand why it took Krebs so long or why he insisted on trusting his insider so much without any corroborating evidence.

wnevets• 4 years ago

On a related note the number of negative Ubiquti comments on HN appears to have fallen since this person was outted.

incomingpain• 4 years ago

Ubiquiti got krebbed: https://www.urbandictionary.com/define.php?term=krebbed

de6u99er2• 4 years ago

Did he just delete the previous articles?

IMO he should have linked to them from this post, and updated them with a big fat impossible to miss disclaimer on the top of the article because some other sites might still link to them and use wuotes which are not accurate any more.

duped• 4 years ago

This is why "sole sources" and publications based on them can't be trusted

ocdtrekkie• 4 years ago

If there's one thing I really hope people take away from this entire story is not to use security researchers' statements in constant appeals to authority. I hear so many questionable-to-bad takes on cyber security that basically amount to Bruce Schneier, Brian Krebs, or Troy Hunt said so, so you're absolutely wrong if you don't obey them.

It's really important to remember security researchers and experts convey what they feel is the most accurate or best advice or information they have at the time, and it may very well turn out to be completely wrong or misguided later. The fact that these individuals are popular does not mean they are an authority on anything.

Melatonic• 4 years ago

I agree - and while I certainly trust the people you listed quite a bit it is important to not elevate anyone to cult status or revered leader type stuff. I think we can trust that they have more authority than most but that does not make them the authority.

johncalvinyoung• 4 years ago

Of that list, I'll listen to Schneier or Hunt way before Krebs.

And that was before this story.

hartAtWork• 4 years ago

I think this was absolutely warranted. Ubiquiti's stance as a reliable and secure networking company was damaged in my mind. Krebs absolutely did damage to their reputation.

hot_gril6• 4 years ago

Reminds me of the Bloomberg SuperMicro article with a single anonymous source alleging that several big companies were compromised, which they deny. Funniest part is how Bloomberg itself also claims it wasn't compromised:

Bloomberg LP has been a Supermicro customer. According to a Bloomberg LP spokesperson, the company has found no evidence to suggest that it has been affected by the hardware issues raised in the article.

happyopossum• 4 years ago

Biggest difference - Bloomberg still hasn't retracted that garbage story...

hot_gril6• 4 years ago

Yeah, that's why I'm reminded. Is the moral of the story that reputable, medium-sized reporters without huge legal resources are more trustworthy than something like Bloomberg?

jandrusk• 4 years ago

Sure sounds like a response to the law suite: https://twitter.com/QuinnyPig/status/1508965090019577856?t=L...

InTheArena15• 4 years ago

It was, but it’s important to note that Quinnypig was reacting to the assertions made by extortionist claims made by Krebs. I think he’s acting in good faith here - if not somewhat melodramatically..

system22• 4 years ago

Is there another brand to use other than UI for small businesses? Unifi makes it really easy to manage things and very affordable.

chriscjcj• 4 years ago

Journalists will sometimes go to great lengths to get a scoop and to make a name for themselves. This passionate desire "break the story" makes many journalists vulnerable. They become easy marks for bad actors who can gain by manipulating them.

There are many examples of this occurring.

Sometimes the manipulation is just designed to make the journalist(s) look stupid as in KTVU's scoop on the names of the Asiana Airlines pilots responsible for the deadly crash at SFO: https://www.youtube.com/watch?v=L1JYHNX8pdo

Sometimes it's more serious as in the case of Matthew Keys, who went to prison. I suspect he thought he was chasing a story but was too naïve to realize he was being played. https://www.wired.com/2016/04/journalist-matthew-keys-senten...

Dan Rather's desire to take down former president George W. Bush for all intents and purposes ended his own career by not vetting documents provided to him by a source. His producer Mary Mapes was forced to resign as well. https://en.wikipedia.org/wiki/George_W._Bush_military_servic...

I think we can all learn from these people's mistakes. Our own desires for a particular outcome or to have our personal beliefs confirmed can make us vulnerable to people who might have an incentive to manipulate us. For this reason, it's probably wise to employ a healthy level of skepticism when consuming "news" regardless of how trustworthy we believe the source to be.

WaitWaitWha• 4 years ago

[x] I made a mistake

[x] This is how I made the mistake

[x] I am sorry

[x] I am going to do better

[ ? ] These are the details how I am going to do better

fennecfoxy3• 4 years ago

I hate Ubiquiti anyway. After helping with an install in our office a couple of jobs ago, I bought a UDM in 2020 thinking I'd have the same experience as I did back then.

Oh boy was I wrong. Everything is hidden behind layers of nice looking but useless UI, their software is generally pretty unstable and I get the weirdest things happen that I've never had with other routers, stuff like: * Phones will remain connected but say "connected without internet" without any reason why in the logs * Logs constantly spammed with random errors and other crap that UB is clearly being lazy about * 2.4Ghz unbearably slow, even in an area with no other transmitters (I've tested) and seems to be a software problem but I can't quite pin down why * No real support apart from the forums where often the "solution" to problems is to massively downgrade

Sure, they might have "best in class" APs or something but I think I'm gonna go Mikrotik next time I buy. Idc if I get like 70% the speed of UB wifi as long as it's consistent and works properly.

Final thoughts on Ubiquiti

Related Stories