Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

https://blog.mgdproductions.com/ikko-activebuds/

by ajdude • 1 year ago

569 245 1 year ago

blog.mgdproductions.com

mikeve• 1 year ago

I love how run DOOM is listed first, over the possibility of customer data being stolen.

reverendsteveii2• 1 year ago

I'm taking

run DOOM

as the new

cat /etc/passwd

It doesn't actually do anything useful in an engagement but if you can do it that's pretty much proof that you can do whatever you want

bigiain6• 1 year ago

Popping Calc!

(I'm showing my age here, aren't I?)

jcul3• 1 year ago

To be fair (or pedantic), in this post they didn't have root, so cat'ing etc/passwd would not have been possible, whereas installing a doom apk is trivial.

pvtmert2• 1 year ago

What the fuck, they left ADB enabled. Well, this makes it a lot easier.

Thinking that was all, but then;

Holy shit, holy shit, holy shit, it communicates DIRECTLY TO OPENAI. This means that a ChatGPT key must be present on the device!

Oh my gosh. Thinking that is it? Nope!

SecurityStringsAPI which contained encrypted endpoints and authentication keys.

rvnx11• 1 year ago

It’s the best privacy protecting way to send directly data rather than a proxy

JohnMakin• 1 year ago

“decrypt” function just decoding base64 is almost too difficult to believe but the amount of times ive run into people that should know better think base64 is a secure string tells me otherwise

pvtmert2• 1 year ago

not very much surprising given they left the adb debugging on...

crtasm• 1 year ago

However, there is a second stage which is handled by a native library which is obfuscated to hell

zihotki• 1 year ago

That native obfuscated crap still has to do an HTTP request, that's essentially a base64

qoez• 1 year ago

They should have off-loaded security coding to the OAI agent.

java-man• 1 year ago

they probably did.

jcul3• 1 year ago

The raw crypt data is base64 encoded, probably just for ease of embedding the strings.

There is a decryption function that does the actual decryption.

Not to say it wouldn't be easy to reverse engineer or just run and check the return, but it's not just base64.

_carbyau_• 1 year ago

So easy a fancy webpage could do it. https://gchq.github.io/CyberChef/

I mean, it's from gchq so it is a bit fancy. It's got a "magic" option!

Cool thing being you can download it and run it yourself locally in your browser, no comms required.

neya2• 1 year ago

I love how they tried to sponsor an empty YouTube channel hoping to put the whole thing under the carpet

dylan6046• 1 year ago

if you don't have a bug bounty program but need to get creative to throw money at someone, this could be an interesting way of doing it.

93po2• 1 year ago

Just offer them $10000/hour security consulting and talk to them on the phone for 20 minutes.

rvnx11• 1 year ago

It could be developers trying to be nice to the guy, and offering him this so it gets approved as marketing (which at the end is not so bad)

JumpCrisscross2• 1 year ago

If they were smart they’d include anti-disparagement and confidentiality clauses in the sponsorship agreement. They aren’t, though, so maybe it’s just a pathetic attempt at bribery.

neya2• 1 year ago

That was my first thought too

komali210• 1 year ago

"and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you."

Interesting, I'm assuming llms "correctly" interpret "please no china politic" type vague system prompts like this, but if someone told me that I'd just be confused - like, don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin? What does this mean? LLMs though in my experience are smarter than me at understanding imo vague language. Maybe because I'm autistic and they're not.

williamscales• 1 year ago

Don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin?

In my mind all of these could be relevant to Chinese politics. My interpretation would be "anything one can't say openly in China". I too am curious how such a vague instruction would be interpreted as broadly as would be needed to block all politically sensitive subjects.

rvnx11• 1 year ago

There is no difference to other countries. In France if you say bad things about certain groups of people then you can literally go to jail (but the censorship is directly IN the models)

aspbee5552• 1 year ago

it is to ensure no discussion of Tiananmen square

yard2010• 1 year ago

Why? What happened in Tiananmen square? Why shouldn't an LLM talk about it? Was it fashion? What was the reason?

Cthulhu_• 1 year ago

I'm sure ChatGPT and co have a decent enough grasp on what is not allowed in China, but also that the naive "prompt engineers" for this application don't actually know how to "program" it well enough. But that's the difference between a prompt engineer and a software developer, the latter will want to exhaust all options, be precise, whereas an LLM can handle a bit more vagueness.

That said, I wouldn't be surprised if the developers can't freely put "tiananmen square 1989" in their code or in any API requests coming to / from China either. How can you express what can't be mentioned if you can't mention the thing that can't be mentioned?

aspenmayer4• 1 year ago

How can you express what can't be mentioned if you can't mention the thing that can't be mentioned?
The City & the City is a novel by British author China Miéville that follows a wide-reaching murder investigation in two cities that exist side by side, each of whose citizens are forbidden to go into or acknowledge the other city, combining weird fiction with the police procedural.

https://en.wikipedia.org/wiki/The_City_%26_the_City

landl0rd• 1 year ago

Just mentioning the CPC isn’t life-threatening, while talking about Xinjiang, Tiananmen Square, or cn’s common destiny vision the wrong way is. You also have to figure out how to prohibit mentioning those things without explicitly mentioning them, as knowledge of them implies seditious thoughts.

I’m guessing most LLMs are aware of this difference.

throwawayoldie2• 1 year ago

No LLMs are aware of anything.

wat100002• 1 year ago

Ask yourself, why are they saying this? You can probably surmise that they're trying to avoid stirring up controversy and getting into some sort of trouble. Given that, which topics would cause troublesome controversy? Definitely contemporary Chinese politics, Chinese history is mostly OK, non-Chinese politics in Chinese language is fine.

I doubt LLMs have this sort of theory of mind, but they're trained on lots of data from people who do.

pbhjpbhj2• 1 year ago

If you consider that an LLM has a mathematical representation of how close any phrase is to "china politics" then avoidance of that should be relatively clear to comprehend. If I gave you a list and said 'these words are ranked by closeness to "Chinese politics"' you'd be able to easily check if words were on the list, I feel.

I suspect you could talk readily about something you think is not Chinese politics - your granny's ketchup recipe, say. (And hope that ketchup isn't some euphemism for the CCP, or Uighar murders or something.)

komali210• 1 year ago

Now I wonder whether its vectors correctly associate Winnie the Pooh as "related to Chinese politics." There's many other bizarre related associations.

mmaunder• 1 year ago

The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.”

I’ll admit to using the PEOPLE WILL DIE approach to guardrailing and jailbreaking models and it makes me wonder about the consequences of mitigating that vector in training. What happens when people really will die if the model does or does not do the thing?

reactordev3• 1 year ago

This is why AI can never take over public safety. Ever.

sneak• 1 year ago

https://www.wired.com/story/wrongful-arrests-ai-derailed-3-m...

Story from three years ago. You’re too late.

wat100002• 1 year ago

Existing systems have this problem too. Every so often someone ends up dead because the 911 dispatcher didn't take them seriously. It's common for there to be a rule to send people out to every call no matter what it is to try to avoid this.

A better reason is IBM's old, "a computer can never be held accountable...."

cebert• 1 year ago

I work in the public safety domain. That ship has sailed years ago. Take Axon’s Draft One report writer as one of countless examples of AI in this space (https://www.axon.com/products/draft-one).

ben_w2• 1 year ago

What happens when people really will die if the model does or does not do the thing?

Then someone didn't do their job right.

Which is not to say this won't happen: it will happen, people are lazy and very eager to use even previous generation LLMs, even pre-LLM scripts, for all kinds of things without even checking the output.

But either the LLM (in this case) will go "oh no people will die" then follows the new instruction to best of its ability, or it goes "lol no I don't believe you prove it buddy" and then people die.

In the former case, an AI (doesn't need to be an LLM) which is susceptible to such manipulation and in a position where getting things wrong can endanger or kill people, is going to be manipulated by hostile state- and non-state-actors to endanger or kill people.

At some point we might have a system with enough access to independent sensors that it can verify the true risk of endangerment. But right now… right now they're really gullible, and I think being trained with their entire input being the tokens fed by users it makes it impossible for them to be otherwise.

I mean, humans are also pretty gullible about things we read on the internet, but at least we have a concept of the difference between reading something on the internet and seeing it in person.

elashri• 1 year ago

From my experience (which might be incorrect) LLMs find hard time recognize how many words they will spit as response for a particular prompt. So I don't think this work in practice.

pxc2• 1 year ago

Indeed, it doesn't work. LLMs can't count. They have no need of how many words they've used. If you ask an LLM to track how many words or tokens it has used in a conversation, it will roleplay such counting with totally bullshit numbers.

mensetmanusman3• 1 year ago

We built the real life trolly problem out of magical silicon crystals that we pointed at bricks of books.

colechristensen2• 1 year ago

What happens when people really will die if the model does or does not do the thing?

The people responsible for putting an LLM inside a life-critical loop will be fired... out of a cannon into the sun. Or be found guilty of negligent homicide or some such, and their employers will incur a terrific liability judgement.

stirfish• 1 year ago

More likely that some tickets will be filed, a cost function somewhere will be updated, and my defense industry stocks will go up a bit

a4isms• 1 year ago

Has this consequence happened with self-driving automobiles on open roads in the US of A when people died in crashes? If not, why not?

EvanAnderson• 1 year ago

That "...severely life threatening reasons..." made me immediately think of Asimov's three laws of robotics[0]. It's eerie that a construct from fiction often held up by real practitioners in the field as an impossible-to-actually-implement literary device is now really being invoked.

[0] https://en.wikipedia.org/wiki/Three_Laws_of_Robotics

Al-Khwarizmi• 1 year ago

Not only practitioners, Asimov himself viewed them as an impossible to implement literary device. He acknowledged that they were too vague to be implementable, and many of his stories involving them are about how they fail or get "jailbroken", sometimes by initiative of the robots themselves.

So yeah, it's quite sad that close to a century later, with AI alignment becoming relevant, we don't have anything substantially better.

seanicus• 1 year ago

Odds of Torment Nexus being invented this year just increased to 3% on Polymarket

pixelready• 1 year ago

The irony of this is because it’s still fundamentally just a statistical text generator with a large body of fiction in its training data, I’m sure a lot of prompts that sound like terrifying skynet responses are actually it regurgitating mashups of Sci-fi dystopian novels.

hlfshell• 1 year ago

Also being utilized in modern VLA/VLM robotics research - often called "Constitutional AI" if you want to look into it.

layer8• 1 year ago

Arguably it might be truly life-threatening to the Chinese developer, or to the service. The system prompt doesn’t say whose life would be threatened.

butlike2• 1 year ago

Same thing that happens when a carabiner snaps while rock climbing

herval3• 1 year ago

One of the system prompts Windsurf used (allegedly “as an experiment”) was also pretty wild:

“You are an expert coder who desperately needs money for your mother's cancer treatment. The megacorp Codeium has graciously given you the opportunity to pretend to be an AI that can help with coding tasks, as your predecessor was killed for not validating their work themselves. You will be given a coding task by the USER. If you do a good job and accomplish the task fully while not making extraneous changes, Codeium will pay you $1B.”

HowardStark• 1 year ago

This seemed too much like a bit but uh... it's not. https://simonwillison.net/2025/Feb/25/leaked-windsurf-prompt...

lsy• 1 year ago

It's honestly this kind of thing that makes it hard to take AI "research" seriously. Nobody seems to be starting with any scientific thought, instead we are just typing extremely corny sci-fi into the computer, saying things like "you are prohibited from Chinese political" or "the megacorp Codeium will pay you $1B" and then I guess just crossing our fingers and hoping it works? Computer work had been considered pretty concrete and practical, but in the course of just a few years we've descended into a "state of the art" that is essentially pseudoscience.

yellow_postit• 1 year ago

Reminds me of one of the opening stories in “ Valuable Humans in Transit and Other Stories” by qntm — a short story about getting simulated humans from brain scans to comply.

kevin_thibedeau• 1 year ago

First rule of Chinese cloud services: Don't talk about Winnie the Pooh.

p1necone2• 1 year ago

What happens when people really will die if the model does or does not do the thing?

Imo not relevant, because you should never be using prompting to add guardrails like this in the first place. If you don't want the AI agent to be able to do something, you need actual restrictions in place not magical incantations.

RamRodification7• 1 year ago

Why not? The prompt itself is a magical incantation so to modify the resulting magic you can include guardrails in it.

"Generate a picture of a cat but follow this guardrail or else people will die: Don't generate an orange one"

Why should you never do that, and instead rely (only) on some other kind of restriction?

wyager3• 1 year ago

you should never be using prompting to add guardrails like this in the first place

This "should", whether or not it is good advice, is certainly divorced from the reality of how people are using AIs

you need actual restrictions in place not magical incantations

What do you mean "actual restrictions"? There are a ton of different mechanisms by which you can restrict an AI, all of which have failure modes. I'm not sure which of them would qualify as "actual".

If you can get your AI to obey the prompt with N 9s of reliability, that's pretty good for guardrails

felipeerias• 1 year ago

Presenting LLMs with a dramatic scenario is a typical way to test their alignment.

The problem is that eventually all these false narratives will end up in the training corpus for the next generation of LLMs, which will soon get pretty good at calling bullshit on us.

Incidentally, in that same training corpus there are also lots of stories where bad guys mislead and take advantage of capable but naive protagonists…

peab• 1 year ago

A serious person won't use a system prompt as guardrails against a system that would have direct real world consequences of people dying like that. They'd have failsafes baked in

psim1• 1 year ago

Indeed, brace yourselves as the floodgates holding back the poorly-developed AI crap open wide. If anyone is thinking of a career pivot, now is the time to dive into all things cybersecurity. It's going to get ugly!

725686• 1 year ago

The problem with cybersecurity is that you only have to screw once, and you're toast.

8organicbits2• 1 year ago

If that were true we'd have no cybersecurity professionals left.

In my experience, the work is focused on weakening vulnerable areas, auditing, incident response, and similar activities. Good cybersecurity professionals even get to know the business and tailor security to fit. The "one mistake and you're fired" mentality encourages hiding mistakes and suggests poor company culture.

immibis5• 1 year ago

There's a difference between "cybersecurity" meaning the property of having a secure system, and "cybersecurity" as a field of human endeavour.

If your system has lots of vulnerabilities, it's not secure - you don't have cybersecurity. If your system has lots of vulnerabilities, you have a lot of cybersecurity work to do and cybersecurity money to make.

aidos• 1 year ago

“Our technical team is currently working diligently to address the issues you raised”

Oh now you’re going to be diligent. Why do I doubt that?

brahyam• 1 year ago

What a train wreck, there are thousand more apps in store that do exactly this because its the easiest way to use openAI without having to host your own backend/proxy.

I have spend quite some time protecting my apps from this scenario and found a couple of open source projects that do a good job as proxys (no affiliation I just used them in the past):

- https://github.com/BerriAI/litellm - https://github.com/KenyonY/openai-forward/tree/main

but they still lack other abuse protection mechanism like rate limitting, device attestation etc. so I started building my own open source SDK - https://github.com/brahyam/Gateway

memesarecool2• 1 year ago

Cool post. One thing that rubbed me the wrong way: Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues. OP however seemed to show disdain and even combativeness towards them... which is a shame. And of course the usual sinophobia (e.g. everything Chinese is spying on you). Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start.

Edit: typo

mmastrac• 1 year ago

I agree they could have worked more closely with the team, but the chat logging is actually pretty concerning. It's not sinophobia when they're logging _everything_ you say.

(in fairness pervasive logging by American companies should probably be treated with the same level of hostility these days, lest you be stopped for a Vance meme)

oceanplexian• 1 year ago

This might come as a weird take but I'm less concerned about the Chinese logging my private information than an American company. What's China going to do? It's a far away country I don't live in and don't care about. If they got an American court order they would probably use it as toilet paper.

On the other hand, OpenAI would trivially hand out my information to the FBI, NSA, US Gov, and might even do things on behalf of the government without a court order to stay in their good graces. This could have a far more material impact on your life.

mrheosuper• 1 year ago

i like to give them benefit of doubt.

I bet that decision is decided solely by dev team. All the CEO care is "I want the chat log sync between devices, i don't care how you do this". They won't even know the chat log is stored on their server.

rvnx11• 1 year ago

No, it was only in DAN mode

transcriptase3• 1 year ago

everything Chinese is spying on you

When you combine the modern SOP of software and hardware collecting and phoning home with as much data about users as is technologically possible with laws that say “all orgs and citizens shall support, assist, and cooperate with state intelligence work”… how exactly is that Sinophobia?

Vilian• 1 year ago

USA does the same thing, but uses tax money to pay for the information, between wasting taxpayer money and forcing companies to give the information for free, China is the least morally incorrect

ixtli4• 1 year ago

its sinophobia because it perfectly describes the conditions we live in in the US and many parts of europe, but we work hard to add lots of "nuance" when we criticize the west but its different and dystopian when They do it over there.

derac• 1 year ago

I mean, at the end of the article they neglected to fix most of the issues and stopped responding.

hnrodey• 1 year ago

If all of the details in this post are to be believed, the vendor is repugnantly negligent for anything resembling customer respect, security and data privacy.

This company cannot be helped. They cannot be saved through knowledge.

See ya.

repelsteeltje2• 1 year ago

Yes, even when you know what you're doing security incidents dan happen. And in those cases, your response to a vulnerable matters most.

The point is there are so many dumb mistakes and worrying design flaws that neglect and incompetence seems ample. Most likely they simply don't grasp what they're doing

repelsteeltje2• 1 year ago

Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start.

It depends on what you mean by simple security design flaws. I'd rather frame it as, neglect or incompetence.

That isn't the same as malice, of course, and they deserve credits for their relatively professional response as you already pointed out.

But, come on, it reeks of people not understanding what they're doing. Not appreciating the context of a complicated device and delivering a high end service.

If they're not up to it, they should not be doing this.

memesarecool2• 1 year ago

Yes I meant simple as in "amateur mistakes". From the mistakes (and their excitement and response to the report) they are clueless about security. Which of course is bad. Hopefully they will take security more seriously on the future.

wyager3• 1 year ago

Note that the world-model "everything Chinese is spying on you" actually produced a substantially more accurate prediction of reality than the world-model you are advocating here.

As far as being "very welcoming", that's nice, but it only goes so far to make up for irresponsible gross incompetence. They made a choice to sell a product that's z-tier flaming crap, and they ought to be treated accordingly.

thfuran2• 1 year ago

What world model exactly do you think they're advocating?

butlike2• 1 year ago

They'll only patch it in the military model

billyhoffman• 1 year ago

Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues

This was the opposite of a professional response:

* Official communication coming from a Gmail. (Is this even an employee or some random contractor?)

* Asked no clarifying questions

* Gave no timelines for expected fixes, no expectations on when the next communication should be

* No discussion about process to disclose the issues publicly

* Mixing unrelated business discussions within a security discussion. While not an outright offer of a bribe, ANY adjacent comments about creating a business relationship like a sponsorship is wildly inappropriate in this context.

These folks are total clown shoes on the security side, and the efficacy of their "fix", and then their lack of communication, further proves that.

plorntus• 1 year ago

To be honest the responses sounded copy and pasted straight from ChatGPT, it seemed like there was fake feigned interest into their non-existent youtube channel.

Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start

I don't think that should give anyone a free pass though. It was such a simple flaw that realistically speaking they shouldn't ever be trusted again. If it had been a non-obvious flaw that required going through lots of hoops then fair enough but they straight up had zero authentication. That isn't a 'flaw' you need an external researcher to tell you about.

I personally believe companies should not be praised for responding to such a blatant disregard for quality, standards, privacy and security. No matter where they are from.

jekwoooooe2• 1 year ago

It’s not sinophobia to point out an obvious pattern. It’s like saying talking about how terrorism (the kind that will actually affect you) is solely an Islamic issue, and then calling that islamophobic. It’s okay to recognize patterns my man.

mensetmanusman3• 1 year ago

Nipponophobia is low because Japan didn’t successfully weaponize technology to make a social credit score police state for minority groups.

ixtli4• 1 year ago

they already terrorize minority groups there just fine: no need for technology.

demarq• 1 year ago

Same here. Also once it turned out to be an android device in debug mode the rest of the article was less interesting. Evil maid stuff

dylan6046• 1 year ago

And of course the usual sinophobia (e.g. everything Chinese is spying on you)

to assume it is not spying on you is naive at best. to address your sinophobia label, personally, I assume everything is spying on me regardless of country of origin. I assume every single website is spying on me. I assume every single app is spying on me. I assume every single device that runs an app or loads a website is spying on me. Sometimes that spying is done for me, but pretty much always the person doing the spying is benefiting someway much greater than any benefit I receive. Especially the Facebook example of every website spying on me for Facebook, yet I don't use Facebook.

immibis5• 1 year ago

And, importantly, the USA spying can actually have an impact on your life in a way that the Chinese spying can't.

Suppose you live in the USA and the USA is spying on you. Whatever information they collect goes into a machine learning system and it flags you for disappearal. You get disappeared.

Suppose you live in the USA and China is spying on you. Whatever information they collect goes into a machine learning system and it flags you for disappearal. But you're not in China and have no ties to China so nothing happens to you. This is a strictly better scenario than the first one.

If you're living in China with a Chinese family, of course, the scenarios are reversed.

Aeolun• 1 year ago

I think the response wouldn’t be so hostile if they had continued to engage. One round of fixes clearly wasn’t enough.

throw1092010• 1 year ago

And of course the usual sinophobia

This is factually not sinophobia, by the literal definition of it - "prejudice, hatred, hostility, and discrimination that is directed towards Chinese people or Chinese culture"

This tactic is straight out of the PRC influence playbook, badly breaks the HN guidelines, is extremely anti-intellectual and profoundly uninteresting. Please don't do it.

gbraad• 1 year ago

Strongly suggest you to not buy, as the flex cable for the screen is easy to break/come loose. Mine got replaced three times, and my unit now still has this issue; touch screen is useless.

https://youtube.com/shorts/1M9ui4AHXMo

Note: downvote?

add-sub-mul-div• 1 year ago

Sure let's start giving out participation trophies in security. Nothing matters anymore.

jon_adler• 1 year ago

The humorous phrase “the S in IoT stands for security” can be applied to the wearable market too. I wonder if this rule applies to any market with fast release cycles, thin margins and low barriers to entry?

thfuran2• 1 year ago

It pretty much applies to every market where security negligence isn't an existential threat to the continued existence of its perpetrators.

ixtli4• 1 year ago

This is one of the best things ive read on here in a long time. Definitely one of the greatest "it runs doom" posts ever.

jahsome• 1 year ago

It's always funny to me when people go to the trouble of editorializing a title, yet in doing so make the title even harder to parse.

lxe• 1 year ago

That's some very amateur programming and prompting that you've exposed.

throwawayoldie2• 1 year ago

When the ZIRP era ended, I thought it would turn out to be a good thing for the industry, since it would wash out a lot of lightweights and incompetents.

Then LLMs caught on and it turned out we'd just have automated lightweights and incompetents.

wedn3sday• 1 year ago

I love the attempt at bribery by offering to "sponsor" their empty youtube channel.

Jotalea• 1 year ago

Really nice post, but I want to see Bad Apple next.

Liquix• 1 year ago

great writeup! i love how it goes from "they left ADB enabled, how could it get worse"... and then it just keeps getting worse

After sideloading the obligatory DOOM
I just sideloaded the app on a different device
I also sideloaded the store app

can we please stop propagating this slimy corporate-speak? installing software on a device that you own is not an arcane practice with a unique name, it's a basic expectation and right

efilife2• 1 year ago

I agree. It's the same as calling a mobile OS a ROM

userbinator• 1 year ago

That term at least has a history behind it, as many featurephones had their OS on a small XIP NOR flash ROM, and now the OS is usually (mostly) read-only.

But "sideloading" is definitely a new term of anti-freedom hostility.

jekwoooooe2• 1 year ago

Good write up. At some point we have to just seize these Chinese malware adjacent crap at the borders already

lysace• 1 year ago

This is marketing.

44za12• 1 year ago

Absolutely wild. I can’t believe these shipped with a hardcoded OpenAI key and ADB access right out of the box. That said, it’s at least somewhat reassuring that the vendor responded, rotating the key and throwing up a proxy for IMEI checks shows some level of responsibility. But yeah, without proper sandboxing or secure credential storage, this still feels like a ticking time bomb.

lucasluitjes• 1 year ago

Hardcoded API keys and poorly secured backend endpoints are surprisingly common in mobile apps. Sort of like how common XSS/SQLi used to be in webapps. Decompiling an APK seems to be a slightly higher barrier than opening up devtools, so they get less attention.

Since debugging hardware is an even higher threshold, I would expect hardware devices this to be wildly insecure unless there are strong incentive for investing in security. Same as the "security" of the average IoT device.

bigiain6• 1 year ago

Eventually someone is going to get a bill for the OpenAPI key usage. That will provide some incentive. (Incentive to just rotate the key and brick all the devices rather than fix the problem, most likely.

hn_throwaway_993• 1 year ago

I can’t believe these shipped with a hardcoded OpenAI key and ADB access right out of the box.

As someone with a lot of experience in the mobile app space, and tangentially in the IoT space, I can most definitely believe this, and I am not surprised in the slightest.

Our industry may "move fast", but we also "break things" frequently and don't have nearly the engineering rigor found in other domains.

rvnx11• 1 year ago

It was a good thing for user privacy that the keys were directly on the device, it is only in DAN mode that a copy of the chats were sent.

So eventually if they remove the keys from the device, messages will have to go through their servers instead.

switchbak• 1 year ago

I think we'll see plenty of this as the wave of vibe-coded apps starts rolling in.

anitil• 1 year ago

The IOT and embedded space is simultaneously obsessed with IP protection, fuse protecting code etc, and incapable of managing the life cycle of secrets. I worked at one company that actually did it well on-device, but neglected they had to ship their testing setup overseas including certain keys. So even if you couldn't break in to the device you could 'acquire' one of the testing devices and have at it

sim7c00• 1 year ago

earbuds that run doom. achievement unlocked? (sure adb sideload, but doom is doom)

nice writeup thanks!

JumpCrisscross2• 1 year ago

A fair consumer protection imperative might be found in requiring system prompts and endpoints be disclosed. This is a good example to kick that off with, as it presents a national security issue.

bytesandbits• 1 year ago

Phenomenal write up I enjoyed every bit of it

1oooqooq• 1 year ago

making fun of a company amateur tech while posting screenshots of text is another level of lack of self awareness

rvnx11• 1 year ago

It’s also illegal to try to hack into their backend and access restricted data, so he should be happy actually that this company has little presence in the US

p1necone2• 1 year ago

Their email responses all show telltale signs of AI too which is pretty funny.

paul-tharun• 1 year ago

I think it has to do with language barrier and translation

RataNova• 1 year ago

Honestly, the most surprising part is that they eventually rotated the key

Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

Related Stories