Show HN: Live coaching app for remote SWE interviews, uses Whisper and GPT-4

https://github.com/leetcode-mafia/cheetah

301 114 3 years ago

Posting from a throwaway account to maintain privacy.

This project is a salvo against leetcode-style interviews that require candidates to study useless topics and confidently write code in front of a live audience, in order to get a job where none of that stuff matters.

Cheetah is an AI-powered macOS app designed to assist users during remote software engineering interviews by providing real-time, discreet coaching and integration with CoderPad. It uses Whisper for audio transcription and GPT-4 to generate hints/answers. The UI is intentionally minimal to allow for discreet use during a video call.

It was fun dipping into the world of LLMs, prompt chaining, etc. I didn't find a Swift wrapper for whisper.cpp, so in the repo there's also a barebones Swift framework that wraps whisper.cpp and is designed for real-time transcription on M1/M2.

I'll be around if anyone has questions or comments!

imwillofficial2• 3 years ago

This is badass. Even as a proof of concept it’s hilarious how many HR brains will explode over this

test65542• 3 years ago

We have hired back to back people that we had to let go. When you take 4 hours to respond to simple questions you either have two jobs or you are not disciplined enough to be working from home.

fortissimohn2• 3 years ago

As someone laid off and having trouble finding a single job, that is rather discouraging to hear.

jedberg5• 3 years ago

When I saw this, I had a good belly laugh! As someone who does a lot of interviewing who also hates all the leetcode style interviews, I love this.

I hope it ushers in a new era of non-leetcode style interviews.

But I suspect what it will actually do is usher in an era of in-person only interviews or having to use that same crap spyware that schools use to lock down your computer by essentially rooting it.

moose_man3• 3 years ago

They need to come up with something better than leetcode. Build a project and explain it is a much better way to get a sense of skill. It's the sort of skill that won't be made extinct by ChatGPT. Your creativity and ability to architect and build a project and then explain it should be the standard. Leetcode also penalizes good engineers who have anxiety during interviews.

breckenedge3• 3 years ago

I’ve been doing interviews like this for a few years now. Leetcode sucks.

Bonus points the next time a candidate (correctly) uses ChatGPT or Copilot. Let the machines do what the machines do well (grinding leetcode), good riddance.

LanceH• 3 years ago

Worked one place where we received seven (7!) resumes that were almost identical, with the same project of what they could do.

Not saying leetcode is the answer, but it does solve for something things.

My interviews tend to walk through the framework of a project and they can speak to me in the abstract, or an example of what that part of the project does. Say there is a standard way of connecting to the database like in Rails. They can tell me about database.yml and how it has different entries for each of the databases. Then we have a conversation about checking in passwords to git, env variables, secrets managers, etc... This avoids asking stock questions which might be coached/studied more and aims for what the person doing the work might practically know. It also keeps the discussion in a context that makes understanding the questions (hopefully) easier. My style of interviews are very much non-standardized and there has to be some trust that I have some idea what I'm doing.

Leetcode at least has some standardization around the problem. Of course everyone could have looked them up and studied the exact solution, and these solutions don't correspond very well to daily efforts. But, given everyone knows the game, it does demonstrate some horsepower I guess. Or maybe I like these interviews because I'm good at them.

itake2• 3 years ago

Build a project and explain it is a much better way to get a sense of skill.

I HATE project interviews, because their skills aren't transferable to the next interview. Take home projects also tend to "go-over", because you want to put your best foot forward and you're betting that the other people they interview are also going to exceed time.

qup2• 3 years ago

Couldn't you just run a second PC right behind your monitor that was listening and providing tips?

jedberg5• 3 years ago

Yeah but then you'd have to retype the output instead of just copy/pasting it. :)

runnerup• 3 years ago

or having to use that same crap spyware that schools use to lock down your computer by essentially rooting it.

The most extreme one I’m aware of is “Lockdown Browser” and the student who lives with me has effectively rendered it useless. Students use MST with DisplayPort to mirror a monitor in a way such that the OS cannot see that two monitors exist. (I don’t think there’s a windows API for LDB to see this? Could be wrong)

Anyways, one student faces the other, each looking at a mirrored display, one student out of view of the webcam. Then a microphone with a hardwired switch soldered in which, again, doesn’t alert the OS that a microphone has been disconnected, is switched off.

Then student 2 can freely speak at the student taking the test and announce all the answers. The test taker is able to keep their eyes on the monitor at all times (so eye tracking won’t show anything weird).

If I were black-hat enough to lead a product like lockdownbrowser, I would beat this technique by looking for electrical hum frequency signatures in the audio feed which can pinpoint what time the audio was recorded due to fluctuations in the grid electricity. https://en.m.wikipedia.org/wiki/Electrical_network_frequency...

If there is zero audio waveform, that would be a flag for review. If the cheater attempts to loop pre-recorded audio, it will be noticeable that the ~60hz frequency signature is “wrong” for the time that the test was being taken.

This would then in turn be defeated by the cheating student running a remote microphone to a quiet room which can be switched to.

I’ve put a lot of thought into this and no matter what, the cheater eventually wins given infinite cat-and-mouse iterations. But most would be caught somewhere along the way.

jackblemming• 3 years ago

The solution is to trust the people you hire, treat them with respect and decency. And when they start and don't know what a for-loop is because they obviously gamed the process, you fire them. It's really that simple.

If people complain it's impossible to fire anyone, that's the issue. The solution isn't to implement an insane anti-cheat hiring pipeline that will drive away competent people.

weq2• 3 years ago

Thanks for the advice - this is a cool technique and yeh im ex defence! its funny as this remote-video stuff is hardly ever reviewed by the clients anyway in my experience, that this stage in its maturity they use it basically as a bluff. Dont discount the use of manual reviews though, our software would pickup anomolies if your moving your mouth constantly.

We have organised crime targetting our events, they have done things like infiltrate IT departments and trojen horse PCs. For high-stakes, we advise clients that in person inviligation in combination with LDBs is the safest way with a freshly provisioned SOE.

If you want to make this your job, work remotely and like C#, u can find our careers page with this hint: 3F27483F97C94ECF0C8F11148FBBD048DFFCDECBE5C62FA23076297AE804F6C6 Send us an unsolicated resume and say ur from HN LDB

indrora• 3 years ago

Students use MST with DisplayPort to mirror a monitor in a way such that the OS cannot see that two monitors exist. (I don’t think there’s a windows API for LDB to see this? Could be wrong)

I have a cheap $50 capture card that is untraceable. My $75 HDMI mux has hdcp stripping and EDID pass through. I use these for streaming video games and this is a pretty standard set of kit.

The only way to discover this is hoping that custom control commands are passed through to the monitor. OEMs use these to give you access to the OSD options from the operating system, such as a crosshair at the center of the screen for gamers, on when their game is running but off when it exits.

This would then in turn be defeated by the cheating student running a remote microphone to a quiet room which can be switched to.

Honestly, most of the noise comes from the pickup. Using a simple switch to ground will still allow that hum through. You can also just loop a power cord around the mic line a few times and "actively" induce that noise.

My headset makes a terrible warbling modulated at a frequency of 1/120th hz from the bad isolation in my KVM's power supply.

No, a far lower tech option exists: more and more Console gaming targeted headsets like those from Steelseries include an aux in, which can easily be used for the snitch. Egg cartons for sound dampening and impromptu vocal booth and you're well on your way to defeating any technological measure.

leetcode-mafia4• 3 years ago

Unfortunately, I think you're right, at least initially. Not much happens unless the cost of losing access to remote talent becomes too significant to ignore.

Nathanba• 3 years ago

the example video isn't of leetcode though, it's general questions about database tech and it gives you all the right points to mention

weq2• 3 years ago

My cross platform c# apps are used by millions of students weekly to take assessments. We try and work as close with the OS venders as possible to implement lockdowns via native Kiosk functionality (ive posted bugs with all the majors MS/Google/Chrome/Android/Apple as their updates/move fast attitudes are probably my biggest enermies right now). We sometimes cant though and have to find other means because our clients demand we support xOS because vendors agreesively push hardware onto students. We definitely arnt spywhere, schools are very much against anything that will phone home alot of the time. We have pentests and issues like every other game/app in this cat and mouse race but as soon as you close our app it done, nothing lingers.

ChatGTP is a growing problem but we have defenses against it (until its on device maybe?). Things like Frida are probably worse right now.

switch007• 3 years ago

But I suspect what it will actually do is usher in an era of in-person only interviews

A man can dream! I come across really badly on video. I hate the compressed audio, lack of body language, latency etc

angarg12• 3 years ago

I also do tons of interview for a Big Tech. I ran my interview questions through chatGPT (3, the one available at the time), and the answers varied from pretty good to total garbage. I brought this to the interviewer community in the company, suggesting that it might be time to put leetcode-style interviews in the bin and start over.

They dismissed my concerns and will continue as usual. A part of me wants cheating to be rampant to force companies hand in the matter. Unfortunately I'm with you and I suspect they will just enforce in-person, whiteboard interviews again (a colleague explicitly asked for it), rather than trying to come up with a better system.

theRealMe2• 3 years ago

This “burn it all to the ground” mentality is naive.

There is nothing leetcode-specific about this cheating platform. OP just angled it that way because HN readers would lap it up that way.

Assuming someone was using this cheating platform, how would you run your interviews? Wouldn’t this screw up the actual legit interviews(whatever that is…) too?

jedberg5• 3 years ago

I don't do live coding in my interviews because it's silly. We talk about the person's past projects, their contributions to it, insights the gained, mistakes they made and how they would avoid them again, etc. Sometimes they may share some code they wrote and we walk through it and they explain it to me and why they made the choices they made. Sometimes I'll make suggestions and see how well they receive that feedback.

I'm looking for people who can demonstrate that they have faced challenges and overcame them, that they communicate their decisions effectively, and that they can learn new information quickly, can receive and incorporate feedback, and so on.

Can't really cheat your way out of that.

ukuina• 3 years ago

I can see a use-case for this that acts as a constant AI companion to pair-program with!

toomuchtodo• 3 years ago

RubberDuckGPT

breckenedge3• 3 years ago

LOL. I was thinking along the same lines about two months ago, but didn’t have the time to put together an app. Bravo!

https://news.ycombinator.com/item?id=34598251

morkalork• 3 years ago

Okay, so now interviewers will have to run their own copy and compare their answers to yours on real-time. Wonderful. Let the adversarial games commence!

wnolens2• 3 years ago

Need to seed it with a style, "give answers in the style of THESE GitHub repos"

DANmode2• 3 years ago

Local model, seeded with your style!

jedberg5• 3 years ago

If you ask ChatGPT the same question in different context sessions, you get vastly different answers each time. So that would not only give you lots of false negatives, but could also give false positives of people who are legit doing it and happen to come up with the same solution that GPT did.

leetcode-mafia4• 3 years ago

Indeed possible. That's partly why the prompt chain is set up to build a 'cheat sheet' type of answer, instead of a fully-formed answer that can be repeated verbatim.

test65542• 3 years ago

Interviewer: with your eyes closed, tell me the difference between…

codethief• 3 years ago

Easy, just use text-to-speech on the answer ChatGPT gives you and splice it into the audio stream you're hearing.

sdfgdfghj• 3 years ago

Nah they will just make face-to-face interviews the norm again :)

cbsks3• 3 years ago

When I went through a bunch of interviews in 2021 all my interviews were remote. After the first few hour-long interviews it became apparent that there were many repeat questions. I started writing the questions down along with my answers. I ended up with a notepad with ~20 of the most frequent questions and answers that I kept open on my screen next to the video conference. It was super helpful and now I have one of those $300k tech jobs!

Edit: these were the questions, the answers are left as an exercise to the reader (or your preferred AI):

What are you looking for in a role?

How do you deal with a conflict with a coworker?

What leadership experience do you have?

Do you have experience working with multiple teams?

What APIs have you designed?

What is priority inversion?

What are the differences between a mutex and a semaphore?

What is preemption?

How do interrupts work?

What is interrupt latency?

What is the difference between an ISR and a function?

What is the difference between an interrupt and an exception?

What is hard vs soft real time?

What is the boot process of a CPU?

What do you do for board bring up?

What are different memory sections used for (code, data, bss, etc.)?

What is a TLB?

What is the difference between big and little endian?

What is the difference between 32 and 64 bit processors?

What happens if a null pointer is dereferenced?

wordpad25• 3 years ago

courtesy of GPT4

Priority inversion: A situation where a higher-priority task is indirectly blocked by a lower-priority task holding a shared resource.

Mutex vs semaphore: Mutex ensures mutual exclusion for a shared resource, while semaphore controls access to a resource by multiple tasks with a counter.

Preemption: The act of interrupting and temporarily suspending a task, allowing another task to execute.

Interrupts: Signals to a CPU to temporarily stop its current task to handle an event or perform a specific function.

Interrupt latency: The time between the arrival of an interrupt and the start of the interrupt service routine (ISR). ISR vs function: ISR handles an interrupt, cannot be called directly, and must complete quickly; a function is a reusable block of code that can be called as needed.

Interrupt vs exception: Interrupts are external events requiring CPU attention, while exceptions are internal events caused by the execution of an instruction.

Hard vs soft real time: Hard real-time systems have strict deadlines that must be met, while soft real-time systems have more flexible deadlines.

CPU boot process: Initialization sequence a CPU follows upon startup, including loading firmware, running tests, and loading an operating system.

Board bring up: Process of validating and configuring new hardware to ensure correct functionality.

Memory sections: Code (executable instructions), Data (initialized variables), BSS (uninitialized variables).

TLB: Translation Lookaside Buffer, a cache for memory address translations in virtual memory systems.

Big vs little endian: Big endian stores the most significant byte first, while little endian stores the least significant byte first. 32 vs 64 bit processors: 32-bit processors have 32-bit wide registers and address spaces, while 64-bit processors have 64-bit wide registers and address spaces, allowing for larger memory and better performance.

Null pointer dereference: Undefined behavior occurs, often leading to a crash or unpredictable results.

sschueller• 3 years ago

"CPU boot process: Initialization sequence a CPU follows upon startup, including loading firmware, running tests, and loading an operating system."

That's a bit vague.

JambalayaJim• 3 years ago

Are you a systems engineer?

cbsks3• 3 years ago

Yes

mclightning• 3 years ago

Does the job actually require the knowledge behind these questions directly in live discussion environment? or are you able to get away with it by Googling/Stackoverflow/ChatGPT?

cbsks3• 3 years ago

My job does require knowledge of all of these things. It’d be hard to succeed if you had to look them up.

xiande04• 3 years ago

This is my interview strategy as well!

mkl2• 3 years ago

How good is Whisper for real-time, in terms of latency? I would love to hack together a dictation system at some point.

mkl2• 3 years ago

A partial answer: https://github.com/ggerganov/whisper.cpp/issues/10

neilv2• 3 years ago

I'm conflicted, in how I feel about this existing and surely being used by some...

I hate the Leetcode interview, with the fury of a thousand suns.

But I also hate cheating and lying, to a similar degree.

nsxwolf2• 3 years ago

If this helps me find great engineers who are bad at interviewing, great!

neilv2• 3 years ago

Is there any such thing as a great engineer who is dishonest?

ActorNightly3• 3 years ago

A recent M1 or M2 Mac is required for optimal performance.

Got a chuckle out of this bit.

davidatbu• 3 years ago

Can I ask: what did you find funny / worth remarking about?

ActorNightly3• 3 years ago

Its like a half meme that inexperienced programmers (which would be the target audience for this app) buy Macs because supposedly they are the "best" for programming.

google234123• 3 years ago

Can't wait for us to go back to in-person interviews. Since they are more expensive, there won't be a need to take chances on people that don't have the exact expected credentials

Fiahil• 3 years ago

I had a candidate, last week, that was using ChatGPT during a technical coding interview.

So, he managed to find perfect answers to the most difficult parts of the exercise in 30 seconds, and then struggled for 30 minutes, with my help, on getting the print right.

Of course, I used a slightly modified version of well known problems like "Fizzbuzz" and other stuff, mostly as a quick start to get into the real engineering challenge. It explains why ChatGPT was absolutely unable to finish the exercise. During the interview, it acts a a good filter. If it last 3 minutes, fine, no signal ; but if someone is struggling on it for half the interview time, it's a clear "No".

I haven't check if the candidate was invited to later rounds, but considering my negative review I think he was rejected.

I'm afraid these tools are not going to help you when things really matter :)

ainiriand• 3 years ago

I think there are a few edge cases where this could still be useful. I put myself as an example, I have been working as a backend engineer for more than 12 years and I still get very nervous in any interview. I struggle with a simple fib(n) function because my brain sabotages me. But I could happily talk about system design, microservices, or API resilience without problems.

robbywashere_• 3 years ago

That has to feel so awesome to reject these wannabe posers. Please please please tell us more :)

chmod7753• 3 years ago

I have a collection of one-liners by now which any developer could explain easily, but which ChatGPT-3.5, ChatGPT-4.0 and Bing's AI all get wrong in the same way. I think they all learned from a wrong SO explanation somewhere.

I'll part with one:

    Explain in detail what resolve is in this code:

    const promise = new Promise((resolve) => resolve(3));

    Assume I'm unfamiliar with callbacks or promises.

All three AIs will incorrectly state in their very first sentence that resolve is passed to the promise constructor as an argument.

There's also some technologies for which they can't write correct code (they'll hallucinate their API), but any software dev could when given their documentation.

james-revisoai• 3 years ago

Anthropics Claude is better, I'd say...:

The resolve function here is a callback function that is passed to the Promise constructor. It is called the executor function.

A Promise represents the eventual completion (or failure) of an asynchronous operation. The resolve callback is used to mark the successful completion of the asynchronous operation inside the promise.

So in this specific code, the promise is resolved immediately with the value 3. Basically it creates a Promise that is already completed with a value of 3.

To understand this more generically:

The Promise constructor takes an executor function which contains the asynchronous operation. This executor function has two parameters:

resolve: A function to call when the asynchronous operation completes successfully. It passes the result to the next then handler. reject: A function to call when the asynchronous operation fails. It passes the error to the next catch handler. The then and catch methods on a promise are used to handle the resolved or rejected state of the promise.

For example: promise.then((result) => console.log(result)) // Logs 3

So in summary, the resolve function resolves the promise and passes a value to the next promise handler, marking the successful completion of the asynchronous operation.

Does that help explain the resolve callback? Let me know if you have any other questions!

DogLover_2• 3 years ago

I could not explain the code and I have been developing for years... I asked ChatGPT and I feel I have better understanding of it now.

flangola7• 3 years ago

I got the right answer by asking GPT-4 to pretend it is a highly skilled senior software engineer and to write its thought process down step by step.

sizzle• 3 years ago

This blatant cheating should result in an instant 1 year ban from interviewing at company.

sebzim4500• 3 years ago

I don't think I would hire someone with the leet-code ability of GPT-4, though.

It does mean employers have to find questions outside the training set. I asked GPT-4 some of our hiring questions and it made some significant mistakes.

abecedarius• 3 years ago

~18 months ago I wrote in a comment here that I expected AI within 5 years from then to beat most working programmers at fresh new leetcode-style questions. It's not there yet but I wouldn't bet my career against this, when you compare to then.

(Reaction at the time was highly skeptical.)

sdwr2• 3 years ago

What do you want, credit for your wisdom?

That's not so special, lots of people predict the future. For example, I smelled cinnamon buns in the air last week, then yesterday, bought some cinnamon buns. How'als that for foresight?

m_a_g• 3 years ago

Looks good, but I'm having trouble installing it as an app and couldn't find relevant steps in the README. Do I need to build it with xcode or something?

leetcode-mafia4• 3 years ago

You can download the binary here: https://github.com/leetcode-mafia/cheetah/releases/tag/v1.0

muzani• 3 years ago

I hope this does to whiteboarding what Firesheep did to unencrypted http.

Still, this is impressive tech. It's a little like Iron Man's Jarvis where you can just talk to the computer and have it write code for you.

cramjabsyn• 3 years ago

I love this for whiteboard interview companies. Ask cs trivia questions, get AI responses and unqualified candidates. Hope it leads to an increase in conversational interviews.

JoshuaRogers• 3 years ago

While this is an interesting idea and somewhat amusing (kudos on the beautiful creativity of it), it’s worth noting that one of the second-order effects of this approach, were it used heavily enough to actually impact interview styles, is that it would further increase incentive to hire “in-network” with people that your team already know, while making completely unknown candidates a higher risk.

Effectively, it changes incentives and risk in a way that encourage the creation of echo chambers.

That said, once again, as far as a proof of concept this is wonderful. I’d say that it’s more art than product, and it makes a wonderful statement in the process.

lhnz• 3 years ago

  > and remember not to use the loopback device as input for the video chat app.

I laughed out loud.

htag• 3 years ago

The answer to this is simple. Make the "on-site" interview on-site again.

* You can rule out many types of cheating, which is becoming a more expensive problem.

* You can get many more details for the 'culture fit' in person.

* If you purchase airplane tickets for the candidate then you have built in identity verification via TSA.

* It demonstrates a higher degree of buy-in before the final interview from both parties.

Peroni• 3 years ago

This is categorically the right solution and the startups I work with that still have some form of office have all shifted to this structure. Take-home technical tests and virtual live tests are dead in the water.

raymondh• 3 years ago

From one point of view, this isn't a problem at all. If a candidate can use tools to generate, understand, demonstrate, and explain a solution in real time, that is likely what you want them to do on the job as well.

Also, it is likely that in the near future that most programmers will be "pair programming" with Copilot, ChatGPT, or some other tooling to augment their capabilities.

wintorez• 3 years ago

Love it!

armatav• 3 years ago

Hahaha brilliant

Show HN: Live coaching app for remote SWE interviews, uses Whisper and GPT-4

Related Stories