sumanyusharma

Congratulations on the launch. Few qs:

How do your agents decide a suspected issue is a validated vulnerability, and what measured false-positive/false-negative rates can you share?

How is customer code and data isolated and encrypted throughout reconnaissance, exploitation, and patch generation (e.g., single-tenant VPC, data-retention policy)?

Do the agents ever apply patches automatically, or is human review required—and how does the workflow integrate with CI/CD to prevent regressions?

Ty!

How is this different from Integuru? They posted a few weeks back here: https://news.ycombinator.com/item?id=41983409

Appreciate the info; I'll double-check Fidelity again!

I'm actually pretty interested in what you're building. Sure, Vanguard and Fidelity are well-established giants, but they've barely moved beyond standard ETFs for decades. Having the option to tweak weightings at a more granular level and do daily tax-loss harvesting at scale seems like a genuine step forward.

I also like that you're transparent about how you might eventually introduce additional revenue streams like margin lending or maybe even PFOF. Knowing that upfront is better than a sudden terms-of-service surprise down the road. Still, I'd hope you'll consider giving users some say over how their shares are handled — like opting out of lending — so your incentives stay aligned over the long run.

Congrats on hitting $10M AUM. I'm rooting for more low-fee alternatives that keep the user in the loop!

No plans for acquisition :)

Building product, talking to customers and making something people want!

We're focused on end-to-end evals focused on function-call accuracy, style, tone & latency of the conversations between our sims and your voice agent. Less focused on pure TTS evals at the moment!

Pipecat looks awesome! I'll run the examples over the weekend and try to see what the integration hooks need to look like: https://github.com/pipecat-ai/pipecat/tree/main/examples

It should be pretty straightforward at first glance!

Likely outsourced call centers since call complexity is low to medium. We also expect rapid adoption in industries like customer service, healthcare, and retail, where 24/7 availability could be high-impact for businesses and convenient for consumers!

Should be fixed now; could you try again please?

We forgot to enable non-US numbers in our config for the demo. (oops)

We're working on a fix right now!

I am curious - how was the team solving this at Kea?

I use Superwhisper (no affiliation, just a happy user), which runs a local Whisper model, to create most of my email drafts and post-meeting notes. I find Whisper more accurate than Mac’s built-in speech-to-text, plus I’m faster at speaking than typing.

Sometimes, I even ‘talk’ into Cursor’s chat window instead of typing. The only downside? It can get a bit annoying for others when you're talking to yourself all day.

I'm curious to learn more about what's blocking the widespread adoption of the LLM capabilities. Lack of knowledge, reliability, or something else?

This tracks. Text evals to test core logic and voice evals for overall end-to-end performance!

It's a bit of a catch-22.

Making current voice agents reliable is incredibly time-consuming and complex. This challenge has kept many teams from pushing their agents into production. Those who do launch often release a very limited, basic version to minimize risk. We frequently talk to teams in both camps.

As a result, there aren't many 'killer' voice products on the market right now. But as models improve, we'll see more voice-centric companies emerge.

Teams are already calling their agents by hand and keeping track of experiment runs in a spreadsheet. We're just automating the workflow and making it easier to run experiments!

Our customers, who build voice agents, are often asked by their customers to make their voice agents more human-like and flexible. Their clients — businesses like pest control and automotive repairs — value providing a personalized experience but want the convenience and reliability of a 24/7 booking and answering service.

Bolna looks awesome! We've considered going open-source, but we're not sure how to effectively manage a community.

I'll reach out async!

Absolutely agree that creating effective evals requires domain expertise. Right now, we're co-building evals with customers, but we're identifying which aspects can be productized.

Regarding text-based evals — part of testing voice agents involves assessing their core reasoning logic. To do that, we bypass the voice layer and simulate conversations via text. So yes, the core simulation engine is reusable for both conversational text and voice interactions.

We're also excited about shipping the ability to replay a simulated conversation inspired by a real user!

Yes! We're aiming to build a tool that both engineers and non-engineers love.

We've discovered that it's often faster for non-technical domain experts to iterate on prompts in a structured, eval-driven way, rather than relying on engineers to translate business requirements into prompts.

While storing prompts in code offers version control benefits, it can hinder collaboration. On the other hand, using a pure CMS for prompts enhances collaboration but sacrifices some modern software development practices.

We're working towards a solution that bridges this gap, combining the best of both approaches. We're not there yet, but we have a clear roadmap to achieve this vision!

I wonder if a more optimistic version of this could be used to train humans and improve their skills. I'm thinking along the lines of LeetCode / Project Euler, but more dynamic and personalized!

Few examples:

1) Customer service: Simulating challenging customer interactions could help reps develop patience and problem-solving skills.

2) Emergency responders: Creating realistic crisis scenarios (like 911 calls) that could improve decision-making under pressure.

3) Healthcare: Virtual patients with complex symptoms could speed up the learning rate for med students.

4) Conflict resolution: Practicing with difficult personalities could aid mediators and negotiators.

5) Sales: AI-simulated tough customers could help salespeople refine their pitches and objection-handling skills in a low-stakes environment.

Thoughts?

Yes! Drive-through customers can be very impatient. We tried to make the demo persona maximally annoying.

Testing for edge cases is especially important because getting an order wrong can cause health hazards, long line-ups, and churn!

Nice! What's the use case your agent solves for?

I'm happy to spin up some scenarios that are more relevant for you instead of our stock demo personas :)

Feel free to email me at sumanyu@hamming.ai

Yup, we named it after Richard Hamming. His essay 'you and your research' was deeply influential during my undergrad; I re-read it every quarter.

Our current product draws inspiration from Hamming distance because we're comparing the `distance` between current LLM output vs. desired LLM output.

I appreciate your candid feedback. Our aim isn't to push for replacing humans but to ensure that when companies do use LLMs, they work as intended and don't create more problems. We'd rather see well-functioning systems than glitchy ones that frustrate everyone involved!

Hi HN - In collab with UWaterloo, we published a new code-focused needle in the haystack benchmark.

TLDR - GPT-3.5-Turbo showed lower accuracy on the BICS benchmark than the BABILONG benchmark at the same context length and target depth, indicating that LLMs struggle more on code-based tasks than text-based tasks at long context length. - The hype is real. GPT-4o showed the best performance, closely followed by GPT-4-Turbo. The GPT-4-Series especially performed well at long context lengths compared to other models. - Llama3-70B reached GPT 3.5-Turbo levels (yay open-source!) - Gemini 1.0 Pro was bad across the board (super surprising)

Link to repo: https://github.com/HammingHQ/bug-in-the-code-stack See full results here: https://hamming.ai/blog/bug-in-the-codestack

Hi HN - We built a new benchmark called "Bug In The Code Stack" (BICS) to test how well LLMs can find syntactic bugs in large Python codebases. (similar to a text-based needle-in-the-haystack test)

GPT-3.5-Turbo showed lower accuracy on the BICS benchmark than the BABILONG benchmark at the same context length and target depth, indicating that LLMs struggle more on code-based tasks than text-based tasks at long context length.

GPT-4o showed the best performance, closely followed by GPT-4-Turbo. The GPT-4-Series especially performed well at long context lengths compared to other models. Gemini-1.0-pro performed the worst, surprisingly worse than Llama3-70B.

Generally, longer context length resulted in lower accuracy. However, there were some exceptions to this. Models react differently to the placement of the bug within the source code. GPT-3.5-Turbo and Claude 3 Opus were the most sensitive, and GPT-4-Series was the least sensitive. Generally, less sensitivity means a more robust model.

This benchmark has lots of limitations. I would love your feedback & suggestions on how we can make this benchmark more useful!

Link to results: https://hamming.ai/blog/bug-in-the-codestack Repo: https://github.com/HammingHQ/bug-in-the-code-stack

Love this; grats on the launch Sam + Jitesh!