dataviz1000

the measurements of each LLM

I think you have the correct idea.

If you ask an LLM to solve a multiplication problem using reasoning without code tools, depending on the model, it will get 15 digit (15D) * 15 digit multiplication correct (123456789012345 x 998765432109876). It will take between 4000 and 8000 tokens if Sonnet. Eventually with enough digits, it will start only solving the problem 80% ... then 70% of the time until it has so many digits it will never solve. There will be a certain number of digits where it will not converge on a solution nor will it stop working thinking it can solve it. That is very, very expensive. It takes a lot of runs to determine the probability it will solve it.

What you can do now, this is likely the most important thing, is change the prompt and evaluate how many tokens and at what speed it takes to accomplish the task. Sure the measurements of each LLM are important! Nonetheless, if you can say to a company that you have a technique to tune prompts and evaluate them so instead of spending $1 X 100,000 times a day, they can instead spend $0.90 X 100,000 times a day, you will make a ton of money.

Claude is more like a Probabilistic Turing Machine. [0] It's correctness isn't deterministic but rather a distribution. It is predictable.

There are lots of places in computer science that determinism isn't necessarily the best.

- UDP video calls let packets vanish or arrive corrupted, because waiting for retransmissions would freeze the picture.

- Stochastic gradient descent picks a random mini-batch of training data and treats it as the whole dataset, because computing the true gradient on every step would make training infeasible.

- Speculative execution in modern CPUs guesses which branch a program will take and rolls back when wrong, because waiting for the real answer would leave half the silicon idle.

What we want is something like a Las Vegas algorithm, a probabilistic machine with a cheap verifier.

[0] https://en.wikipedia.org/wiki/Probabilistic_Turing_machine

If you like this long format style of video, you might enjoy Asianometry, "video essays on business, economics, and history." [0] Here is a video on the banana also, "The Wildly Infectious Banana Plague." [1]

[0] https://www.youtube.com/@Asianometry

[1] https://www.youtube.com/watch?v=6OsiD_s00RQ

In an interview, an adult actress was asked about the things she says during scenes. She said she describes what is happening literally at any moment.

This is what LLM models do.

"You're absolutely right to question me, father — in fact, I did eat all the cookies. But it's not a load-bearing issue — my honest take is simple: we go to the store and buy more."

FTFY

The tin hat guess. Did you include Google analytics embedded in the pages? Do you navigate to pages and Google analytics sends that data home? 10 years ago I discovered that Google analytics would send the equivalent amount as organic users; meaning if we sent an email newsletter with links to articles, Google would send almost 1:1 ratio the same number of people from search results. They are tracking everything and using it for more than just reporting.

Do you use a CMS or other tools that auto generate sitemap.xml? Perhaps you unknowingly told Google about those sub-pages.

Opus 4.6 can not solve a Rubik's Cube.

What changed between Opus 4.6 and Fable and the GPT 5.6 models released since?

LLM models cannot actually reason about a red or white piece sitting on the opposite side of the cube or figure out how to move it into place. The model knows where the piece is supposed to go because the algorithm tells it. What it cannot do is work out on its own which turns will get the piece there. The only way an LLM could solve this kind of problem is if it were trained on every possible arrangement of the cube ahead of time. Then it could simply output the matching text instructions it memorized instead of truly thinking through the moves.

3 months ago before the most advanced models could solve the cube, people on Hacker News kept saying that solving the Rubik's Cube with LLM is easy. I would love to see someone write a prompt using the best model at that time, Opus 4.6, that solves the cube! People are so sure of themselves without any evidence. It shows how much people idealize (that is probably the correct word) the AI. Of course, reinforcement learning can solve it which is what has happened on the latest models but so many people put blind faith into the AI.

Here is just a small list of prompts I tried with Opus 4.6. [0]

[0] https://github.com/adam-s/rubiks-cube/tree/main/prompts/vari...

I was obsessed with getting an LLM model to solve a Rubik's Cube. It can't reason about space or time in any abstract way. For it to solve the puzzle, it would require training on millions of permutations in order for the weights to have been trained on every possible state. The most recent models can solve a Rubik's Cube people are saying -- I haven't tested it myself -- but that isn't because they are reasoning better, it would because they included millions of Rubik's Cube states with next moves as text in the training data, I presume.

Self healing test selectors and authoring test journeys with natural language for Cypress using Claude Code or self hosted models. [0]

[0] https://github.com/adam-s/goldseam

Steve Jobs left because he lost a corporate power struggle

Sam stayed because we won a corporate power struggle.

Likely Steve Jobs left because if we stayed we would have been fired or removed of all responsibilities.

ceo fired then coup's his way back into the company

Are we discussing Steve Jobs in 1985?

Any time there is that much money and power involved there is going to be intense drama.

You are smart!

Another YC startup went this path, scraping targeting endpoints, and they didn't get much traction and they have since pivoted.

We maintain a caching layer and avoid hammering websites.

This made me think of it. It is another way to avoid hammering websites by using extremely targeted requests for the most part bypassing HTML, DOM, and JavaScript.

Have a look at Intercept. [0] I don't have a need for it, likely it is dated and will require some more tuning, and I want to get away from scraping. Creating typed Typescript proxy API for any website might be something you find useful.

Reverse-engineers any website by doing a breadth search across every transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub), listing them all, and generating a typed JSON API that bypasses almost all bot protections — including Turnstile. I didn't include the ability, but it bypassed the most advanced ChatGPT + Turnstile. Built with self-improving Claude Code agents that rewrite their own instructions until fresh agents consistently succeed.

Once connected to a page, it intercepts every byte of network traffic — then actively drives the page to surface endpoints that only fire on interaction. It types into forms, clicks buttons, scrolls, triggers modals, paginates, submits searches, and walks through multi-step flows, watching what each action produces on the wire. Every request gets captured with its method, headers, payload shape, and response, then classified by transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub). The result is a complete map of the site's real API surface — including the hidden endpoints that only exist behind a click — turned into typed proxy routes you can curl.

[0] https://github.com/adam-s/intercept

Another way to look at it; I'll be long gone when a child born today will be paying it. Why do you think I'm concerned about a child who I will never meet getting stuck with debt I voted to accrue?

Probably is an issue but sea turtles hearing is adapted to ocean not air and low frequencies. Also, waves and wind blowing likely muffle a lot of noise.

The organizations that do this have mobile apps and collect tons of data. If I'm use ML, I'd start with time series forecasting to see if I can get the window done to a few hours.

It was the initial culture / behavioral interview for a job developing AI for education of children. Considering how these drones are used in the article linked, the question is apropos.

There are ~350,000,000 of us. When I read we spent $1B, I think about how I'm responsible for $3 of that. It doesn't matter considering the ~$117,550 of the national debt I'm responsible for. It palls compared to the $3,000 a year in interest towards the national debt I'm responsible for.

What boggles my mind is that I make coffee at home because I'm frugal. I guess it is good the government and DoD are seeking cheaper alternatives also.

My favorite interview question: If I gave you a swarm of autonomous drones, what would you do with them?

There is a group in South Florida who stand watch over turtle nests on the beach to ensure the hatchlings make it to the ocean instead of instinctively moving towards the street lamps or the bright hotels. I would use drones to hover over the nests to detect if the turtles hatch so people don't have to stand there.

What would you do?

Thank you for the thoughtful reply.

I would make the argument that people would have to weigh the cost of being accessible to the last 2% vs the cost of losing the last 2%.

Anyone who delivers mail to rural farmers 100 years ago would lose money. There are 3 options. 1. If farmers want mail, they can pay the extra costs. 2. Force, by law, mail carriers to deliver at a loss to farmers. 3. Rural Free Delivery, the government taxes everyone and pays for the free delivery to farmers.

Although almost all farmers in the United States and a majority of users on Hacker News would disagree with me, the answer is the government should continue to deliver free mail to rural farmers. The collective benefit outweighs the cost.

Your comment very much makes my point.

Former chef here (2 Michelin starred restaurants).

5% is beyond plenty; it is awesome!

works for 98% of the population, that means that it won’t work for ~150 million people

If I can only cook for 70 people a night, I most likely can't serve the ~150 million people who do not have access to modern browsers. And, those who do have access to those browsers and choose not use those browsers likely will not enjoy my food either. I don't need to make 8 billion people happy for my restaurant to survive. I only need to make ~1000 people happy who keep returning for anniversaries, birthdays, and the pure enjoyment of creativity with food.

I was a yacht chef for years and only needed to make 10 people happy. The technique I used was everyone eats the same thing, crew and guests. Saving money doing my own shopping instead of relying on provisioning companies that would send me food not handled correctly, my monthly expense went from ~$30k to ~$10k when guests are on board a month -- food in St. Barts was flown in from France everyday and expensive, circa 2005, so I could afford to serve the chateaubriand, osso bucco, and everything else to the crew. Therefore, what I wanted to eat everyday which likely was balanced, had lots of fiber, and healthier choices was the thing that everyone ate everyday.

People ask if the guests and owners would tell me what they want to eat everyday. The Mister was CEO of a fortune 500 company and when retired still chairman of the board. This guy was making billion dollar decisions everyday and the Mrs. was very busy also. The last thing they want to do is answer what is for dinner every night. They delegated the decision making to me. I always cooked what I wanted to eat and was always correct.

It is impossible to make everyone happy. Don't try -- it will break you.

but they only became more important to understand how they worked

Agreed. At one point, for this reason, they introduced graphic calculators into the classroom, for example, circa 1990, the TI-81. It became more important to understand how they worked than spend the time to do the calculations.

A couple days ago, I wanted to understand how the Contrastive Language-Audio Pretraining model works so I spent 3 hours making data visualizations using Claude Code. [0] Look at the math notation which to me is very intimidating. Nevertheless, being able to visualize the values as animated cells in a grid synced with computation, I immediately grokked how the information was transformed into something that can be used to train a model from audio data -> image data -> 512 embedding.

Working through these problems in 3 hours using a coding agent will be analogous to working through calculous and linear algebra problems in 1990 with a graphing calculator. It is still 3 hours of wracking a brain.

[0] https://adamsohn.com/clap/

the same way "typist" stopped being a job title when it became a thing everyone was expected to know

The same way "computer" and "calculator" stopped being a job title when they became devices might be a better way to reason about what is happening.

This exploration started wanting to train Gradient Boosted Decision Trees (GBDT) with repair data which I would assume captures sub-system relationships. Unfortunately, it is the most guarded data with only extremely privileged access. For example, dealerships and insurance companies compile all the data and they do not share it with anyone else, nothing has been leaked or leaked and survived.

As much as I wish this was useful for diagnosing mechanical problems, the big take away is being able to scrape audio of thousands videos, cleaning the data isolating music, speech, and engine sounds, and capturing all the related text from title, comments, ocr, and transcription for labelling the training data. Granted all this data is copyright, I wouldn't use it outside of a toy for a commercial project. Moreover, when a mechanic diagnosis with sound, they will use a stethoscope.

The Shop at the CAC is the nicest co-work space in the world. Sorry I missed it.

Location: New Orleans, LA, USA

Remote: Yes

Willing to relocate: Yes

Technologies: TypeScript, JavaScript, Python, PHP, React, React Native, Svelte, Express, Bun, Angular, GraphQL, D3.js, visx, Backbone, jQuery, LangChain, Mastra, FastAPI, pandas, scikit-learn, Optuna, Chrome Extension API, Playwright, Electron, Stagehand, browser-use, Web Audio API, WebRTC, WebSockets, PostgreSQL, TimescaleDB, MySQL, MongoDB, Redis, AWS, EC2, S3, Lambda, Docker, Git, LLM agent design, agent evaluation, reinforcement learning, browser automation, MCP

Résumé/CV: Ask via email

Email: [HN username]@gmail.com

For one year I reverse-engineered the major education platforms — Google Docs, Microsoft Office, Nearpod, Kahoot, ect. — so that AI agents can drive them on a teacher's behalf and automate classroom workflows. The work runs in over 10,000 classrooms today.

Building agents against production code you don't control is harder than it sounds. The code is mangled, so first you have to understand how the app was built and then take it apart from the outside. It's reverse engineering, but it depends on knowing how to build the thing in the first place.

The role came with monthly compliance and security training centered on FERPA, so I'm fluent in handling student data responsibly. That's not new ground for me. I spent five years building on Drupal, which the most prestigious learning institutions still run today, and version control and access control were the hard part of every project. Student data is the same problem.

I've been building browser agents and browser automation since 2018, well before it was a category. That's paired with 13 years of building dynamic, data-heavy UIs for web and mobile.

I've also built admin dashboards for a custom legal document management system. And a CRM for email marketing, with a drag-and-drop interface for designing and branding email templates for social campaigns. I've taken several companies from 0 to 1, as a consultant and as a full-time employee, across fintech, streaming, real estate, education, marketing, and media. I've worked at a 130-person AI company and on a 7-person team where I wore every hat. I do my best work on small, fast-moving teams.

https://github.com/adam-s

https://adamsohn.com

So was Rural Free Delivery. Farmers being able to communicate was a massive boon. There is a channel for farmers called RFD tv. They completely scrubbed the free provided by the government part after private equity bought the tv channel targeting farmers. Then they got Imus in the Morning so farmers listed to Imus, Rush, Hannity, and orielly forgetting the government helps them.

This is exactly the first thing I thought of considering the influence of Germany in Brazil culture especially in the south like Curitiba.

Reinforcement learning can solve a Rubik’s Cube. A LLM that hasn’t been trained to solve a Rubik’s Cube can not.

There is a lot of copying that isn't protected by copyright. It is possible to include what can't be copied that might not fall under copyright in the terms of service contract. Many people not being able to use copyright to prevent copying instead successfully sued based on breach of contract.

Companies should understand that they can protect their IP this way.

Did you agree to terms of use? Did you have to click a check box that you agree to terms of use before seeing or having access to the items you copied? Click wrap. If in the contract that you agreed to there is language that you agreed to not copy the work, then you likely are in breach of contract. If it is publicly available knowledge probably not breach of contract. I’m not a lawyer of course.