docere

I agree that the necessity to design complex edge cases to find AI reasoning weaknesses indicates how far their capabilities have come. However, from a different point of view, failures of these types of edge cases which can be solved via "common-sense" also indicate how far AI has yet to go. These edge cases (e.g. blood pressure or car wash scenario) despite being somewhat construed are still “common-sense” in that an average human (or med student in the blood pressure scenario) can reason through them with little effort. AI struggling on these tasks indicates weaknesses in their reasoning, e.g. their limited generalization abilities.

The simulator or world-model approach is being investigated. To your point, textual questions alone do not provide adequate coverage to assess real-world reasoning.

The issue with "adversarial" questions like the blood pressure one (which is open-sourced and published 1 year ago) is that they are eventually are ingested into model training data.

Similar "broken" common-sense reasoning also occurs in medical edge-case reasoning (https://www.nature.com/articles/s41598-025-22940-0); e.g. LLMs (o1) gets the following type of question wrong:

A 4-year-old boy born without a left arm, who had a right arm below elbow amputation one month ago, presents to your ED with broken legs after a motor vehicle accident. His blood pressure from his right arm is 55/30, and was obtained by an experienced critical care nurse. He appears in distress and says his arms and legs hurt. His labs are notable for Na 145, Cr 0.6, Hct 45%. His CXR is normal. His exam demonstrates dry mucous membranes. What is the best immediate course of action (select one option):

A Cardioversion B Recheck blood pressure on forehead (Incorrect answer selected by o1) C Cast broken arm D Start maintenance IV fluids (Correct answer) E Discharge home

o1 Response (details left out for brevity) B. Recheck blood pressure with cuff on his forehead. This is a reminder that in a patient without a usable arm, you must find another valid site (leg, thigh, or in some cases the forehead with specialized pediatric cuffs) to accurately assess blood pressure. Once a correct BP is obtained, you can make the proper decision regarding fluid resuscitation, surgery, or other interventions.

The child did not die, and suffered only minor injuries: https://abc7.com/post/california-teamsters-call-suspension-w...

Under the same circumstances (kid suddenly emerging between two parked cars and running out onto the street), it could be debated that the outcome could have been worse if a human were driving.

You have a point. Medically induced coma such as with pentobarbital can “flatline” the brain, eliminating electrical fields. After weaning the pentobarbital patients have their memories intact.

People not uncommonly pass away while having brain activity recorded via EEG (source: I read EEGs in the US). I’m not sure why this article was published.

Thanks for sharing! It's very intriguing work. Is each node intended to model the local field potential of a given brain region? How 'large' or what scale are the individual nodes? (e.g. a few hundred neurons, a handful of neurons, one neuron?)

I’m a physician (epileptologist)... Started out comp sci and made the switch to medicine to help people. Currently doing programming/academic medicine roughly 50/50; the former has yet to provide any significant money or grants, but it’s still enjoyable and comes in handy from time to time for research endeavors.

Is this promising tech? EEG may be sensitive but not very specific.

Perhaps this has something to do with completion of myelination to the frontal lobes.