HN user
Medical expenses would be a combination of cost of insurance / treatment (PPP relevant) and government transfers in a country with partially or fully public healthcare, which is why it's so important to do both.
I see. That paper is looking a median income from a PPP perspective but without accounting for taxes and transfers, which is a tiny bit unorthodox in this exact context which tries to get into the weeds slightly more than just standard PPP adjusted GDP per-capita or something like that. It's valid statistically, but typically if you're trying to adjust for cost of living it makes a lot of sense to use the amount of actual income the individuals have left over after paying taxes and receiving transfers from social programs. Otherwise you'll have a number which accounts for differences in prices but not differences in the proportion of "income" which is actually retained by the earner. This is what that wikipedia article I linked is reporting. Comparing two different countries with very different tax/social program policies with PPP adjustments but without tax/transfer adjustments is less than ideal.
It's not wrong, it's just not very useful. It's typically a more intermediary statistic rather than a final one when doing this type of comparison.
median is higher in Germnay
I cannot think of any standard by which this is true, certainly not by nominal or PPP income for either personal or household income.
[1]https://en.wikipedia.org/wiki/Median_income
[2]https://worldpopulationreview.com/country-rankings/median-in...
Yes, since the weights being updated are a small subset of the overall total it's manageable. Just like how each separate conversation currently requires you to store a separate KV cache, you'd need to store the fast weights separately. Both KV cache and fast weight content stores have to be conversation specific, so just setting a bit of extra RAM aside for "memory" isn't really a new ask, just a different format for an old problem.
The idea of periodically stopping to write blocks of recent context into a fast-weight state is interesting, but I think it liked it better when E2E-TTT[1] did it. It's a more flexible and elegant continuous learning approach.
Essentially it goes "You know how your model can remember its training data? Well, what if you treated its recent context like more training data and updated (some of) the weights using (mostly) the same process used to train it?"
The end result is very good at remembering things but also really good at adapting to new unseen distributions.
You can merge it into OpenWhip https://old.reddit.com/r/ClaudeAI/comments/1sek7gk/someone_m...
Its turn around time is ridiculous, it has to be maintained with specialized equipment/hangers, along with external contractor assistance.
Compared to the Gripen, as an example, which can land on a freeway and be up in the air again in a few minutes.
I have no idea where people got the idea that the F-35 requires a major refit after each sortie or that it needs climate controlled hangars, but there's literally no truth to any of it.
The turnaround time for an F-35 after a mission in a wartime scenario isn't going to be much different from any other older fighter jet. Refuel, rearm, get back in the air.
One of the key requirements for the F-35 programs was to minimize extra care needed for the RAM (Radar Absorbent Material). Unlike older stealth aircraft the F-35's ram is "baked in" to the aircraft skin, rather than being a coating. The F-117 and B-2 require climate controlled hangars because their coatings are old and delicate, the F-22 doesn't, but needs regular touch-ups for its coating, the F-35 is just left sitting outside most of the time regardless of where it's operating, a desert, the arctic, a jungle, the deck of a ship, you just leave it out there. The only common maintenance done on the F-35's RAM is replacing a relatively small amount of special RAM tape which is usually used around the edges of the access panels which are opened for other types of maintenance.
Cerebras has effectively 100% yield on these chips. They have an internal structure made by just repeating the same small modular units over and over again. This means they can just fuse off the broken bits without affecting overall function. It's not like it is with a CPU.
That's what it's running on. It's optimized for very high throughput using Cerebras' hardware which is uniquely capable of running LLMs at very, very high speeds.
You prevent enemy air defenses from shooting down your aircraft by blowing them up as part of SEAD/DEAD missions, which is exactly what the US did.
Takes much longer to build, requires a much larger up-front investment, and requires a lot more land.
The footprint needed when trying to generate this much power from solar or wind necessitates large-scale land acquisition plus the transmission infrastructure to get all that power to the actual data center, since you won't usually have enough land directly adjacent to it. That plus all the battery infrastructure makes it a non-starter for projects where short timescales are key.
Precision bombing during WW2 was not possible at the required scale. To put a bomb precisely on target back then you needed something like a dive bomber, a tactic which is incompatible with strategic-scale bombing. Even "precise" methods using advanced analog computers like the Norden bombsight could only do so much.
Under combat conditions the Norden did not achieve its expected precision, yielding an average CEP in 1943 of 1,200 feet (370 m)[1]
This means that 50% of bombs fell within 1,200 feet of the target, which is an absolutely awful accuracy if you're trying to hit anything specific.
This was further compounded during the campaign against Japan by the heavy reliance of Japanese wartime industry on cottage industries which were dispersed almost randomly within Japanese population centers, rather than being located within specialized industrial districts. From a purely strategic standpoint which is only concerned with destroying the enemy's ability to make war, the most effect way to disrupt these kinds of industry with 1945 technology was essentially to burn every building in the city to the ground. Other options were simply ineffective.
Nice to see that they added that, but that section wasn't in the article when I wrote that comment.
This article doesn't mention TPUs anywhere. I don't think it's obvious for people outside of google's ecosystem just how extraordinarily good the JAX + TPU ecosystem is. Google several structural advantages over other major players, but the largest one is that they roll their own compute solution which is actually very mature and competitive. TPUs are extremely good at both training and inference[1] especially at scale. Google's ability to tailor their mature hardware to exactly what they need gives them a massive leg up on competition. AI companies fundamentally have to answer the question "what can you do that no one else can?". Google's hardware advantage provides an actual answer to that question which can't be erased the next time someone drops a new model onto huggingface.
[1]https://blog.google/products/google-cloud/ironwood-tpu-age-o...
People would correctly identify that their standard of living is being reduced for ideological reasons without tangible individual benefits and would likely not respond well to that, resulting in a loss of political power for whatever movement instituted those policies and a reversal of said policies.
Yeah, that's the normal outcome for papers like this. Papers which claim to be groundbreaking improvements on Transformers universally aren't. Same story roughly once a month for the past 5 years.
If that were totally true you would expect to see more or less uniform ratios of HP/HD cells mixes across different product types, but that's very much not the case. Dennard scaling may be dying but it's not dead yet. You can still sacrifice efficiency to gain performance. It's not zero sum.
If it were that simple fabs wouldn't offer a standard cell libraries in both high performance and high density varieties. TSMC continues to provide both for their 2nm process. A tradeoff between power efficiency and raw performance continues to exist.
Intel on 18A is literally TSMC's 3nm process + backside power delivery, which means more power efficiency, performance also less heat.
That's a pretty serious abuse of the word "literally" given that they have nothing in common except vague density figures which don't mean that much at this point.
Here's a line literally from the article
Based on this analysis it is our belief that Intel 18A has the highest performance for a 2nm class process with TSMC in second place and Samsung in third place.
Given what we currently know about 18A, Intel's process appears to be less dense but with a higher emphasis on performance, which is in line with recent Intel history. Just looking at the density of a process won't tell you everything about it. If density were everything then Intel's 14nm++++ chips wouldn't have managed to remain competitive in raw performance for so many years against significantly denser processes. Chip makers have a bunch of parameters they have to balance when designing new nodes. This has only gotten more important as node shrinks have become more difficult. TSMC has always leaned more towards power efficiency, largely because their rise to dominance was driven by mobile focused chips. Intel's processes have always prioritized performance more as more of their products are plugged into the wall. Ideally, you want both but R&D resources are not unlimited.
The cost for both training and inference is vaguely quadratic while, for the vast majority of users, the marginal utility of additional context is sharply diminishing. For 99% of ChatGPT users something like 8192 tokens, or about 20 pages of context would be plenty. Companies have to balance the cost of training and serving models. Google did train an uber long context version of Gemini but since Gemini itself fundamentally was not better than GPT-4 or Claude this didn't really matter much, since so few people actually benefited from such a niche advantage it didn't really shift the playing field in their favor.
No, additional context does not cause exponential slowdowns and you absolutely can use FlashAttention tricks during training, I'm doing it right now. Transformers are not RNNs, they are not unrolled across timesteps, the backpropagation path for a 1,000,000 context LLM is not any longer than a 100 context LLM of the same size. The only thing which is larger is the self attention calculation which is quadratic wrt compute and linear wrt memory if you use FlashAttention or similar fused self attention calculations. These calculations can be further parallelized using tricks like ring attention to distribute very large attention calculations over many nodes. This is how google trained their 10M context version of Gemini.
Frankly, yes.
Many creative works these days require the effort and input of so many people, so much time, and so much money that they can't have a specific creative vision. Mediums like book, comics, indie movies, and very low budget indie games, where the the end product was created by the smallest number of people, have the most potential to be interesting and creative. They can take risks. This doesn't mean they will be good, most aren't, but it means that the range of quality is much broader, with some having a chance to shine in ways which big budget projects just can't. The issue with small teams and small budgets is that they are inherently limited in what they can create. Better tools allows smaller groups of people to make things that previously would have required an entire studio but without diluting the creative vision.
Will this also result in a tidal wave of low effort garbage? Of course it will. But that can be ignored.
The APU offerings from both AMD and Intel have been improving pretty rapidly recently but they're still pretty low end by dGPU standards. I can certainly see them causing the death of dGPUs in laptops but it's difficult to imagine a scenario where they're competitive with mid to high range dGPUs in desktops. I can't see either AMD or Intel trying to cram an iGPU that large onto one of their chips, it would be an extremely niche product and would be badly bandwidth starved.
The 18A naming convention is as simple as not wanting to put a decimal point in their node name. Simple as that. It's not like the the "nanometer" node names actually mean anything at this point anyways. Neither TSMC N3 or Intel 3 have any dimension in their transistors which are 3nm. I would be very unsurpassed to see other foundries copy the convention, it's a lot cleaner.
Intel 3 is an economically interesting node. Performance seems to be somewhere between TSMC N4P and N3B which is respectable but Intel 18A is due very soon after it debuts and incorporates GAA transistors and backside power delivery. It seems like Intel 3 is intended as an option for customers who need something very modern, but not absolutely cutting edge. I guess we'll see how much volume it actually gets over time.
The ISA is much less important than many people seem to think. The RISC vs CISC debate is beyond outdated at this point because no modern architecture actually works strictly like either under the hood. Organizations who did x86 architectures historically had much more emphasis on performance while organizations who did ARM had more emphasis on low power devices. The lingering engineering consequences of that history and the experience of the organizations doing design are orders of magnitude more relevant than the difference in actual ISA.
Reconnaissance is a prominent role but is hardly the only thing that these could be used for. What exactly is possible is going to vary a lot between designs but with the right design you can do electronic warfare, acting as a communications node, submarine hunting with sonobuoys, light strike using small precision munitions, light logistics for when someone needs literally 1 or 2 specific parts right this instant, and many other things.
I hope aviation is next. It is equally behind.
SpaceX took the launch market from a niche, almost artisan thing to something which actually does volume. There is no comparable improvement to be made in commercial aviation unless someone snaps their fingers and wills into existence both all the technologies and regulatory changes necessary to make flying cars work.
It's interesting to think that rather than being destroyed by becoming too expensive, ornamentation may have died because it became too cheap. A lot of ornamentation existed to show off wealth and status, but if everyone can have it thanks to improvements in production then it doesn't do that anymore.
It's unfortunate that making buildings look nice seems to be secondary to other types of status signaling. It's hardly a new issue either. When the Chrysler Building was completed in 1930 it was criticized for being gaudy for having the nerve to actually try to have some style.
"Lewis Mumford, a supporter of the International Style and one of the foremost architectural critics of the United States at the time, despised the building for its "inane romanticism, meaningless voluptuousness, [and] void symbolism".
Gaudi3 is supposedly due this year with a 4X bump in Bf16 training over Gaudi2. Gaudi is an interesting product. Intel seems to have something pretty decent but it hasn't seen much of a volume release yet. Maybe that comes with V3? Not sure exactly what their strategy with it is.
We do know that in 2025 it's supposed to be part of Intel's Falcon Shores HPC XPU. This essentially takes a whole bunch of HPC compute and sticks it all on the same silicon to maximize throughput and minimize latency. Thanks to their tile-based chip strategy they can have many different versions of the chip with different HPC focuses by swapping out different tiles. AI certainly seems to be a major one, but it will be interesting to see what products they come up with.