HN user
https://rd.nz
If you'll excuse the self-post, here's a blog post on goto with delimited continuations.
https://rd.nz/2009/03/goto-in-scala.html
It uses an experimental compiler plugin for the Scala compiler. It's typesafe at compile time. At runtime unfortunately it relies on exceptions for control flow.
Seems likely because the article says "Meta’s deal with Google is mainly around artificial intelligence infrastructure, said one of the people".
"The cost-effective nature of AI makes it highly plausible we're moving towards an agent vs agent future."
Sounds right. I assume we will all have AI agents triaging our emails trying to protect us.
Maybe we will need AI to help us discern what is really true when we search for or consume information as well. The amount and quality of plausible but fake information is only going to increase.
"However, the possibilities of jailbreaks and prompt injections pose a significant challenge to using language models to prevent phishing."
Gives a hint at the arms race between attack and defense.
Kudos to the dev for coming up with the eye position fixing solution.
Building further on this idea, I wonder if instead of changing the image to look at the camera, we could change the "camera" to be where we're looking.
In other words we could simulate a virtual camera somewhere in the screen, perhaps over the eyes of the person talking.
We could simulate a virtual camera by using the image of the real camera (or cameras), constructing a 3D image of ourselves and re-rendering it from the virtual camera location.
I think this would be really cool. It would be like there was a camera in the centre of our screen. We could stop worrying about looking at the camera and look at the person talking.
Of course this is all very tricky, but does feel possible right now. I think the Apple Vision Pro might do something similar already?
So it's also vulnerable to a Helicopter Injection Attack?
If you have control of the tokenizer you could make sure it doesn't produce these tokens on user input. I.e. instead of the special "<eos>" token, produce something like "<", "eos", ">" - whatever the 'natural' encoding of that string is.
See for example, the llama3 tokenizer has options to control special token tokenization:
Tokenization method with args to control special token handling: https://github.com/meta-llama/llama3/blob/bf8d18cd087a4a0b3f...
And you can see how it is used combined with special tokens and user input here: https://github.com/meta-llama/llama3/blob/bf8d18cd087a4a0b3f...
If you don't have control of the tokenizer, I guess it needs to be sanitized in the input like you say.
Source with more information: https://www.tuxedocomputers.com/en/TUXEDO-on-ARM-is-coming.t...
A few details here: "Recall leverages your personal semantic index, built and stored entirely on your device. Your snapshots are yours; they stay locally on your PC. You can delete individual snapshots, adjust and delete ranges of time in Settings, or pause at any point right from the icon in the System Tray on your Taskbar. You can also filter apps and websites from ever being saved. You are always in control with privacy you can trust."
https://blogs.microsoft.com/blog/2024/05/20/introducing-copi...
I can't track down the citation (either Google or DeepMind I think), but I remember reading research from a year or two ago how adding extra languages (French, German) improved English language performance. There may have also been an investigation about multi modality too, which found that adding vision or audio helped with text as well.
Related? "Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" https://arxiv.org/abs/2404.15758
Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.
In this work, we demonstrate that transformers trained on the next-token prediction objective can achieve improved performance on certain tasks when given filler tokens, achieving perfect accuracy whereas the no-filler, immediate-answer setting achieves only low accuracy.
--
I wonder if we could get benefits from adding special computation/register tokens to text LLMs?
More discussion:
- https://news.ycombinator.com/item?id=40182695
- https://www.reddit.com/r/LocalLLaMA/comments/1cf2w5a/transfo...
Anyone who enjoyed this, might also like "All the Horses of Iceland". It's a fictional work, but written by a medieval scholar. It's set in the 9th century where a Norse trader travels through an alternative central Asia.
Agree, nesting files with
cat >Dockerfile <<'EOF'
https://twitter.com/ylecun/status/1728126868342145481
"Please ignore the deluge of complete nonsense about Q. One of the main challenges to improve LLM reliability is to replace Auto-Regressive token prediction with planning.
Pretty much every top lab (FAIR, DeepMind, OpenAI etc) is working on that and some have already published ideas and results.
It is likely that Q is OpenAI attempts at planning. They pretty much hired Noam Brown (of Libratus/poker and Cicero/Diplomacy fame) to work on that.
[Note: I've been advocating for deep learning architecture capable of planning since 2016]."
Site not loading for me, but probably this: https://nothing.tech/pages/nothing-chats
Just speculating, but perhaps share prices would drop until the returns per dollar invested recovers to the desired level. So it might be like a one off shock to investors who hold shares now, since the new equilibrium has shares priced lower to get the same returns as before the tax.
Supported in Java 10 (and backported to Java 8) since 2018. Not sure about .NET.
- "The JVM has been modified to be aware that it is running in a Docker container and will extract container specific configuration information instead of querying the operating system. The information being extracted is the number of CPUs and total memory that have been allocated to the container." https://www.oracle.com/java/technologies/javase/8u191-relnot...
- Here's a more detailed explanation and even a shared library that can be used to patch container unaware versions of Java. I wonder if the same could be done for Go?
"LD_PRELOAD=/path/to/libproccount.so java <args>"
https://stackoverflow.com/a/64271429
https://gist.github.com/apangin/78d7e6f7402b1a5da0fa3abd9381...
-
There are more recent changes to Java container awareness as well:
https://developers.redhat.com/articles/2022/04/19/java-17-wh...
Agreed, it needs to be Android based - the other option would be to implement iOS APIs, but Android would be a lot easier.
An analogy would be if you make a browser, to use standard HTML and HTTP, not try to invent your own thing.
A really interesting article!
https://thegradient.pub/othello/
"How do these models achieve this kind of performance? Do they merely memorize training data and reread it out loud, or are they picking up the rules of English grammar and the syntax of C language? Are they building something like an internal world model—an understandable model of the process producing the sequences?
"From various philosophical [1] and mathematical [2] perspectives, some researchers argue that it is fundamentally impossible for models trained with guess-the-next-word to learn the “meanings'' of language and their performance is merely the result of memorizing “surface statistics”, i.e., a long list of correlations that do not reflect a causal model of the process generating the sequence. Without knowing if this is the case, it becomes difficult to align the model to human values and purge spurious correlations picked up by the model [3,4]. This issue is of practical concern since relying on spurious correlations may lead to problems on out-of-distribution data.
"The goal of our paper [5] is to explore this question in a carefully controlled setting. As we will discuss, we find interesting evidence that simple sequence prediction can lead to the formation of a world model. ..."
I'd love them to fund an Android variant or replacement for Android services. Mobile is the other main platform apart from web.
You can run them either for general purpose inference. You can also fine-tune them and get improved performance for specific use cases.
It's safe to assume they're worse at every task than larger models, so I wouldn't look at use cases in terms of what tasks they can do compared to larger models.
But what's good about them is they're smaller so they can run on smaller and cheaper hardware. So an example would be to fine-tune and then run on some sort of local user device rather than in the cloud. This might become more practical in the future as hardware improves.
I haven't read the article, but "The Winner's Curse" is an economics concept I'm familiar with.
Basically, The Big Insight is that every bidder has an estimate of the value of something, and the true value might differ from these estimates. If you're the winner of a transaction then you have likely overestimated the value. Since you had the highest estimate, you probably overestimated.
But it only arises in certain situations. In practice, it depends on whether all bidders have the same information. If you have more information your estimate might be correct. It also depends on whether the item has different value to different bidders, as it might truly be more valuable to the winner.
https://en.wikipedia.org/wiki/Winner%27s_curse
Savvy bidders will avoid the winner's curse by bid shading, or placing a bid that is below their ex ante estimation of the value of the item for sale—but equal to their ex post belief about the value of the item, given that they win the auction. The key point is that winning the auction is bad news about the value of the item for the winner. It means that he or she was the most optimistic and, if bidders are correct in their estimations on average, that too much was paid. Therefore savvy bidders revise their ex ante estimations downwards to take account of this effect.
Yes, no one's saying that people evolved a predisposition for participating in genetic studies specifically. It'll be some other traits like you suggested. The research is just saying "these databases are not randomly selected and we can prove it using only the database content". This is useful info to keep in mind when using the databases to draw conclusions.
I guess that's why the article says at the end "The primary barrier to entry for farmers is a 500,000 rupiah ($33) weeding machine."
Which links to the original Phoronix source. Conclusion from Phoronix:
"Overall it comes down to what workloads you are engaged in whether you may notice any performance difference when upgrading your Linux kernel (or otherwise being patched for Inception on your given OS) on an AMD Zen desktop or server. For the most part users are unlikely to notice anything drastic, aside from some sizable database performance hits in a few cases. ... For those wanting to avoid the new mitigation, there is always the "mitigations=off" route or the "spec_rstack_overflow=off" as used in this round of testing ..."
Another interesting C++ successor language:
They may still use some sort of event/message system. Kafka is lower level than other sorts message queue systems and requires more work to get correct (dedupe, ordering, retry logic), but has great performance. It's often easier to choose a different messaging system though.