HN user
Reinforcement learning, deep learning, and artificial intelligence.
https://rl.ai https://github.com/rldotai
Damn. I bought rl.ai when I was a grad student but it's been laying fallow since I can't really blog about the stuff I'm working on right now. How does one go about selling their domain for millions of dollars?
According to others, that story is severely embellished:
https://savingjournalism.substack.com/p/i-talked-to-elon-mus...
I see a lot of really divergent results with these time series database benchmarking posts. Timescale's open source benchmark suite[0] is a great contribution towards making different software comparable, but it seems like the tasks/metrics heavily favor TimescaleDB.
This article has Clickhouse more-or-less spanking TimescaleDB, but the blog post it references[1] is basically the reverse. Are the use cases just that different?
-----
0. https://github.com/timescale/tsbs
1. https://blog.timescale.com/blog/what-is-clickhouse-how-does-...
A good idea and very cleanly implemented. I imagine that there's a ton of other possible applications that don't require much modification to the code. Thanks for sharing!
None of these solutions are ideal, although Zenodo's better than most. As far as I can tell, they're all targeted more towards the final, authoritative release, so it seems you're still out of luck during the paper writing process. What if I'm just trying to share a dataset/pre-trained model with remote collaborators?
I ran into this when doing some OCR experiments[1], finding acquiring data and pre-trained models to be the most time-consuming part of the enterprise. This ended up adding enough additional hassle that I didn't manage to get anything really interesting going, although figuring out how to containerize other peoples' code was educational. Personally, I think I'll be relying on some combination of institutional repositories + torrents/IPFS for any large datasets/models I end up releasing in the future.
-----
Who, whom?
Less tersely: this article is one in a long procession of journalists trying to exert control over tech. The opening example (Speech2Face, which they aver is transphobic) is inflammatory and utterly unrepresentative of the usual topics of AI conferences. The other references are far better, but the choice is revealing-- it's not so much an abstract concern about an unaccountable few exerting control from the shadows, but alarm that someone else might be muscling in on their territory.
This was great, actually. I don't program in Scala, but it was very interesting to hear about the difference between types as abstractions vs types as they are used.
For unfamiliar topics or when presented with uncommon insight, I believe rants, monologues, even diatribes are actually some of the best things to read.
Depends what you mean by "practical". Assuming you've done CV/ML stuff before, you could probably get something working pretty well over a weekend, and I think I could solve it completely with a bit more effort via 3D scanning + synthetic dataset generation... but unless you have cubic meters of lego to sort, doing it by hand would be faster, albeit less fun.
In the words of John von Neumann, there's a lot more that's known than is proved.
It's often frustratingly difficult to go from the known to the proven; still, just because something's not proved doesn't mean that scientists are ignoring it.
It would be nice if citing repositories were easier-- either for generating a reference for my own code or acknowledging when I've used someone else's code in my research.
There's tons of math and physics blogs that contain useful results that the author wanted to make available but didn't manage to incorporate into a paper. I wonder if there'd be any interest in a sort of GitHub for proofs? It could even use git, since (assuming consistency) isn't math just a DAG anyways (and therefore isomorphic to a neural net, as are all things).
A theory is nothing but a way to turn (a finite number of) measurements into predictions.
That's a very nice way of putting it. Is there a particular book, lecture, or other resource you could recommend for this interpretation of theorizing?
Furthermore, over the long term it can change the texture of the Reddit community. What sort of person quits using Reddit after becoming frustrated with yet another imposed hassle? What sort of person stays, or is retained by pop-up notifications and whatever other tedious blandishments their app/redesign provides?
There are other sites with similar features and layout to Reddit (Voat, Ruqqus, Saidit, Raddle, communities.win, etc.) but none are really in competition to be "the front page of the Internet". This is because Reddit has a huge community with varied interests which provides self-perpetuating advantages. For example: the activity level is generally higher; if you start a subreddit for a niche topic you have a chance at finding an audience; great comments/submissions will garner thousands of upvotes rather than a few dozen (smaller sites have a proportionately tinier vote ceiling, making it hard to differentiate between "good" and "amazing").
I can see increased retention/monetization maybe leading to a larger user-base in the short term, but if it makes good content less likely to crop up, or leads to less discerning users (ingenious effortposts lose out to low-effort pandering memes), or even if it just narrows the interests/hobbies represented on there, then over time it will destroy Reddit's main source of value.
It's consistent with it.
Rich used to be very bullish on neural nets, then somewhat dismissive of them (due to the fragility/inadequacy of FCNs), and then increasingly enthusiastic as the renewed interest demonstrated that those problems could be overcome-- e.g., through better initialization, training, and (as you note) different architecture choices. His main concern was whether a method could keep working as more resources became available, as otherwise you would tautologically end up with something short of true artificial intelligence.
The important thing is that the technique can scale with increasing data or compute without hitting a hard or soft limit.
You have misunderstood Sutton's argument. Quoting Rich:
One thing that should be learned from the bitter lesson is
the great power of general purpose methods, of methods that
continue to scale with increased computation even as the
available computation becomes very great.
It's a loathsome book for some people, even angsty teenagers, because not all of them are angsty for the same reasons.
I didn't enjoy it either when I was reading it, but later appreciated that it depicted a certain kind of angst— that of someone leaving childhood for adulthood, feeling anxious and threatened by this shift, and wishing to just remain as the most adult-like child (the "catcher in the rye" that gives the book its title) instead of having to go through that awkward period of being the most childlike adult.
Under that reading, it's much less irritating, because Holden isn't supposed to be valorized, just understood.
I believe "crank" suffices, or possibly "poseur".
It depends– if you're trying to run a specific algorithm on your new supercomputer, then you'd almost certainly be better off paying for researchers to optimize or improve on that algorithm. If that's the situation (which it is for e.g. weather forecasting or computational fluid dynamics), then a billion pound supercomputer is likely to be more of a boondoggle than a sharp-eyed investment. A good implementation on a desktop can beat a bad one running on a supercomputer.
But if it's a time-sharing system, then it might not matter as much. The supercomputer at my university tends to run a lot of one-off jobs like an experiment repeated thousands of times with different parameters. On a desktop that might take weeks, but if run in parallel it's like a couple hours. Tightly optimized code might bring that down to an hour on the cluster (or a mere week on my home PC) but I wouldn't bother because making the code more efficient might itself take a week or more. So the fastest way to get the results I need would be to just run it on the supercomputer.
Hey Will-- congratulations on the publication. It's very cool to see an algorithm reflected in the brain, and the turnaround (from idea to implementation to experiment) in... what, three years? is astonishingly fast.
On a more technical note, I was curious about whether the distribution is being approximated "properly" (in the sense of probabilities summing to one, no negative probabilities) via expectile regression. Does that hold here? I'm less than an expert on neurology so I am unsure if that would be necessary in vivo (since it's clearly not needed for good performance in silico).
Slow compared to what? You could write equivalent verbiage for the slow progress of mathematics, chip design, or cinematography-- the only difference is that uninformed randoms feel less bold when critiquing fields these fields (as opposed to AI), or perhaps people are less inclined to waste time reading such half-formed speculation.
The reason progress towards general artificial intelligence is "slow" is because it's a hard problem.
Formulating a definition of intelligence precisely enough that it can be optimized is incredibly difficult. We can capture facets of it, in some settings, but also have to deal with the cost of hardware and the difficulty of acquiring data. Sometimes you can get inspired by examining how animals behave, or analyzing the brain, or considering what it means to learn in abstract, but considering one thing in isolation means that you tend to hit a wall eventually.
We refine ideas, we try things, and are buoyed by the advances in technology that makes some strategies possible, but for every major milestone we have a ton of things that people tried and couldn't get to work. I personally have spent months working on stuff that ultimately yielded a minor improvement; I've spent days proving results that ended up taking half a page in some papers. It's not easy, and no one expects it to be easy, although some jerks shilling something singularity-related might make such claims.
But saying (to pick an example from the article) "oh, you just need to incorporate emotions, bro" into a learning agent is just about the dumbest thing I have ever heard.
My word choice is more driven by exposure to the dataset that I'm working with. There's been some recent successes with autoencoders trained on virtual sensory input (i.e., video games) with surprising results, e.g., neural networks that can simulate the dynamics of these environments with surprising fidelity.
Of course, learning to play video games at a high level is trivially easy, as everyone in the field now knows. The next challenge is, naturally, to make money doing this. But how? After the traditional thirty seconds of research before undertaking a major project, I determined that the only way to make money from video games is to become a popular streamer.
So now I am training an agent to generate video of it playing and reacting to an imaginary game and equally fictitious Twitch viewers, with a dataset drawn from the top Fortnite streamers. The reward function is comprised of a blend of subscribers, donations, and (logarithmically scaled) misogyny in the chat. Thus far, I've only managed to create some sort of window into hell, where the "game" consists of unceasing violence, murder after murder after murder as towers of mismatched material swell and fall in ever transforming locations on the isle while the chat endlessly subscribes, spams, and emotes in cackling glee and the superimposed webcam video features a... thing with too many eyes and hands screaming incoherently.
At first I thought it was a problem with my dataset, so I started watching some of the streams myself. This has not yielded insight into the whole "nightmare vision" output of my model, but it has expanded my vocabulary on the twin subjects of combustibles and comestibles, which I feel is a reasonable trade-off for the sanity battering associated with this whole endeavour.
Supposedly, yes-- during his final stay in the hospital, his brother read to him from a book they'd enjoyed during their childhood, Dickens' A Tale of Two Cities. When his brother had to turn the page, John would continue the narration from memory while his brother found his place on the subsequent page.
Given that he could be occasionally absent minded, I suspect that it had to be something that piqued his interest, but his sense of what was interesting was extremely broad.
He did in fact speculate on the workings of the brain in The Computer and the Brain, which is based on a lecture series he had planned out but did not deliver. It was more in the context of automata theory, but as someone with an interest in AI, automata, and neuroscience, it was frankly rather dank[0]. A lot of the pioneering work was, and is enjoyable in part because it's original and speculative, so you don't have to master the literature to make sense of it, you can just pick a paper and go. I'd recommend reading Pitts and McCullogh, plus also Lettvin, but others might have some equally lit[1] recommendations.
--
0. In the contemporary sense, c.f. "cool", "dope", or "excellent"; not dank like a root cellar.
1. vide supra
Von Neumann is something of a hero of mine, in part because of how he demonstrates (by example) that you can achieve surpassing insight by patiently applying logic and analysis to whatever problem is at hand. But also because he shows that you didn't have to be narrowly focused in your area of competence, you could pursue other interests and even excel in those, even if they aren't the things that you're known for.
He really liked history, particularly European history. There's some examples of this in his letters[0], but my favorite von Neumann story on this topic is how he managed to deduce the answer to a literary magazine's poetry trivia question when queried by his brother. They would both be teenagers at the time, and the magazine was in English, as opposed to their native martian, yet John got the solution at once.
Paraphrasing from here[1], although his brother's limited run biography/reminiscences has the story too.
The prize contest had the lines
They know this well my baron and my men
Gascony, England, Normandy, Poitou
That I had never follower so low
Whom I would leave in prison to my gain
I say this not as a reproach to them
But prisoner I am
"Did you know the poem?"
"No."
"Then how did you identify the poet?"
"Very simple" he said. "Gascony, England, Normandy, and Poitou were in one feudal hand only once during the early Plantagenets, and from there it was quite easy to associate with Richard's crusades and European captivity. But of course this is a translation, since quite obviously the early Plantagenets spoke Norman-Midieval French."
His brother then goes on to state: I found out much later that the translation was that of Henry Adams, and the Prison Song is only one of Richard's most perfect poems, usually referred to as gems of English literature!
---
0. One to his daughter springs to mind, where he expresses fatherly concern about her getting married during her undergrad-- he's worried that she's too young and this will derail her career. After that's addressed, he then moves on to talking about one of her term papers, and proceeds to suggest an insightful take on-- I believe, I haven't read the letters in a decade-- some medieval French bishop. The juxtaposition of concerned father plus European historian in a man more known for axioms and automata was jarring.
1. "The Legacy of John Von Neumann" from the American Mathematical Society, https://books.google.ca/books?id=XBK-r0gS0YMC&dq (see pg. 22-23)
Here in Canada, the "hysteria" is due to the perceptible economic slowdown, bordering on recession, associated with the carbon tax. If such a minor tweak is enough to make businesses pull investment, then surely more than tripling the price would be ill-advised.
Then you have the fact that transport, and hence fuel, affects the price of most goods, so fears of a recession (decrease in purchasing power) are not unfounded.
Also consider that, while justified as addressing a "tragedy of the commons scenario", rarely do such tax dollars go towards mitigating the externalities in question. The tax would sting less if it was being spent building renewable energy rather, than initiatives that are charitably described as contentious and perhaps more accurately described as pandering. If we could avoid the graft and self-dealing usually associated with federal spending, that would likely cut down on the complaints as well.
It depends how deep you want to go into the web archiving rabbit-hole, but using something like `archivenow`[0] would probably be the local optimum in terms of convenience, robustness, and permanence.
It's "a tool to push web resources into web archives", which basically entails making requests of archiving services (like the Wayback machine, for example). Using `archivenow --all <url>`, you can back up <url> to a whole bunch of different services, and get links to snapshots of that page at the time of access.
This is mostly sufficient, so long as at least one service is trustworthy and doesn't shut down unexpectedly. One issue that might come up is that the archives might miss content if the user has to interact with the page a bunch in order to reveal it, but that's unlikely to be an issue for the sorts of things an academic would be archiving.
As another poster mentioned, some bibliography managers have this capability baked in, or can acquire it with a plugin.
However, if you want more control, you might consider using something like `pywb`[1] to make your own web archive, and then host it yourself (e.g., on S3 or something). For more robustness, you could also publish those WARC files as torrents, or on IPFS, or if you're really cool, as part of the transaction messages on some blockchain (provided that you can stomach the transaction fees and don't anticipate having a large audience in China).
The above will probably suffice so long as the Internet continues to function as a going concern. To accommodate concerns about what happens in the case of total societal collapse, I've been backing up the papers that I cite by converting to binary[2] and preserving the result via aluminum punchcards, which I then mail to Sam Altman since he's got a way better game plan for the end times than I do[3].
I'm not sure if aluminum is optimal for this purpose though, both for material science reasons and because I've been informed that I am under investigation for my role in some sort of aluminum smuggling ring. If that fails, my last backup plan entails constructing an allegorical representation of the papers in question (which is surprisingly easy-- the Hero's Journey from Joseph Campbell[4]'s exegesis is basically about differential equations anyways), which I will then craft into a compelling mythology and teach to the local youths. Hopefully this will ensure that the citations are preserved via oral tradition even in the wake of cataclysmic change. Figuring out the BibTeX for this archival format will be pretty tricky, though.
But yeah, unless you have some extra aluminum lying around, `archivenow` or whatever's built-in to your reference manager should do the trick.
---
0. github.com/oduwsdl/archivenow
1. github.com/webrecorder/pywb
2. After stripping most of the markup removing images using `pandoc`, as aluminum is expensive.
3. https://www.bloomberg.com/features/2018-rich-new-zealand-doo...
You mention possible score functions in Problem 5, so I have a query about that. This has been bugging me for a bit-- is there a characterization of applicable loss functions for GANs? I'm curious about their effects on the results, but also if there's room for different losses. Or do we capture most desirable behavior using the listed score functions?
Richard Bellman coined the name, and according to legend it was because the phrase 'dynamic programming' was so anodyne that not even the most officious bureaucrat could object.
In Bellman's own words[0]:
"An interesting question is, ‘Where did the name, dynamic programming, come from?’ The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word, research. I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would turn red, and he would get violent if people used the term, research, in his presence. You can imagine how he felt, then, about the term, mathematical. The RAND Corporation was employed by the Air Force, and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word, ‘programming.’ I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying—I thought, let’s kill two birds with one stone. Let’s take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that is it’s impossible to use the word, dynamic, in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It’s impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities."
-----
0. From http://arcanesentiment.blogspot.com/2010/04/why-dynamic-prog...
Bertsekas' earlier books (Dynamic Programming and Optimal Control + Neurodynamic Programming w/ Tsitsiklis) are great references and collect many insights & results that you'd otherwise have to trawl the literature for. I'm very interested to see what a book focused more narrowly on RL will be like-- Sutton's Introduction to Reinforcement Learning[0] is fantastic, but if you're going to do research on RL, another text such as this one is necessary.
Cool! I'd also like to plug my own RL-related repositories: https://github.com/rldotai/rl-algorithms and https://github.com/rldotai/mdpy .
The first one implements some of the more "exotic" temporal difference learning algorithms (Gradient, Emphatic, Direct Variance) with links to the associated papers. It's in Python and heavily documented.
The second one (mdpy) has code for analyzing MDPs (with a particular focus on RL), so you can look at what the solutions to the algorithms might be under linear function approximation. I wrote it when I was trying to get a feel for what the math meant and continue to find it helpful, particularly when I'm dubious about the results of some calculation.
Yes.
I don't know what your thresholds for impressive might be, or what your basis of comparison is, but pure control theory seems to be at a bit of a dead end, whereas reinforcement learning allows for greater flexibility and robustness for control tasks, subject to your willingness to gather (or simulate) a lot of training data[0].
See for example this video[1] in which OpenAI shows off a robotic arm with "unprecedented" dexterity.
-----
0. https://www.youtube.com/watch?v=ZVIxt2rt1_4 for a video (there's also an associated paper) about setting up RL tasks for robots.
In imperfect information games (like SC2) the outcome prediction implicitly takes into account the unknown. Given what it has observed and what it has not observed, it is essentially comparing the present state to similar situations from the past.
You can see this in the replay-- at seemingly random times, its outcome prediction jumps up, even when it hasn't had any interaction with its opponent. But that's precisely why it's going up-- it notices that its opponent has not executed a faster rush or cheese that it's unprepared for, hasn't expanded early, and the scout has not been destroyed.
Similarly, after having won a fight, the worst thing that could happen is that a bigger force emerge from your opponents base, destroying your army and giving them a chance to rebuild. When that does not happen, you know that your opponent probably doesn't have such an army (because otherwise they are falling behind in resources due to being bottled up). Either way, after a certain point it's worthwhile to press on to cause more damage, because you're now far enough ahead in resources that you will win regardless if they manage to repel that particular attack.