dpaleka

More targeted training won't do good, but why wouldn't more search help?

My understanding is that gwern above linked solid evidence in the paper for more search not being enough, as in, the model's evaluation NN is so way off target when searching, that realistic amounts of search don't help. Go seems to have many possible moves per position, so the search doesn't go very deep anyway.

Feel free to correct me if I'm wrong, it might be that I misremembered how AlphaGo-style systems work.

More search won't do good, but why wouldn't targeted training help? The way I see it is that the adversarial policy search discovers positions which are off-distribution for anything seen in the victim's self-play training.

But training on that particular sort of adversarial states should help against the human player which has learned the strategy, just like training on patch adversarial examples in vision helps against the same type of patches.

Of course if the adversarial policy is again allowed to find off-distribution states (by playing against the victim), it will certainly find ways to beat it, until the model is playing perfectly. (Emergent gradient obfuscation could also theoretically happen, but I don't know if it has been demonstrated to actually happen.)

That paper (ROME) was the most famous paper in the field last year :)

See also new interesting developments breaking the connection between "Locating" and "Editing":

https://arxiv.org/abs/2301.04213

Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models

[I mean no bad faith in this comment, I'm a fan of yours.]

Why answer questions about harmlessness/safety in such a roundabout way? Both OpenAI and Anthropic are clear about what words like "safe" are intended to mean: a stepping stone to "AI does not kill all people when given control".

Avoiding to state this clearly only invites unnecessary culture war disagreements in every discussion about these models.

A weird property of the described abstractions is that as you go tighter (interval -> zonotope -> polyhedra), the trained networks counterintuitively become less robust. Why does more precision in verification hurt training?

A recent work not mentioned in the last chapter "Adversarial Training with Abstraction" is [1], which kind of explains this issue using the notions of continuity and sensitivity of the abstractions.

[1] https://arxiv.org/abs/2102.06700

I too use large files with random notes, but I can't be bothered to write dates -- so I use git and cron to automate a searchable, persistent diary.

Let me write a blog post about it. The author of this article in particular might find it useful. Does anyone do something similar?

I feel the current marketing of topological data analysis and similar is directed towards people who have at least taken several graduate math courses.

I'm really not an expert in TDA, I only know Ghrist's book and have read Hatcher. Are there really important topology ideas in topological learning, or it can all be phrased combinatorially?

If it's an important research area, the current language really raises the bar to entry.

Thought that the article would be abou the quest for a better random graph model. It's weird that Quanta didn't talk about this yet.

For the other direction (ML helping graph theory and not vice versa), see Constructions in Combinatorics via Neural Networks [0] by Wagner (2021). I made an implementation to play with [1].

[0] https://arxiv.org/abs/2104.14516

[1] https://github.com/dpaleka/cross-entropy-for-combinatorics

How do models trained with Lightly compare with other approaches wrt adversarial robustness?

Can using Lightly introduce additional bias in the model, since only a select few of inputs are being labeled? This may be a concern for publicity purposes.

By the way, I thought ETH spinoff requirements were incompatible with YC requirements - nice to see it can be made to work.

No one is reaching out in Switzerland right now, but I guess they should.

Implementation where you can try to disprove conjectures yourself: https://github.com/dpaleka/cross-entropy-for-combinatorics