RETRO is fast

http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html

by bilsbie • 4 years ago

141 20 4 years ago

PKop3• 4 years ago

I find the font contrast far too low, and therefore the text is hard to read.

Unless I have something specific to my computer going on, or something was altered in the last 30 minutes, I'm seeing black on white.

Palomides• 4 years ago

I get #3c484e text with weight of 200

rightbyte• 4 years ago

I barely can read it. I got that thing where circles are oval-shaped at the eye exam.

dang• 4 years ago

"Please don't complain about tangential annoyances—things like article or website formats, name collisions, or back-button breakage. They're too common to be interesting."

https://news.ycombinator.com/newsguidelines.html

PKop3• 4 years ago

OK I didn't know this was against the rules, I see it on here often. Small quibble: difficulty to actually read the submission doesn't seem completely tangential.

bitforger• 4 years ago

Hey! Author here. I've noticed on certain mobile devices the contrast can be weird, not sure why.

I updated the default theme's text color to be 10% darker, lmk if that looks better to you.

slater• 4 years ago

Chiming in here w/ same complaint ;)

On macOS (display: 15.4-inch, 2880 × 1800), it's really difficult to read. I set the font to ''400 1.2rem/1.5 "Fira Sans",sans-serif'' and color to #111 in dev tools, way better readability.

(sidenote: is Fira Sans a default installed font on Linux systems? I'm on macOS and don't have that, and don't see a font embed anywhere in your source code. So that might be the issue - 'sans-serif' at 200 weight is way too faint)

PKop3• 4 years ago

Pretty sure it's the font-weight that needs increased, when you do the current color is fine.

visarga2• 4 years ago

This is great for speed, maybe we can also increase the window size if the model is so small, but how about the quality of the generated text? With a 20x smaller model does quality drop?

How many chunks do you retrieve? The paper shows best results at k=1 and then at k>50.

sanxiyn• 4 years ago

If you are talking about Figure 1 of https://arxiv.org/abs/2112.04426, you are reading it wrong. Y axis is bits-per-byte and lower is better.

visarga2• 4 years ago

I saw the graph. Bits-per-byte is not clear for me (for example lower loss does not always mean better accuracy when you train a classifier). What I needed is to know if the generated text has the same "literary" qualities, and also if it can compete in few shot mode on as many tasks as GPT-3 can do.

In other words, can we rely on it instead of GPT-3 in realistic scenarios or is it good only for low BpB?

mbforbes• 4 years ago

Did anyone else read the generations? They’re… really poor quality, right? Im not sure whether I’m misunderstanding though.

ShamelessC4• 4 years ago

I don't think they are generations, but rather samples from The Pile that are semantically close to the input.

Actually, as far as I can tell - the RETRO arch itself isn't trained in this article. It focuses more on how to build the retrieval system with a fast KNN index over all of the Pile.

hwers• 4 years ago

I guess an interesting way to translate this technique to text-to-image would be to get an image from a database that matches the text query (via CLIP) and feed that + noise into a diffusion model that only does a few denoising iterations (and no clip guidance maybe). Would be a lot faster than from-scratch diffusion.

(Another way could be to redo the architecture to include a “inspired by this image” input, which is queried from an image server at inference time.) Anyone have other ideas?

ShamelessC4• 4 years ago

This was one of the motivations for `clip-retrieval`, a faiss index over the CLIP embeddings (CLIP ViT-L/14 to be precise) for all the captions/images in the LAION5B-Aesthetic dataset.

https://rom1504.github.io/clip-retrieval

Try the reverse image search - it can be shockingly effective.

You can pretty easily rehost the index or build a lookup over your own data if you check the GitHub repo.

If you don't have any data of your own, enter a query and hit that download icon to get a CSV of `URL,Caption,CLIP score`.

DoctorOetker• 4 years ago

I tried "a man with shopping bags stopping a tank" and was hoping to get the Tiananmen Tank Man, but I'm having no luck with variations either.

EDIT: it does contain a blurry picture of the tank man and some LEGO re-enactments when I query "tiananmen tank man", but was hoping it would more intelligently deduce the picture from the description

nutanc• 4 years ago

Have been thinking about this idea. Are there any open source implementations of diffusion models which take an image as an input?

RETRO is fast

Related Stories