Dragonfly: A large vision-language model with multi-resolution zoom

https://www.together.ai/blog/dragonfly-v1

by jasondavies • 2 years ago

143 36 2 years ago

ilaksh2• 2 years ago

I have been testing out LLMs with the together.ai API, but I can't figure out how to use the multimodal models with the API. I don't see any in their model list.

cateye2• 2 years ago

Me too. I use now https://fireworks.ai/ for vision models to test.

GaggiX2• 2 years ago

Is there a demo or API to test the model? There are so many vision language models these days, it's hard to say which one is better, they also use in many cases different benchmarks.

alexey-salmin• 2 years ago

Huggingface links are in the article https://huggingface.co/togethercomputer/Llama-3-8B-Dragonfly...

GaggiX2• 2 years ago

I wasn't asking about running the model locally, also for that I have to wait for someone to quantize the model.

vessenes• 2 years ago

together.ai started as an API-based inference provider, so I bet you can target these models directly with your API credits there.

ilaksh2• 2 years ago

How? I don't see any vision-language model in their model list for the API.

achristmascarl• 2 years ago

For the model fine-tuned on biomedical image data, does anyone with domain knowledge know how the model's answers compare to the "Gold" answers?

TechDebtDevin2• 2 years ago

I don't have exact domain knowledge but I'm fairly certain this type of tech has already been employed to do some of the heavy lifting for radiologists reviewing imaging results.

rrsp• 2 years ago

Both the ‘gold’ answer and the model reference a PA and AP view respectively as well as a lateral chest radiograph. The picture only contains a lateral radiograph though.

quantumwoke• 2 years ago

Ranges from weird phrasing to outright incorrect. Oh well, early days

TechDebtDevin2• 2 years ago

Ive been sorta following together.ai for a while. Cool company. Is this available to be used by anyone atm? Could I potentially use the model to look at my own chest xrays (I've had a lot)?

imjonse• 2 years ago

They have released the models https://huggingface.co/togethercomputer/Llama-3-8B-Dragonfly...

darby_nine3• 2 years ago

I can't speak for others obviously, but this sort of caption is nauseous:

In the heart of a vibrant skatepark, a skateboarder is caught in a moment of pure exhilaration. The skateboarder, dressed in a black t-shirt adorned with a yellow graphic and black pants, is suspended in mid-air, performing an impressive trick on a concrete ramp. The skateboarder's arms are outstretched, adding balance to the daring stunt. The skatepark itself is a concrete playground, with the skateboarder's ramp being the main focus. In the background, palm trees sway gently, adding a touch of nature to the urban setting. A few spectators can be seen in the distance, their attention riveted on the airborne skateboarder. The image captures not just a moment, but a story of skill, courage, and the joy of skateboarding.

This seems a lot more like a puff piece from a local publisher trying to fill space, or description of a stock photo to an advertiser, than a description I'd describe as accurate from a human to another human.

esafak2• 2 years ago

That's the price of getting an image that accurately represents what you had in mind. Otherwise you could just prompt it with "skateboarder in a skatepark".

Jackson__2• 2 years ago

In the heart of a vibrant skatepark,

Vibrant? It's made of white concrete.

a skateboarder is caught in a moment of pure exhilaration.

Oh, great we're mind-reading from pictures now? His face isn't even visible.

adding balance to the daring stunt.

What incredible commentary.

palm trees sway gently, adding a touch of nature to the urban setting.

The trees aren't swaying. But because sometimes trees sway, I guess it'll just say it.

A few spectators can be seen in the distance, their attention riveted on the airborne skateboarder

The spectators look more like they are just glancing over, nowhere near "riveted".

The image captures not just a moment, but a story of skill, courage, and the joy of skateboarding.

This adds literally nothing.

darby_nine3• 2 years ago

It doesn't accurately represent the photo, hence the issue at hand. In many ways "skateboarder in a park" is more accurate than the large number of small inaccuracies this description manages to accrue. (Like many humans, but still the verbosity is very odd for the little detail it actually conveys!)

I'm not trying to argue against the idea of ai-generated titling, just that the product is very inferior to what even mild diletantes of the field might expect from advertised capabilities.

jebarker2• 2 years ago

This isn't a text to image model it's an image captioning model. The images in the figures are confusingly labeled since it's the caption that's generated, not the image I think.

Jackson__2• 2 years ago

Yeah, this is my number one complaint with all recent open source vision models, and it seems like it is only getting worse. It's verbose to the point of parody, making it extremely difficult to evaluate what it can actually _see_, and what it's just dumbly markov-chaining based on previous text tokens.

In GPT4V, you can prompt around this if you know about it, but none of the people collecting datasets for open models appear know or care to apply that, and so we just get this default GPT4V contamination everywhere.

The only vision model I enjoy is Google Gemini, simply because it will give you a no-nonsense caption. Of course it still hallucinates things that are not there, but getting a color or object wrong is orders of magnitude less bad than having 3 sentences that have nothing to do with the image.

fragmede• 2 years ago

I got chatGPT to say

A skateboarder attempts an awkward mid-trick in an average skate park, with distracted onlookers.

about the image, but where's the fun in that,

throw310822• 2 years ago

Came to say the same. Might be the task "describe the picture" that puts it into that mode; however I still hope that no human being would really write such tosh.

darby_nine3• 2 years ago

I really am very pro ai describing images, but it's the editorializations like "The image captures not just a moment, but a story of skill, courage, and the joy of skateboarding." that struck me as—idk, quite odd and uncanny-valley like.

jebarker2• 2 years ago

It's not only nauseous to read but it's also too speculative, e.g. how do you know the trick is impressive? How do you know the skateboarder is exhilarated?

natch2• 2 years ago

Exactly. Ugh.

In the heart of

It's clearly the edge of a skatepark. Not "the heart."

etc. etc... others have gone through it extensively so I won't do that again, but it's full of gratuitous added content that does not match what's in the picture at all. It seems to be aping some writing style, not going for accuracy.

What's more remarkable to me is that the authors of this do not seem to notice.

It's bizarre that you would create a project using an approach and then when assessing that project prior to publication, you would just glance at the paragraph without even reading it, and say "looks great" and move on. Details matter. This is absolute crap and we need researchers who can discern crap, not just accept anything.

Further down in the post they ask why an image showing a dog as the Mona Lisa is funny. It's actually not funny. It's an old trope, and, as such, super dull. They should realize it's funny only to a subset of people, but they don't seem to even realize that much. This team needs to get out more.

worstspotgain• 2 years ago

It's just an artifact of the training sample pairs IMO. You can reduce it by fine-tuning the prompt, or even LoRA if you'd like to nuke this mosquito.

davidhyde3• 2 years ago

“ Question: Write a detailed radiology note based on the chest X-ray. Gold Answer: AP upright and lateral views of the chest were provided. Left chest wall pacer pack is again seen with leads extending into the right heart. ”

The bit about a “wall pacer pack is again seen…” leads me to believe this was based on another doctors note about a similar looking X-ray which was probably paired with other information like another scan at the time. That would be problematic imo.

BriggyDwiggs42• 2 years ago

Problematic for the functionality? If it works well enough, I’m pretty fine with them stealing data to create a useful medical tool.

davidhyde3• 2 years ago

It’s problematic because the LLM is describing another person’s scan and not the one presented to it. It should at least present the other scan as workings and the percentage difference between the two. Finding a similar looking scan is very useful no doubt but if the result is hallucinated that it is less so. Dangerous even. There is no confidence percentage and there should be.

planb• 2 years ago

The Gold Answer is not the output of the model but the expected answer in the benchmark. Probably the benachmark contains multiple consecutive images of the same patient.

davidhyde3• 2 years ago

Thanks, I probably shouldn’t be commenting on an LLM article if I don’t even know what a gold answer is! Oh well.

esafak2• 2 years ago

Is there a comparable service for audio analysis?

jerpint• 2 years ago

Not yet, some models in that direction are LTU-AS

cateye2• 2 years ago

It is strange that this model is not available on Together.ai to try it out after reading the blog artcile.

stainablesteel• 2 years ago

this looks quite impressive

if image generation gets to be near perfect then it might have a larger impact on communication than gpt does, no paragraph beats a good diagram but drawing is always hard

Dragonfly: A large vision-language model with multi-resolution zoom

Related Stories