HNGlance

Top Best New Show

Using Multimodal LLMs to Understand UI Elements on Websites

https://qa.tech/blog/using-multimodal-llms-to-understand-ui-elements-on-websites/

by daniel_mp • 2 years ago

11 5 2 years ago

QA

Related Stories

Loading related stories...

Comments

5

patricklef• 2 years ago

MLLMs are surprisingly bad at this out of the box and to some extent even with fine tuning. https://jina.ai/news/the-what-and-why-of-text-image-modality...

while1• 2 years ago

Loving this! Very surprising that the LLMs of today are so bad at understanding interfaces but it also makes it a very interesting case for finetuning!

antonoo2• 2 years ago

Love that you made an interactive app to visualize how the model performs similar to how Meta AI usually releases their models

(direct link: https://qa-tech-minicpm-demo.gptengineer.run)

antonoo2• 2 years ago

Did you use AI to generate this app?

daniel_mp• 2 years ago

yeah, we used https://gptengineer.app/ to create the playground! highly recommended

Source preview

qa.tech

HNGlance

© 2024-2026 HN Glance. Story content belongs to its original sources.

Privacy
Terms