Show HN: VisionScript, abstract programming language for computer vision

https://github.com/capjamesg/visionscript

by zerojames • 3 years ago

93 23 3 years ago

Hello! I'm James and I am working on VisionScript. With VisionScript, I want to empower people -- including everyone without any prior programming experience -- to build cool apps with vision.

This weekend, I recorded a demo for VisionScript, in which I made apps that count how many cats are in an image and hides people in a video. Each app was < 10 lines of code.

https://vimeo.com/856043804

VisionScript is built for the 10 year old inside of me who would have loved more visual programming languages with which to play. I want to show people the potential of programming and how you can make what you want with computers, whether it be a game that counts cats or an app that monitors how many birds flew past a tree. Those "wow" moments should come as soon as possible in one's learning experience.

VisionScript is in active development. I started work on this project in July. There will likely be bugs; this is a passion project. Inspiration comes from Wolfram and Python. Behind the scenes, I am adopting what I am calling "lexical inference", which is to say there is a last state value on which functions infer; the language manages types and state.

tgv2• 3 years ago

Looks cool. I'm not sure about the implicit state in the language. It sort of worked for Hypercard, but not always.

And can you iterate over multiple objects with the same label, e.g. if you want to replace all faces by emoji?

zerojames8• 3 years ago

Thank you for your comment! Iteration is in the works right now.

I wrote this script at the weekend:

    Load["./bus.jpg"]

    In[Detect["person"]]
        Greyscale[]
    Endin

    Show[]

In[] is our iterator (which came from "in a folder", which was the original use). I'm adding data types. In the above program, In[] is iterating over a Detection object. Behind the scenes, each bounding box is selected, the nested statements (in this case, Greyscale[] are evaluated), then any transformations are applied to only the regions in the bounding box.

Thus the impact is:

    1. Load a file
    2. Find all the people
    3. For each person, turn that region greyscale
    4. Show the resulting image

My rationale for implicit state (I haven't heard that term until now -- thank you!) is that it may be more intuitive for a beginner. Maybe that will be wrong, but I'm eager to experiment!

(I am beyond elated to be on the front page of HN!)

(edited to get the code snippet and list formatting right!)

tgv2• 3 years ago

I don't know where I got the term "implicit state" from. It might have been an article or a discussion or even a spontaneous thought.

it may be more intuitive for a beginner

It keeps things simple, true, but the conceptual leap is bigger when it no longer suffices. But that's a bit of a luxury problem.

I'll be keeping an eye out on your repo.

rocauc2• 3 years ago

This is really powerful. Can't wait till there's the ability to make custom detection blocks!

zerojames8• 3 years ago

I have been thinking about this! What would you like to be able to do?

On a separate note, one idea I had was to have rock paper scissors as a quickstart. You could Use["rock paper scissors"] to use that as a model; there could perhaps be a registry of curated, fun models that people can play with out of the box (aka ones good for classroom use).

bippihippi1• 3 years ago

huggingface integration?

ulrikhansen54• 3 years ago

How do you plan to expand the functionality? I've been looking for a more modern alternative to OpenCV for a while - especially replacements some of the aged object tracking functions, there's been a wad of new stuff coming out that OpenCV has been slow to implement.

symisc_devel• 3 years ago

Shameless plug: take a look to our embedded computer vision library SOD: https://sod.pixlab.io. It's a lightweight OpenCV alternative targeting embedded devices with most of the modern image processing algorithms already implemented including an experimental Stable Diffusion implementation.

zerojames8• 3 years ago

Great question! Development has been guided by my asking myself -- and discussing with friends -- what problems I/we would like to solve (or what would be fun!), then building for those use cases. This is evident in the WIP DetectDistinctScenes[] method.

Example script:

    In["file.mov"]
        Classify["building", "garden"]
    Endin

    DetectDistinctFrames[]

This method is more advanced than most. It uses CLIP behind the scenes to detect when classifications change for more than N frames in a video, and generates a list of timestamps you can use to determine scene changes.

I'm keen to build more specific functions for common CV tasks. My code most certainly needs cleaned (the main lang.py is > 2k LOCs right now) but I do have a contributing guide that you can follow to add features should you be interested: https://github.com/capjamesg/visionscript/blob/main/CONTRIBU...

rocauc2• 3 years ago

a new, popular library for basic functionality (converting between annotation formats, evaluating models, doing object tracking) has been supervision: https://github.com/roboflow/supervision

zerojames8• 3 years ago

To the extent that VisionScript uses it behind the scenes for various things (i.e. counting objects in a zone https://github.com/capjamesg/visionscript/blob/main/visionsc...)!

janalsncm• 3 years ago

This is really cool. I wonder if there’s a way to extend Scratch to include functionality like this.

zerojames8• 3 years ago

Thank you so much! Comments like this have really made my day.

Scratch does have an "Extensions" feature, so perhaps it would be possible (https://github.com/scratchfoundation/scratch-vm/blob/develop...)!

ukuina• 3 years ago

Very cool! We need to normalize powerful capabilities like this in simplified interpreted language form for the next generation of programmers.

zerojames8• 3 years ago

Thank you so much!

I have been thinking about learning progressions with a language like VisionScript. I see a notebook-like environment being a great place to start; dragging and dropping is intuitive (and fun!).

But if that notebook can introduce someone to the same primitives they can use in a text-based programming language? That gives you the ability to help someone graduate to a full programming environment.

Notebook -> Text that uses the same syntax as the notebook (you could drag+drop in the notebook then write out the text as a starting point!) -> Writing your own programs.

A friend suggested that the implicit state approach of VisionScript could be used in other areas too like audio. I reasoned that we could actually use many of the same primitive names and syntax; just a different backend. I don't know enough about audio processing, but the idea is fascinating!

ano88888• 3 years ago

Very cool. Will try it out. Thank you so much for your work.

zerojames8• 3 years ago

My pleasure! I sincerely appreciate your words! It was a pleasure to be on the front page of HN.

CyberDildonics2• 3 years ago

You say python is an inspiration, but how is this better than just using opencv from python?

zerojames8• 3 years ago

VisionScript goes beyond OpenCV in terms of utilities, providing models for use out of the box.

To classify images, you can use Classify["class 1", "class 2", ...]; to run object detection you can use Detect["object"]; to run segmentation, use Segment["class"]. More abstract methods include Search[], that lets you build a semantic search engine in a few lines of code:

    Load["./folder/"]
    Search["coffee cup"]
    Get[1] # get the top result
    Show[]

I am aiming to encode best practices from computer vision behind the scenes (I work in the industry) to enable one to build powerful applications intuitively, with few lines of code!

chpatrick• 3 years ago

Why do you need a new language with new syntax to do this instead of making a Python library?

  vision = Vision()
  
  (vision
    .load("./folder/")
    .search("coffee cup")
    .get(1)
    .show()
  )

I think unless your language semantically does something really different it's almost always a mistake to make a new one. Adding global state and replacing () with [] doesn't seem worth losing compatibility with the rest of the ML/CV ecosystem, and forcing people to learn syntax just for this.

CyberDildonics2• 3 years ago

VisionScript goes beyond OpenCV in terms of utilities, providing models for use out of the box.

That's great, but I have to agree with chpatrick that this seems like a cool library but it is unlikely to get used as a stand alone language.

New languages cut off a huge amount of utility. Decades of IO, libraries, tools, built in features, gc refinement etc. go out of the window. I'm not sure what the advantage is. If it is a simple standalone library that has advantages over other things out there it stands a lot more chance of being used.

dibjuv• 3 years ago

One idea is to feed some examples of this language into an LLM. Then you could take out the programming part entirely and just have natural language description -> program -> result. I'm thinking something like this paper: https://arxiv.org/abs/2211.11559

gitgud• 3 years ago

Very cool! Vision systems can be so daunting to play around with, but this seems very approachable!

There's several segmentation models listed there, do you plan to support "Segment Anything" at some point? I remember reading it was a pretty good model... but I'm not an expert

Show HN: VisionScript, abstract programming language for computer vision

Related Stories