Weave: Interactive Data Exploration Toolkit by Weights and Biases

49 3 3 years ago

slewis2• 3 years ago

Hi! I'm Shawn, founder/CTO at Weights & Biases. We've been working on Weave for a couple years now, and it powers core parts of wandb.ai.

It's a UI toolkit built for programmers that can be reprogrammed from the UI itself. You might call it a "yes-code UI" :)

I'm very happy that we were able to release Weave under Apache2 yesterday, and there is a lot of cool new technology in here that we haven't had a chance to describe yet! I'll just try to give a quick tech summary for now.

There are three core components in Weave: Types, Ops, and Panels.

Ops are typed functions like:

  @weave.op()
  def flip_lr(im: Image.Image) -> Image.Image:
      return im.transpose(Image.Transpose.FLIP_LEFT_RIGHT)

Panels are UI elements that register to render a given type:

  @weave.type()
  class BertvizHeadView(weave.Panel):
      id = "BertvizHeadView"
      input_node: weave.Node[huggingface.ModelOutputAttention]

      @weave.op()
      def render(self) -> weave.panels.PanelHtml:
          html = bertviz_head_view(self.input_node)
          return weave.panels.PanelHtml(html)

And they can render other panels.

Panels can expose editable expressions (compositions of Weave ops) to the user, to give them control over what data transforms happen.

We call it Weave because it "Weaves a compute graph through the UI".

Weave includes a vectorizing DAG execution engine built on apache arrow, serialization and data versioning capabilities, and batteries Panels like Table and Plot.

We'll be writing and sharing a lot more now that the release is done. In the meantime, happy to answer any questions!

rahimnathwani• 3 years ago

Hi Shawn,

From yesterday's demo, it seemed to me that weave is a way to make easier the last few steps of aggregation/visualization, after you've done a bunch of your data joining and reshaping in pandas. (IIRC the starting point in the demo was a pandas dataframe.)

But, during a break in the conference, I was talking with a W&B employee (a weave user, but not a member of the weave team) who said he thinks weave will eventually replace pandas.

Is replacing pandas one of your goals for weave?

slewis2• 3 years ago

I like the ambition!

You can do some amount of iterative and visual data transformation in Weave now. But maybe like 10% of what you can do with Pandas.

Pandas is awesome and lingua franca in data science, unseating it would be an incredible feat if anyone ever pulls that off.

The core data structure in Weave, ArrowWeaveList, has some advantages over a dataframe. It can represent arbitrarily nested data, store references to other object types that can't be stored in arrow (like images), and be visualized in the Weave UI.

For now, we just want to make it really easy to go back and forth.

Weave: Interactive Data Exploration Toolkit by Weights and Biases

Related Stories