slewis

Can you try a simpler less programmery version?

"are any of my recent blog drafts unpublished and nearly ready to go?"

OpenAI created a benchmark for this: https://openai.com/index/paperbench/

It would be really useful to see these evaluated across some of the same evals that the original R1 and deepseek's distills were evaluated on.

Hey, that's me!

Happy to answer any questions about how this works if folks are interested.

I've spent tons of time evaluating o1-preview on SWEBench-Verified.

For one, I speculate OpenAI is using a very basic agent harness to get the results they've published on SWEBench. I believe there is a fair amount of headroom to improve results above what they published, using the same models.

For two, some of the instances, even in SWEBench-Verified, require a bit of "going above and beyond" to get right. One example is an instance where the user states that a TypeError isn't properly handled. The developer who fixed it handled the TypeError but also handled a ValueError, and the golden test checks for both. I don't know how many instances fall in this category, but I suspect its more than on a simpler benchmark like MATH.

Is it stateful? Like can I do a run, read the results, and then do another run from that point?

This 100% matches my experience.

I like to jokingly call founder mode: "fine-grained multi-level oversight". Others might call it the derogatory "micromanagement".

That doesn't mean I control every decision, or that I don't give people space to be creative. What it means is: for whatever is most important for the business, I get involved with the details. The goal is that when I move out of that area, the team I worked with is able to operate closer to founder mode than when I started.

The issue is that vision fundamentally can't be communicated by telephone, or all at once. You're trying to get to a point on the map that most people can't see. The path to it is the integration of all of the tiny decisions everyone makes along the way.

If you only course correct from the highest level you'll never get there.

Creator describing how this works: https://youtu.be/PHQweR1z7pI?si=BpRlWxtxnRmYaeKG

Overplay one’s hand: spoil one's chance of success through excessive confidence in one's position

The memorization use case is brilliant. Put your talk track for a presentation in and say “help me memorize this by quizzing me”. Thanks!

I call this "keep the fingers moving".

Clicked the youtube link. An hour and 20 minutes later here I am. Thanks for sharing it that was awesome.

I like the ambition!

You can do some amount of iterative and visual data transformation in Weave now. But maybe like 10% of what you can do with Pandas.

Pandas is awesome and lingua franca in data science, unseating it would be an incredible feat if anyone ever pulls that off.

The core data structure in Weave, ArrowWeaveList, has some advantages over a dataframe. It can represent arbitrarily nested data, store references to other object types that can't be stored in arrow (like images), and be visualized in the Weave UI.

For now, we just want to make it really easy to go back and forth.

Hi! I'm Shawn, founder/CTO at Weights & Biases. We've been working on Weave for a couple years now, and it powers core parts of wandb.ai.

It's a UI toolkit built for programmers that can be reprogrammed from the UI itself. You might call it a "yes-code UI" :)

I'm very happy that we were able to release Weave under Apache2 yesterday, and there is a lot of cool new technology in here that we haven't had a chance to describe yet! I'll just try to give a quick tech summary for now.

There are three core components in Weave: Types, Ops, and Panels.

Ops are typed functions like:

  @weave.op()
  def flip_lr(im: Image.Image) -> Image.Image:
      return im.transpose(Image.Transpose.FLIP_LEFT_RIGHT)

  @weave.type()
  class BertvizHeadView(weave.Panel):
      id = "BertvizHeadView"
      input_node: weave.Node[huggingface.ModelOutputAttention]

      @weave.op()
      def render(self) -> weave.panels.PanelHtml:
          html = bertviz_head_view(self.input_node)
          return weave.panels.PanelHtml(html)

Panels can expose editable expressions (compositions of Weave ops) to the user, to give them control over what data transforms happen.

We call it Weave because it "Weaves a compute graph through the UI".

Weave includes a vectorizing DAG execution engine built on apache arrow, serialization and data versioning capabilities, and batteries Panels like Table and Plot.

We'll be writing and sharing a lot more now that the release is done. In the meantime, happy to answer any questions!

How dare you! This is a serious place. ^

Wow! What a joy to click on the comments and find a positive comment at the top. Thanks for writing this and thanks HN upvoters for expressing your gratitude.

It’s also surprising at first that infinite series can add up to a specific, finite number. If I go halfway to X, halfway again, and so on, where do I end up?

All next realities are weighted based on their probability given the current one. The probabilities are based on our knowledge of the equations of physics. Some realities are more likely than others and adding this all up results in an average outcome. At the smallest scales the average outcome is not actually what happens, but time is a great equalizer.

Finally! All you ungrateful users who expect your search engines to pore through billions of pages in milliseconds without complaining are going to have to show some humility for once.

The response here makes me think most commenters don’t have experience with this particular footgun.

To clarify: Python can gc your task before it starts, or during its execution if you don’t hold a ref to it yourself.

I can’t think of any scenario in which you’d want that behavior, and it is very surprising as a user.

Python should hold a ref to tasks until they are complete to prevent this. This also “feels right”, in that if I submit a task to a loop, I’d think the loop would have a ref to the task!

It’d be interesting to dig up the internal dev discussions to see why the “fix this” camp hasn’t won.

For one thing: if you’re going to make a post like this, put your contact info in your profile!

nbdev2, which this article is about, is a solution to this problem.

It makes notebooks testable, composable, versionable, and more.

“Everybody already knows that code running on a computer cannot become sentient,” said the temporarily coherent chemical blob.

Very cool! This might be a naive question. Are your arithmetic patterns equivalent to dependent types, like those found in Idris?

Yes, I can certainly see how this is confusing. We will work with our legal team to clarify the terms.

Thank you for responding!

I am certainly not a lawyer, however, 3b and 3c (from the terms link you posted) state that user content, including specifically Models, are property of the user.

Are you saying you think there is a conflict between 2 and 3b, 3c, or did you miss section 3?

``` 3. Intellectual Property & Subscriber Content

a. All right, title, and interest in and to the Services, the Platform, the Usage Data, the Aggregate Data, and the Customizations, including all modifications, improvements, adaptations, enhancements, or translations made thereto, and all proprietary rights therein, will be and remain the sole and exclusive property of us and our licensors.

b. All right, title, and interest in and to the Subscriber Content, including all modifications, improvements, adaptations, enhancements, or translations made thereto, and all proprietary rights therein, will be and remain Subscriber’s sole and exclusive property, other than rights granted to us to enable (i) Subscriber to process its data on the Platform, and (ii) us to aggregate and anonymize Subscriber Content solely to improve Subscriber's user experience.

c. Subscriber Content means any data, media, and other materials that Subscriber and its Authorized Users submit to the Platform pursuant to this Agreement, including, without limitation, all Models and Projects, and any and all reproductions, visualizations, analyses, automations, scales, and other reports output by the Platform based on such Models and Projects. ```

Oh, great! HN as bug resolution mechanism++.

Founder of Weights & Biases here (wandb). There are a couple issues raised in this thread: api key shouldn’t be required to download a public model, cache in home directory is annoying for this case. We will fix them.

Founder of Weights & Biases here (wandb). We don’t forbid anything, models are property of the people who created them.

Why do you think that?

EDIT: I'll edit respond, since you did. Look at sections 3b and 3c in the terms, they cover Models and other user content specifically. Those are user property, not our property. But I can see how this is confusing. We will clarify it.

Switching to the "latest tweets" view instead of the algorithmic "home" view helps.

This comment is a roller-coaster and I love it.