razcle

https://www.metaculus.com/questions/5121/date-of-artificial-...

Ok I think I need to go into more depth on the examples.

I think HN knows that anyone can prompt LLMs. I do think its interesting though that this has allowed PMs/SMEs to direclty influence products that are deployed to millions of people. That seems genuinely novel. Maybe I over egged it

Hey Tombert,

wrt did you read the article? I was quite specific about the ways I think LLMs are blurring the lines. I don't think its true for general engineering but I do think its true for applications being built with LLMs.

Also its still very early

I think I'm just trying hard to be overly polite in the face of negative criticism and that sounds a lot like ChatGPT!

Nope :(

But I guess I need to up my game if you can't tell the difference

I agree with that. What do you think about the point thought that for LLM agents and applications, prompts and tool definitions might matter more than code?

Hi,

I totally agree that we're not at a point where AI can write most code. Though, I didn't ever say that. I just think its blurring the boundary between engineers and PMs with both taking on more of the others role.

Also, it shouldn't be surprising that the product we're building is aligned with what we believe about the world :)

R

Hi Hexator,

OP here. Thanks for the (harsh!) feedback, I'll take it in a growth mindset.

The post does genuinely reflect my experiences and I do believe what I said.How would you advise I change the post to make it better?

Which parts do you think are untrue?

Thanks!

Not the OP but can confirm that Humanloop has full support for OpenAI function calling.

I think I worded this poorly. What he said was that a lot of people say they want open-source models but they underestimate how hard it is to serve them well. So he wondered how much real benefit would come from open-sourcing them.

I think this is reasonable. Giving researchers access is great but for most small companies they're likely better off having a service provider manage inference for them rather than navigate the infra challenge.

Reading this atm. About half way through and already it's one of my favourite books. Would love to contribute to the notes if you're accepting PRs

Hi Raza here, one of the other co-founders.

I know that HN likes to nerd out over technical details so thought I’d share a bit more on how we aggregate the noisy labels to clean them up.

At the moment we use the great Skweak [1] open source library to do this. Skweak uses an HMM to infer the most likely unobserved label given the evidence of the votes from each of the labelling functions.

This whole strategy of first training a label model and then training a neural net was pioneered by Snorkel. We’ve used this approach for now but we actually think there are big opportunities for improvement.

We’re working on an end-to-end approach that de-noises the labelling function and trains the model at the same time. So far we’ve seen improvements on the standard benchmarks [2] and are planning to submit to Neurips.

R

[1]: Skweak package: https://github.com/NorskRegnesentral/skweak [2] Wrench benchmark: https://arxiv.org/abs/2109.11377

Humanloop | Infrastructure for AI | Backed by YC and Index | London + Remote Hiring

- Software Engineers: front-end specialist

- Machine Learning Engineer

- Interaction designer

(see jobs.humanloop.com for full details)

We're a team of ML researchers and Engineers who've worked at Google, Amazon and Microsoft research on some of the biggest ML projects out there.

ML and deep learning are a new software paradigm that needs new tools. We're building a platform for Human-in-the-loop ML that drastically reduces data needs and accelerates time to deployment. In the future people will program by teaching and curating datasets. (https://medium.com/@karpathy/software-2-0-a64152b37c35), we're making software 2.0 possible.

Team: humanloop.com/about

Contact the founders at founders@humanloop.com

Hi all, I wrote this piece and will be around for the next hour or two if anyone fancies a chat about GPT-3 and large scale language models!

No, not 100% accuracy. I've left out details for the sake of brevity but with a precision and recall high enough for the team to be able to answer the questions they cared about.

Maybe so but most data science workflows still don't acknowledge this "obvious" truth.

I very nearly said this myself!

I think the mistake of this quote is in the application of the expertise. The bitter lesson is that data + compute can outperform inductive biases but that doesn't mean you don't need domain expertise to get the right data.

I think this is one of those points that is obvious in retrospect but almost universally under appreciated.

Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks.

Perhaps it should be obvious but its definitely over looked in much of academic ML and in MLops.

We see ourselves as quite different to Scale really as we don't provide annotation services, mainly the software.

One of the main differences is that we've pretty exclusively focussed on language rather than vision which has quite a different tech stack.

We also view human-in-the-loop not just as a way to get better data but actually as a better deployment paradigm.

P.s You're right that David is awesome btw!

Hi Andy, thanks for the feedback on the site! We're actually redesigning at the moment so it should hopefully be fresher soon :P. Also great pointer to Rob Munroe's book. He actually used to be CTO at figure 8 before they were acquired.

You seem to be pretty clued up on the area, what do you see as the pros and cons of an end-to-end approach?

One of the makers of Humanloop here. We have recently started trialling our active learning tech with real customers and are excited to share the results. Happy to answer any questions!

Raza here (one of the founders of Humanloop). Just wanted to share that you can now train any Huggingface model on your own data just by labelling and curating a small portion of that data, with Humanloop. Humanloop provides a labelling interface with active learning and then hosts your trained model. We're still trialing the Hugging Face models in beta but would appreciate your thoughts and feedback.

Humanloop | Infrastructure for AI | Backed by YC and Index | London + Remote

Hiring

- Software Engineers: full stack and front-end

- Machine Learning Engineer

(see jobs.humanloop.com for full details)

We're a team of ML researchers and Engineers who've worked at Google, Amazon and Microsoft research on some of the biggest ML projects out there.

ML and deep learning are a new software paradigm that needs new tools. We're building a platform for Human-in-the-loop ML that drastically reduces data needs and accelerates time to deployment. In the future people will program by teaching and curating datasets. (https://medium.com/@karpathy/software-2-0-a64152b37c35), we're making software 2.0 possible.

Team: humanloop.com/about

Contact the founders at founders@humanloop.com

Raza here (one of the founders of Humanloop). Just wanted to share that you can now train any Huggingface model on your own data just by labelling and curating a small portion of that data, with Humanloop. Humanloop provides a labelling interface with active learning and then hosts your trained model.

We're still trialing the Hugging Face models in beta but would appreciate your thoughts and feedback.

Yeah I think is a good point. I'm actually planning to do a follow up post that is a case study with some real world data and the plots in that are much more like what you describe.

I may update the post. thanks!

TL;DR I think its complementary.

Synthetic data is particularly valuable when even the unlabelled data is expensive to obtain. For example if you want to train a driverless car, you may never see an ambulance driving at night in the rain even if you drive for thousands of miles. In that case, being able to synthesise data makes a lot of sense and we have lots of tools for computer graphics that make this easy.

Synthetic data can also be useful to share data when there are privacy concerns but my own feeling here is that there are better approaches to privacy preservation, like federated learning and learning via randomised response (https://arxiv.org/pdf/2001.04942.pdf).

In general though, outside of some vision applications, I'm pretty sceptical of synthetic data. For synthetic data to work well, you need a really good class conditional generator. E.g "generate a tweet that is a negative sentiment review" but if you have a sufficiently good model to do this, then you can probably use that model to solve your classification task anyway.

For most settings, I think synthetic data will work for data augmentation as a regulariser but will not be a substitute for all labelled data.

For the labelled data, active learning should still help.

So entropy based active learning methods are an example of pool based sampling. Even within pool based sampling there a few different techniques.

Entropy selection for pool based methods looks at the output probability for each prediction of the model in the unlabelled data-set. Then it calculates the entropy of the distributions. (in classification this is a bit like looking for the most uniform predictive distributions) and prioritises those.

Entropy based active learning works ok but doesnt distinguish uncertainty that comes from a lack of knowledge (epistemic uncertainty) from noise. Techniques like Bayesian Active Learning by disagreement can do better. :)

Great point Andrew. I was shooting for an easily digestible example rather than a realistic one.

Some examples that we've actually worked on/are working on: * Contract classification

* Content moderation

* NER

* Customer review understanding

* support ticket routing

Hi lwhsiao,

Raza here, author of the post.

My high-level answer is weak-labelling overcomes cold starts and active learning helps with the last mile.

More detail:

We see weak learning as very complementary to active learning. By using labelling functions, you can quickly overcome the cold start problem and also better leverage external resources like knowledge bases.

But most of the work in training ML systems often comes in getting the last few percentage points of performance. Going from good to good enough. This is where active learning can really shine because it guides you as to what data you really need to move model performance.

At Humanloop, we've started with active learning tools but are also doing a lot of work on weak labelling.

I'd also point out that people always focus on just the labelling savings from active learning but there are other benefits in practice too: 1) faster feedback on model performance during the annotation process and 2) Better engagement from the annotators as they can see the benefit of their work.