The Coming of Local LLMs

https://nickarner.com/notes/the-coming-of-local-llms-march-23-2023/

by yarapavan • 3 years ago

235 204 3 years ago

emrah• 3 years ago

It's great they got LLMs running on resource constrained devices but are they any good? Or I should ask, with the limited resources they get, what good are they for?

BulgarianIdiot3• 3 years ago

They’re crude but will be getting better quickly.

cgearhart• 3 years ago

Even with smaller models & more optimized hardware, I think edge compute is going to be power-limited first. Batteries today just won’t support constantly running LLMs. But I joked recently that as long as they prove useful then consumers would be willing to swap their iPhone for the old car battery with a phone handle attached.

tudorw• 3 years ago

I guess if the hardware is cheap for some uses speed is not so important, you can just walk away and let it grind.

kbrkbr3• 3 years ago

From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones).

Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like.

I see them as creativity and thought testing tools, also knowledge exploratory.

synergy20• 3 years ago

are we talking about training or inference for local LLM here? it's hard to do any meaningful training on the edge unless we all carry a heavy gaming pc, even that, the training quality will be subpar?

rubidium• 3 years ago

Is there a "getting started" guide you'd recommend to a newbie to the space?

wing-_-nuts2• 3 years ago

is there a difference in the quality of llm one would be able to train or run on a gpu with 8, 12, 16, all the way up to 24gb?

I'm trying to decide whether it's worth while to splurge on a more expensive 4090 vs a 4070 or whatever.

whimsicalism22• 3 years ago

If they are amazing, then ChatGPT must be God-like in your view.

I've been underwhelmed by Alpaca and Alpaca-LORA and LLaMA all at 13B but I have not tried higher params.

turmeric_root3• 3 years ago

I like using them for memeing

cs7024• 3 years ago

I expect we will see the biggest jump in performance if (when) consumer-grade coprocessors like mobile GPUs start incorporating attention layers as a primitive building block at the hardware level, e.g., with instructions and memory layouts engineered specifically to make ultra-low-precision (say, 4-bit) transformer layers as compute- and memory-efficient as possible on consumer devices. That seems almost inevitable to me.

liuliu• 3 years ago

Chip's capability planning seems need about ~2yr lead time. So we are expecting fastest would be somewhere around end of 2024. (Transformers probably earlier than that (2023?), 4-bit would be later).

cs7024• 3 years ago

Unless there are groups that have been thinking about and working on this for a while...

jjoonathan• 3 years ago

Low precision: agreed

Attention: Isn't it quadratic in context length? I dunno, this feels like the crude first iteration of something that will get inevitably passed by something that scales better.

dpflan• 3 years ago

Sharing a comment from a similar line of discussion:

"""

Complexity is quadratic in sequence length. For 512 tokens it is 262K, but for 4000 tokens it becomes 16M and goes OOM on a single GPU. We need about 100K-1M tokens to load whole books at once.

Since 2017 there have been hundreds of attempts to bring O(N^2) to O(N), but none of them replaced the vanilla attention yet in large models. They lose on accuracy. Maybe Flash attention has a shot (https://arxiv.org/abs/2205.14135).

"""

- source: https://news.ycombinator.com/item?id=34171503

cs7024• 3 years ago

Memory use is already linear in practice, thanks to FlashAttention. It's an open question whether computation can be made sub-quadratic without impacting model performance, although there are ongoing efforts seeking to do exactly that.[a]

Keep in mind: Once you go into precision as low as 4 bits (or lower?), all sorts of optimizations can become practical. Off the top of my head, maybe you could cache and reuse common attention sub-matrices (e.g., a 16×16 sub-matrix with 4-bit elements occupies only 16×16×4÷8=128 bytes of space)?

My sense is there's so much money at stake here, that whoever does this first will win big even if they end up having to replace or augment it with something better down the road. Hypothetical example: Imagine Intel or AMD coming out with a $1K or $2K card that has "built-in 4-bit attention," enabling you to run transformers of much greater scale on a run-of-the-mill desktop PC. I'd buy that in a heartbeat.

[a] Here's a recent post about a new approach from a group at Stanford that looks promising to me, although I don't fully understand all its details yet: https://news.ycombinator.com/item?id=35502187

throwawaymaths2• 3 years ago

In theory one could use the nysromformer. Don't know if anyone does in practice

sebzim4500• 3 years ago

Probably once context lengths get really long it might be better to use ANN rather than exact attention at inference time. I would imagine this would only pay off with ~100,000 token contexts, and even then it would only work if only a few tokens meaningfully contribute to each attention head.

whimsicalism22• 3 years ago

I agree - unfortunately we have gone so far down the rabbit hole optimizing transformer models.

Alternative models like S4 have been able to get transformer level performance with O(N) sequence length scaling.

seydor4• 3 years ago

maybe one could use physics to do that in analog. Or even better, in biology. I think with a clump of ~1.5kg of neurons we can have a pretty efficient coprocessor that is fed with pizza.

bick_nyers• 3 years ago

On the other hand, securing the biological coprocessor in your homelab and running queries on it 24/7 appears to have legal implications.

zamalek• 3 years ago

There are those Google accelerators that plug into an M.2 slot. You could plausibly do this today, although I am not sure what sort of memory constraints those accelerators have.

BrutalCoding2• 3 years ago

These ones can be plugged in with USB type-c, see: https://coral.ai/products/accelerator/

It’s used for boosting interference (offline) on Linux, Mac and Windows.

Haven’t bought or used them but I’ve had my eyes on these for a little while!

beauzero• 3 years ago

https://www.electronicsweekly.com/blogs/gadget-master/boards...

binkHN• 3 years ago

This is wonderful. As hardware and software continues to improve, everything seems to find a way to run on ever smaller devices. Guess your own pocket-AGI is not too far away after all.

18271622• 3 years ago

By the way, I was thinking of something along the lines of a powerful FPGA with direct access to large quantities of very fast NAND flash, likely many chips in parallel, which will save having to load the model into RAM..... So it will be able to directly run from NAND flash, which opens up the possibility of using very large models???

Power consumption would not be an issue if it's used sporadically throughout the day, it's not like it needs to run continuously?

There is still the issue of NAND flash read disturb, which I haven't fully looked into yet.

anentropic• 3 years ago

Apple should get working on a version of the Neural Engine that is useful for these models, and remove the 3GB size limit [1] to take full advantage of the 'unified' memory architecture. Game changer.

Waste of die space currently (on Macbook at least, I'm sure they find uses for it in the iPhone)

[1] https://github.com/smpanaro/more-ane-transformers/blob/main/...

leetharris• 3 years ago

It's not a waste on Mac, it will dynamically switch between GPU and NPU whenever CoreML is called. There are a decent amount of applications that use CoreML.

But I do agree it should be improved!

yacine_5• 3 years ago

I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters

4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make them much, much more productive. A lot of office work is clicking through sheets and doing manual labeling.

It's truly wild what is becoming accessible! Really excited to see the next gen software that the open community comes up with :)

tyingq• 3 years ago

Agree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters.

Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?

yacine_5• 3 years ago

More like: the accessibility is what will make it go around. The accessibility is what changes everything! Getting an easy to use interface (instructions) over python changes the accessibility from the denizens of this website to ~anyone with a computer

If you're interested, see this paper that argues that point: https://arxiv.org/abs/2302.06541

Essentially, being label efficient is more important than being compute efficient, because the biggest computing constraint we have is enough humans doing the labeling (and knowing how to work a jupyter notebook), not tensor smashing nvidia cards

londons_explore2• 3 years ago

I think the real benefits will be for those with an ad-hoc task and no programming/scripting ability.

Sure, you and I know how to write a little script to sort a directory of documents into "schoolwork" and "other stuff".

But most people don't have that ability, so giving them that would really help accessibility.

travisjungroth2• 3 years ago

Ease of use is huge, so is deployment. Even as a software engineer the overhead for a random classification is so much lower.

xbmcuser• 3 years ago

Ease of use is what is going to change everything. Using natural language to ask something and getting an answer is different from what we had before. I knew I could automate a lot of my paperwork with scripts but as I am not a programer I never gave it more than a cursory look. Last Dec while playing around with ChatGPT I was able to get it to write some python scripts that resulted in my spending less than 20-25 min on tasks that I was spending 20-25 hours on. Now could I have written the scripts myself probably but it would have taken me months whereas with chat gpt it took me 2-3 hours to get a working scripts and another 1-2 hours to optimise them.

rcme4• 3 years ago

LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.

gamegoblin4• 3 years ago

I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site.

Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

muyuu5• 3 years ago

spam and especially phishing will become much, much better

it will be hard to trust anything at all

nathan_compton2• 3 years ago

Like almost all machine learning stuff, I expect these things to only be useful in places where it doesn't really matter if the results are correct. When you apply a classifier in real life its critically important to understand its statistical characteristics which is typically done via model characterization, which involves cross validation or boot strapping or whatever. I think the idea that you can just zero-shot or few-shot deploy these things as classifiers and forget about it is incredibly naive unless (as I said earlier) the results just don't really matter.

I've already used LLMs in my work as a data scientist but it requires a ton of work to just make the results tractable (and I have been using GPT4, which behaves pretty well). These smaller language models ain't so regular. Ok, like consider a basic thing you want to do with a classifier: understand its behavior on a held out data set. Since no one knows what is really in the training data (since its so large), its quite hard to understand what the model can generalize about and what it has just accidentally memorized. CF reports that GPT4 doesn't perform nearly as well on even simple programming exercises that are chosen in such a way as to be sure they weren't in the training data.

There is enormous potential for statistical fuck ups here. Prompt engineering, for instance, is an easy place for over-fitting to happen as a prompt is fine tuned on data the prompt engineer has and thus fails to generalize to new data.

I do think there is a lot of value here, but I'm also sure that sloppy use of large language models is going to cause a bunch of trouble in the short to medium term, generate a lot of garbage, pollute a lot of databases, etc, while we figure all this stuff out.

czbond• 3 years ago

Can you explain what you did set up wise for your test? I'm following this "space" but the exact, simple pipelines are eluding me.

yacine_5• 3 years ago

fsreadfilesync and json

ticviking2• 3 years ago

The big thing for me and many others is the ability to use the tool without sending NDA data to a 3rd party.

The potential amplifying power of that is enormous.

syntaxing• 3 years ago

Super curious how you did this! Doesn’t 30B model require a hefty computer to run locally (assuming you’re tuning a non-quantized version)

drdaeman• 3 years ago

Not at all. Even a Raspberry Pi would do - you only need ~6GiB RAM for a 4-bit quantized LLaMA model (though it's gonna be quite slow). A decent modern desktop machine would do just fine, no need for anything extra fancy.

What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes.

yacine_5• 3 years ago

I'm running 30b quant on my 3090. that many 4bits fortunately fit into my precious vram

bob1029• 3 years ago

Sure, semantic text classifiers were possible for a while, but making it accessible changes everything.

Binary classification can actually take you all the way in terms of classification if you are clever with set theory. It's also one of the most traceable & deterministic ways to understand how the natural language is being interpreted at each step.

The amount of performance required to run something like an SVM is laughable compared to what is required to run even baby-tier LLMs. If you can reduce the cost of running models to a <1ms invocation over a few megabytes of black box, you can easily test thousands of these per-user-query. Re-training and iterating is much more enjoyable for these reasons. You also don't need any GPUs for this.

At the end of the day, the quality of your data will be the biggest issue with older techniques. LLMs can bandaid all sorts of weird things that crop up in the real world and aren't present in the training data. SVMs cannot tolerate requests delivered in the format of Shakespeare (if unexpected). In a well-controlled domain, you would probably be able to get away with much cheaper options that are also more flexible.

hospitalJail• 3 years ago

What model are you using? What program are you using?

Curious how you run the model then interface with it.

alden5• 3 years ago

What makes it so much better than normal text classification for me is it doesn't require tons of training data to accurately classify text. using it to parse craigslist posts which i might find interesting showed very promising results although it's fairly slow on my base m1 machine.

18271622• 3 years ago

I found this to be very liberating, that I can finally type whatever I want into the LLM, without the possibility of the government knowing what I am writing. Just being able to do that, and have the watchful eye of the state not being able to monitor you is amazing.

seydor4• 3 years ago

you have to check your screen's firmware for that

ticviking2• 3 years ago

I mean I have moderately high certainty that I'm escaping casual surveillance. It's not perfect but it doesn't need to be perfect to be good enough.

croes2• 3 years ago

Local LLMs remove the data protection problem but open the door for malicious use on a larger scale.

colonwqbang• 3 years ago

It was never possible to keep this technology secret for any length of time.

croes2• 3 years ago

I thought the possibility to run these LLMs on everyday hardware would be further on the future and in the beginning more limited to big servers.

But this could be the equivalent of the Low Orbit Ion Canon for phishers and scammers

sendfoods• 3 years ago

How realistic is CPU-only inference in the near future?

travisjungroth2• 3 years ago

It’s in the near past. https://github.com/ggerganov/llama.cpp

abetlen2• 3 years ago

Also worth checking out https://github.com/saharNooby/rwkv.cpp which is based on Georgi's library and offers support for the RWKV family of models which are Apache-2.0 licensed.

abetlen2• 3 years ago

You can see for yourself (assuming you have the model weights) https://github.com/abetlen/llama-cpp-python

I get around ~140 ms per token running a 13B parameter model on a thinkpad laptop with a 14 core Intel i7-9750 processor. Because it's CPU inference the initial prompt processing takes longer than on GPU so total latency is still higher than I'd like. I'm working on some caching solutions that should make this bareable for things like chat.

sharemywin• 3 years ago

I could certainly see them being trained/used as a front end for a database(file system, api calls).

layer82• 3 years ago

It would be nice to be able to run an LLM-driven spamassassin on a VPS for acceptable cost.

londons_explore2• 3 years ago

I don't think it will help. Actual friends occasionally send me mail that says "test" from a random account. And spammers do too... There is no way to seperate them.

layer82• 3 years ago

Personally I haven’t had that problem. I can readily distinguish spam from non-spam by manual review ~99% of the time. If I could train or instruct an LLM to do the same as I do now, I would be happy. My current false-negative rate with Bayesian spamassassin is more like 50%.

autoexec2• 3 years ago

Don't you do it somehow? Plus a filter doesn't have to be 100% right all the time. Filtering out what is 99% certain to be or not be spam and leaving the human to cover the tiny number of messages that fall into a grey area would still save a ton of time.

FL33TW00D• 3 years ago

I wrote a similar blog post in Nov 22: https://fleetwood.dev/posts/a-case-for-client-side-machine-l...

Torkel• 3 years ago

Things are moving so fast in this space I felt like I needed a [March] in the title on this one :)

turnsout2• 3 years ago

Too real–early March even felt different from late March. :D

monkeydust• 3 years ago

Wait till these are embedded into soft toys.

'You are a koala who plays with 5-7 year olds, you are friendly natured and curious and like to ask questions'

jjtheblunt• 3 years ago

Like Teddy from A.I. the movie?

https://www.youtube.com/watch?v=YRsICbxDEiI

waffletower• 3 years ago

While many NLP related Apple ML job listings have been added since this article was written, there were several recent listings at the time of its writing. While I feel that Apple does not focus well on intangible technologies, products that can't be readily carried, worn and given their boutique product development fetish focus, I have some hope that they can overcome this bias somewhat, and see how behind they are.

hackernewds• 3 years ago

I disagree with the assessment that Apple is behind. Apple is known for executing well and putting their weight behind the things they launch.

Like Jack Dorsey would often say "it's not important to be first to market, you can just be best to market". And the world got CashApp.

I'm sure however Apple enters the space, it will be fleshed out (vs Bard).

rootusrootus• 3 years ago

I want to agree, but it's pretty easy to find instances where Apple has dabbled but not delivered best-in-class solutions. Siri. iCloud. Home automation.

whimsicalism22• 3 years ago

Apple has very little ML expertise and very few cloud resources.

They are absolutely behind in the space and anyone who works in the industry will tell you that. Only feasible way IMO would have to be a very big budget acquisition of one of the major LLM startups, but most of those already have big tech backers.

0xDEF• 3 years ago

I know it's secret how GPT-4 actually works but are there any of these local LLMs that are also multimodal?

whimsicalism22• 3 years ago

Oranguru• 3 years ago

Yes, there are. For example: https://laion.ai/blog/open-flamingo/

jrm4• 3 years ago

As I see these things come out, it feels like there's not a lot of discussion on which hardware (that isn't one of the fancy new Macs?) As in, there might be a lot of graphics cards out there that could be used here? Is it only Nvidia still, is AMD a possibility? Maybe I'm missing something on how the tech works?

seydor4• 3 years ago

This has a list of models and their VRAM requirements

https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...

boppo14• 3 years ago

30B llama needs a 3090 or 4090. 13B I think you can get away with a 3/4080. If you have 64 gigs of ram and a beefy CPU you can run even 65B, but boy it's slow.

13B is pretty meh, but 30B is great, if not quite Chatgpt. But I can ask it why my highschool geometry teacher was such a cunt and it will happily discuss the matter without reservation. Very therapeutic.

yieldcrv• 3 years ago

So interesting how you can already tell that is a 3 week old epiphany

imaurer• 3 years ago

Tracking repos and resources for running LLMs locally here:

https://github.com/imaurer/awesome-decentralized-llm

turnsout2• 3 years ago

Also, this continuously-updated spreadsheet is extremely helpful: https://docs.google.com/spreadsheets/d/1O5KVQW1Hx5ZAkcg8AIRj...

ingenieroariel3• 3 years ago

In order to run large language models, we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.) and putting Linux on it to remove the artificial restrictions.

Yes, we will be running them soon in low end hardware, but we need to get at least to GPT-3.5-turbo level of inference speed and quality before we try to make it small.

I already started.

    $ neofetch
                   -`                    x@decpti 
                  .o+`                   ------- 
                 `ooo/                   OS: Arch Linux ARM aarch64 
                `+oooo:                  Host: Apple Mac Studio (M1 Ultra, 2022) 
               `+oooooo:                 Kernel: 6.1.0-rc6-asahi-4-1-ARCH 
               -+oooooo+:                Uptime: 4 hours, 23 mins 
             `/:-:++oooo+:               Packages: 177 (pacman) 
            `/++++/+++++++:              Shell: bash 5.1.16 
           `/++++++++++++++:             Resolution: 1920x1080 
          `/+++ooooooooooooo/`           Terminal: /dev/pts/0 
         ./ooosssso++osssssso+`          CPU: (20) @ 2.064GHz 
        .oossssso-````/ossssss+`         Memory: 717MiB / 129540MiB 
       -osssssso.      :ssssssso.
      :osssssss/        osssso+++.                               
     /ossssssss/        +ssssooo/-

simlevesque• 3 years ago

Please, this is not r/unixporn.

beiller• 3 years ago

Which LLM can run on apple's neural cores / GPU cores? I can only run on plain ol' CPU cores (llama), and it runs fine on my Ryzen CPU for less than half the price of that system. That being said I'm switching from Ubuntu to Arch cause I'm sick of all my packages being way out of date!

ingenieroariel3• 3 years ago

tinygrad

https://github.com/geohot/tinygrad/tree/master/accel/ane

But I have not tested it on Linux since Asahi has not yet added support.

Same machine but OSX, llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) on CPU using float16.

dns_snek• 3 years ago

Is this a joke or an ad? Paying Apple hardware premiums (for hardware that isn't well supported by PyTorch, et al.) just to load Linux onto it and run AI models in a closet somewhere, all seems like an incredibly wasteful way to go about it.

whimsicalism22• 3 years ago

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force?

I've posted this before, but it seems like this genre is just getting more and more popular - and more and more untethered from any actual metrics of how good these models are.

ingenieroariel3• 3 years ago

They are comparable to the first ChatGPT proof of concept from say early 2022. The reason many of us are excited is because we may be a year or two away from being able to run a ChatGPT if the open source models follow a similar curve.

whimsicalism22• 3 years ago

I have been using ChatGPT from the beginning :) ChatGPT did not have a PoC in early 2022.

They are not comparable with late 2022 ChatGPT.

simonw2• 3 years ago

I assume you mean GPT-3 - ChatGPT didn't launch until November 30th 2022.

simion3143• 3 years ago

People are enthusiastic about the possibilities, imagine a black box on your desk contains only RAM and matrix multiplication chips, you install on it your favorite AI assistant and you train it with your private data/code, you remove all prudish restrictions and get productive on your work and on your off times.

Llama has the potential to reach ChatGPT it needs tunning to get better at responding to questions, llama if I am not worng is mostly attempting to predict what is next.

I can see it similar like Midjourney and Stable diffusion, midjorny can make any stupid prompt look like a digiatal art in the style of Midjorney but look how many stable Diffusion innovation happens, a competent person that is on top with all the new stuff can produce absolute anything in any style they want.

whimsicalism22• 3 years ago

I am similarly enthusiastic and super, super impressed by Llama.cpp

But I just don't see the need to mislead and suggest that these models are "on par" with ChatGPT or something like that. They just aren't.

alchemist1e92• 3 years ago

My hunch is model weights will be commercialized as a purchased object. They can be watermarked so easy to trace any leak.

Then hardware will be a separate business. I think Apple might be caught off-guard by Nvidia on hardware. The latest NVlink and 400Gbps interconnects when combined with H100 next iterations and also rumors of advanced PCIe motherboards with high lane Nvidia CPUs and it looks to me that next year they can be selling $100-300K physical systems optimized for LLM inference that physically remind me of mainframes.

seydor4• 3 years ago

You make it sound as if ChatGPT itself isnt severly limited. Vicuna and newer 13B models are quite close . And the uncensored models have a capability that ChatGPT will never have.

whimsicalism22• 3 years ago

They just aren't quite close. Maybe you guys are asking shallow questions, but I'm asking subject matter questions and this is just not true.

It would be great if we could have detailed QA evaluation to show this, but of course then the open source people would train their models on it as a fine-tuning datasaet.

avereveard2• 3 years ago

What are the newer models? I am testing them in batteries across complex tasks, so far vicuña is the most flexible but they all choke on reflective instructions (I. E. Knowledge to extract is not in the model but in the user text)

causi2• 3 years ago

The way OpenAI has set their models up for prompting and follow-up is where their real advantage is. It's very hard to take a model you can run locally and just go "Hey, write me a powershell script to convert all these files into this format. Split them up so they're no more than an hour each. Ok, change it to also denoise them."

eikenberry• 3 years ago

I'm interested as LLMs are interesting tech but ChatGPT is proprietary and useless for free software.

gitfan86• 3 years ago

Because OpenAI or Google being the gate keepers is a show stopper for most serious people in the space.

They will be able to shutdown your startup on a whim. Even if they didn't politicians and regulators would be a huge risk. Without democratization we get blade runner . Not that democratization has no problems, just that it is the way a lot of us are wanting it to go.

whimsicalism22• 3 years ago

And basing your startup on a model pirated from Meta isn't risky?

I get it, I understand why people like decentralization - but the open source community doesn't even have close to the capability to train an actually open-source LLaMA equivalent.

d4rkp4ttern• 3 years ago

Agreed. I’ve said it elsewhere: how do we know that ClosedAI has fully published all their tricks used to build GPT3.* or later? The main “clues” people have are from the InstructGPT paper, but it should shock nobody if it turns out that paper reveals less than 10% of their techniques, which may have taken years to come up with. Competitors would need to rediscover those on their own or come up with new ideas. Simply repeating the ideas from that paper is likely not going to get them to a competitive model performance.

gigel82• 3 years ago

I don't understand why people are so excited to build this big thing on top of Llama, which is closed source, severely license restricted and we now know for a fact that Meta is going after users with the legal hammer.

I'm sure if we'd pool resources together we could build a truly open alternative worthy of building on top of.

whimsicalism22• 3 years ago

I'm sure if we'd pool resources together we could build a truly open alternative worthy of building on top of.

This would require like ConstitutionDAO level of resource pooling without direct monetary payoff.

I mean, good luck.

transfire• 3 years ago

We really need optical computing to take this to the next level.

agumonkey2• 3 years ago

Were transformers used in other contexts ? biochemistry ? geometry .. whatever.

skinner_• 3 years ago

Alphafold uses something they call Evoformer, it is an attention mechanism. Our group has tried, and so far failed to utilize transformers for a very very specific search problem in geometry (https://bit.ly/unit-distances).

agumonkey2• 3 years ago

thanks a lot man, very nice

txomon• 3 years ago

Out of doubt, which seems to be spreading around the internet. The LLaMa model weights weren't "leaked" AFAIK but rather explicitly given access to to researchers, isn't it right?

I know the article goes on to speak about something else, but I'm not sure why this claim that the LLaMa model weights were leaked, as in unintendenly made available is being done.

ftxbro• 3 years ago

My understanding is that researchers could ask for access to weights, but then also they were leaked so that anyone could get them without asking. There is another layer, where Facebook seems to accept it on some level (I mean they don't have a choice anymore anyway); they put a cheeky comment in the open pull request instead of closing it.

turmeric_root3• 3 years ago

The model weights were only shared by FB to people who applied for research access. Github repos containing links to the model weights have been taken down by FB.

simonw2• 3 years ago

"I think we’re going to eventually see a demo showing an open source model running on an iPhone as well"

I have Kevin Kwok's SheepyT running on my iPhone right now - it uses GPT-J, which is an openly licensed LLM by EleutherAI.

https://twitter.com/antimatter15/status/1644456371121954817

Sparkyte• 3 years ago

Everyone be like ChatGPT is everything. I am like it won't be the last or the first.

la64710• 3 years ago

One simple thing these LLM models cannot do yet .. that is to simply point a LLM to a URL and it will start scraping - ie follow the hyperlinks and start consuming the content. I am not an AI guy but I guess this has to do with the context limitations of most model? How did they train OpenAI with all internet data till 2021? This I think will be a most popular feature for LLM models and I seriously hope it is OSS whenever it comes out.

marijnz• 3 years ago

This already works with ChatGPT plugins: https://openai.com/blog/chatgpt-plugins

causi2• 3 years ago

LLAMA isn't there and probably never will be, but the possibility of running something equivalent to ChatGPT has certainly made me reconsider my GPU purchases. I wonder if in the end will it be Nvidia's CUDA advantage or AMD's larger amount of memory that will end up being more important when we do get it.

rambojohnson• 3 years ago

This service has been suspended by its owner.

The Coming of Local LLMs

Related Stories