HN user
For a more informed opinion than folks on the internet, here's some work from Microsoft with early/internal access to gpt4: https://arxiv.org/abs/2303.12712 . I don't think people close by these systems share the same dumb parrot sentiment at all.
These projects are awesome to see, there are similar efforts for everquest. Is anyone aware of anyone trying to create different clients/renderers for these MMOs? A VR client for any of these worlds would instantly be amazing.
Just to further this, it's not just 'big names' that feel this way. Read this paper from a team at Microsoft Research: https://arxiv.org/abs/2303.12712 . These folks spent months studying properties of GPT-4, that paper is ~150 pages of examples probing the boundaries of the model's world understanding. There is obviously some emergent complexity arising from the training procedure.
This seems like (clearly?) bad journalism. RFID tags on their own are for sure not taking active sensor readings, that'd be left for something closer to TPMS style systems.
No, your OP is mistaken. The model weights have to all be accessed for the forward pass. What has happened is that using mmap changes where the memory is consumed (kernel vs process) and so it was being incorrectly interpreted. There are still 30B parameters, and you'll need that times however big your floating point representation is to use the model still.
Ahh, this would do it, thanks :).
That shouldn't be the case. 30B is a number that directly represents the size of the model, not the size of the other components.
I also have this question, yes it should be. The forward pass should require accessing all the weights AFAIK.
Does anyone know how/why this change decreases memory consumption (and isn't a bug in the inference code)?
From my understanding of the issue, mmap'ing the file is showing that inference is only accessing a fraction of the weight data.
Doesn't the forward pass necessitate accessing all the weights and not a fraction of them?
So can we at least take a second to appreciate the orders of magnitude of growth between BBS' and tiktok? With data structures and algorithms we're very willing to accept that orders of magnitudes shifting can change our basic assumptions about things.
The point isn't that these things haven't existed in some form forever, it's that they're scaling to points where these effects are becoming increasingly dangerous, proportional to the growth of these systems.
This thread is exposing one of the dirty realities of modern SEO. Most of the time, SEO folks will measure their success based on having some way of determining their ranking for a given search query.
The problem is that in 2022, google personalizes (aka re-ranks) the SERPs on enough dimensions (device type, location, user, etc) that a single mapping of search query -> ranking fails to capture any of the nuance that google is applying to their user understanding. I'm not sure how or when the SEO space will actually reckon with this, or if they'll keep just pushing poor 'visibility' metrics that can't actually be reduced to a single dimension.
It's mostly a curiosity. Occasionally things like this occur: https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-i... . TL;DR exploited an accidentally turing complete file format to execute instructions inside a virtual machine.
For big organizations that have the capacity and data, online advertising becomes a ROI optimization game, and one that they perform quite well at.
For a random business that wants to advertise online, without the infrastructure and data capability to back it, they will struggle to compete unless they exist in a segment full of similar peers. When the former happens, we see articles about how PPC doesn't actually work, etc.
Reality is that it takes engineering work and infrastructure, coupled with some data capabilities to unlock real value in the online advertising space.
As noted, online advertising brought all sorts of insight and visibility over traditional 'offline' marketing channels, but with that comes more savvy competitors that will do all the data things you're not.
Given the usual markers of attractiveness, wouldn't you expect age and weight to be strong predictors usually?
There's nothing 1960s about it, that's just not well reasoned. These computation constructs/tools just don't currently have better abstractions while maintaining the desired computational performance.
It's a strange intersection of needs where one wants or needs something like CUDA, but doesn't care to ensure their computation is actually running optimally. If you don't want to be bothered with control and granularity, why are you trying to write high performance CUDA code in the first place?
Would you mind elaborating as to what your hobby project was?
What? CUDA is intentionally granular and low level, why do you feel that is a negative thing at this level of abstraction? Are you suggesting the tools should be better so that doesn't have to be the case? I can't figure out what you're actually trying to express here.
You're not wrong that the dataset and compute are important, and if you browse the author's previous work, you'll see there are datasets available. The reproduction of DALL-E 2 required a dataset of similar size to the one imagen was trained on (see: https://arxiv.org/abs/2111.02114).
The harder part here will be getting access to the compute required, but again, the folks involved in this project have access to lots of resources (they've already trained models of this size). We'll likely see some trained checkpoints as soon as they're done converging.
To expand a bit for the grandparent, if you check out this authors other repos you'll notice they have a thing for implementing these papers (multiple DALLE-2 implementations for instance). You should expect to see an implementation there pretty quickly I'd guess.
Why does everyone assume this guy has zero business attempting this? If you read his credentials, he should be every bit as qualified as you to attempt this kind of work while understanding the pitfalls.
According to his CV he's been active in the field for quite some time. The default assumption that he's an idiot and going to kill people just seems too cynical here.
Grandparent - you specifically mention having noted methodology problems, would you mind sharing where in the methodology you think he's gone wrong?
So that's actually one of the awful implications of this paper. It's probably actually happening at a rate higher than would be noticed by humans.
If a given piece of silicon is hosing up a GEMM (matrix multiply), in graphics scenarios this may be invisible to the human eye as it could potentially just introduce artifacts in a scene rendering that could be entirely ephemeral to the frame.
In the case of crypto mining though, it's completely possible (probable?) that there are GPUs that can't possibly ever calculate a proper SHA3 hash (see the paper on AES instructions that fail in symmetric ways).
It's boggling to me that everyone is so entirely convinced of this fact. Check out this paper from google where they observe this exact process occurring in their CPUs at scale: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...
Given their conclusion that this is the result of manufacturing tiny process transistors, it follows that this can occur in GPUs as well.
Yes, but for different reasons. The grandparent is probably incorrect though, there is emerging evidence that silicon is actually changing/failing over time. See this paper from google on their CPU cores where they have practical evidence of this occuring: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...
If their data is correct, it should follow that these exact issues will happen on the small transistor process GPUs as well.
Actually there's reasonable evidence to believe this isn't true. This paper from google: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s... outlines specific scenarios where CPUs fail over time. Given the evidence that these are silicon defects that are actually worsening over time, there's no reason to imagine these failures don't extend to GPUs as well.
The difference in data here is obviously scale, google has -way- more CPUs than GPUs so the absolute counts of failures will be different.
Lua definitely caught a stray here. The reference VM is pretty fast, the JIT is just way faster. No idea how Lua ended up wrapped in the conversation with Python...
So if you want someone to answer precisely how they'll extract hundreds of millions of dollars from an emerging market, I have to imagine this isn't the correct forum to expect such answers.
Enabling the general software community access to SOTA language models will absolutely unlock an order of magnitude more money (than 100mm) over time. At least for now their obvious strategy for capturing this value is providing these APIs to enable it, and I suspect they'll gladly host such versions for the orgs that don't have the capacity to fine tune / host their own LLMs.
So this question can only come from a place where you have no idea what they do in their field. For every news article or arxiv post that you see talking about how this amazing new GPT-N model has broken all sorts of language benchmark scores, you'll notice that basically nobody can reproduce those results. That's mostly due to the barrier of entry with respect to hardware for training the models.
Huggingface is releasing APIs and model checkpoints that allow any random internet user to execute (almost) SOTA language models in production. FYI - that's an amazing leap forward and a strong piece of kit for MLEs to have access to.
So let me rephrase your question: Is general access to SOTA language models worth 100mm to the software market?
I suspect the answer is a resounding yes.
There’s this interesting dissonance where all the second year CS students and their professors agree it’s the wrong tool for the job yet the most successful products in the world did it anyway.
I guess you could interpret that to mean all these people building these products made a bad choice that succeeded despite using Python but I’d interpret it as another instance of Worse is Better. Just like Linus was told monolithic kernels were the wrong tool for the job but we’re all running Linux anyway.
This isn't the correct perspective or take away. The 'tool' for the job when you're talking about building/scaling a website changes over time as the business requirements shift. When you're trying to find market fit, iterating quickly using 'RAD' style tools is what you need to be doing. Once you've found that fit and you need to scale, those tools will need to be replaced by things that are capable of scaling accordingly.
Evaluating this binary right choice / wrong choice only makes sense when qualified with a point in time and or scale.
Hello whomever from clubhouse is promoting this blog post on the internet. Looking forward, please consider adding more technical depth to the content.
This is meant to be a technical blog post and the only technical description of the system/model pipeline involves telling us its a GBDT. That's unfortunately probably the least interesting thing you could pick out to say about such a recommender.
The content briefly touches on real time features and building a feature store. Tell us about that. Tell us about how you measured the improvement of adding near real time features. Explain how this stateless microservice scales horizontally. Tell us about tail latencies while ranking hundreds of rooms for a user. Just something.
Have you ever seen the OSINT where people figured out that the badges of certain military programs gave away the reason for their existence? https://www.nytimes.com/2008/04/01/world/americas/01iht-pent... , this feels like one of those moments...wormhole.com for cryptos, LOL.