filterfiber

still run on a device like the iPhone, which weights about 170g [1]? The human brain is 8 times heavier.

Why are you comparing the weight? I'm sorry but this is a bizarre comparison. This isn't even apples to oranges, this is apples to a telephone pole.

I'll also throw in that a single Nvidia H100 is 1200 grams. Unless you have a "Bracket with screws" which will add 20 grams (who wouldn't want an extra 20grams of intelligence?).

Like 70%+ of the human brain is water. The human brain needs a massive network of systems to transport nutrients/oxygen which is irrelevant to logical processing.

Similarly the majority of the iphone weight is the battery and frame. The weight of the processing chip is _grams_.

Besides the conflicting variables with the weight, the way that ML works on a physical level is completely different from the human brain.

In their second sentence they have the most honest response I've seen so far at least: " averaged across 4 diverse customer tasks, fine-tunes based on our new model are _slightly_ stronger than GPT-4, as measured by GPT-4 itself."

Their timezone has been wrong in the tzdb for at least 10 years apparently

https://mm.icann.org/pipermail/tz/2023-December/033339.html

Does anyone know if mmWave could differentiate between my cat and I?

Does anyone know where I should look if I want to detect specific sounds? Like a smoke alarm, food bowl dispenser (its very distinct), cat meowing, 3d printer collision, that sort of thing?

Fun monetization strategy for federated apps - federate with your own instance dedicated to ads.

But more seriously what is the monetization strategy for federated apps? Up front pay or subscription for using the app?

Does anyone have any input on how this compares outside of benchmarks?

They don't appear to have any info on how they made it.

Realistically, even with Turbo+LCM, you're still going to 4+ steps (often 8+), with CFG, for reasonable one-generation quality anywhere close to the images people generated at 50+ steps without Turbo/LCM.

For sure the only reason I considered comparing it that way was because the linked repo appears to also be going for a similar approach with 1 step/image on the pi.

From my own experience I've had a hard ever getting a decent image below 6~8steps, but this repo seems more focused on getting it to run in a reasonable amount of time at all, which understandably requires the minimal "maybe passable" settings.

since you have stocks of the billion dollar message

I have no idea what you mean by this?

Are you saying I'm defending apple somehow?

Because my point is nearly every phone/laptop could pull off this attack, not just a single "special hacking device". Which I think is worse for them.

I don't know why the title even mentions the flipper.

The attack can be done from any device that can send crafted BLE packets including laptops/android phones, etc.

Apple just fixed a BLE DOS attack.

Better title: "Apple fixes BLE DOS attack".

I'm tired of media acting like the flipper is some kind of "super special hacking tool", it is very literally getting it banned in some places when all of it's internals are easy and common radios (Not to knock the flipper, it is conveniently well packaged).

You just needed to be able to send crafted BLE packets, this attack doesn't have anything specific to the flipper at all.

It didn't even originate on the flipper: https://github.com/ECTO-1A/AppleJuice

To run these scripts you need a Linux machine with an internal Bluetooth card or a USB Bluetooth adapter.

Versions also exist that run on the ESP32, android, etc.

I mean, if you need a human in the loop to verify the image quality then you HAVE to pre-compute the images.

100 images/s is likey too much volume

You can always generate less

I was just using that as a reference. Stable diffusion will run well with almost any relatively modern gpu.

You don't have to use a 4090, you'll still get double digit performance with a 3060 or whatnot.

for people who can only otherwise afford Raspberry Pis ;)

You can rent a 4090 for 0.7USD/1hr, or get an A100 for 1.1USD/hr. And if your project is a display + raspberry pi then those costs will dwarf the rental cost.

I found this claiming an A100 can generate 1 image/s.

The article you linked is over a year old. Needless to say there have been a LOT of optimizations in the last year.

Back then it was common to use 50+ steps for many of the common samplers. Current methods use a few steps like 1. This OnnxStream are using SDXL-turbo, and you can combine LCM and a few other methods to go very fast.

The reason it's so much faster now is the OnnxStream is only using a single step.

This repo claims 149 images/s on a 4090 https://github.com/aifartist/ArtSpew

However even if you only get 1 image/s with whatever GPU you have I stand by my original statement that unless you want to do it for the cool factor (which is very valid), pre-calculating them makes more sense.

But that's not the point, obviously.

If you want to say the zero2-w is what's making it then sure.

Besides, a 4090 costs more than a car.

They only cost ~0.70USD for 1 hr. In fact you could put this on an A100 for 1$/hr. Renting would make the most sense for this type of thing.

This project is a fun POC but it's not very practical for that type of application.

A 4090 can generate over 100 images a second with turbo+lcm and a few techniques, you can make 2 days worth of images in 1 seconds. You could make a years worth in roughly 3 minutes and put them on the sd card

which means even fewer people finetuning those models.

Finetunes rarely led to "Top 5 performance" for the small ones. Previously the top 10+ were all 70B, with maybe a few 30B in there. There were nearly no 13B's, let alone 7B.

The Zephyr-7b-β was one of the best 7B mistral 0.1 finetunes the past month and a half, and that didn't beat most 70B's.

Even at 7B there are few foundational models as even those take a relatively large amount of money. The only decent one for months has been 7B mistral which again didn't come that close to 70B performance.

I know the hugging face leaderboard isn't wildly accurate.

But the top models right now are almost all under 70B. Most are 7B, and the top is 10B. If the benchmarks are even remotely accurate then this is rather wild.

Apparently multiple groups found different "secret sauces", names upstage and whatever UNA is?

The current bottleneck for most current hardware is RAM capacity than memory bandwidth and last is FLOPS/TOPS.

The coral has 8 MB of SRAM which uh, won't fit the 2GB+ that nearly any decent LLM require even after being quantized.

LLMs are mostly memory and memory bandwidth limited right now.

AFAIK there's no public sdk for it, only a single third party game is in development and it's by the developers of garry's mod (one of the biggest third party source "1" games).

I'd consider it still proprietary, at that point you'd want to add Snowdrop (ubisoft), RAGE (Rockstar), REDengine (CDPR), Frostbite (EA), etc.

I don't understand why they're comparing the parameter sizes to lines of code.

AFAIK you can just increase the layer parameters of a 1B model to whatever you want? Like, the difference between a 1B and 175B model can be just changing a few numbers, and not adding any LOC at all?

LOC has never been a limitation for large models, it's been the compute+training data required.

Most of the LOC is spent on optimization, and they don't address MoE or anything fancy like that?

FWIW - I need to remeasure but - IIRC my system with a 4090 only uses ~500w (maybe up to 600w) during inference of LLMs, the LLMs have a lot harder time saturating the compute compared to stable diffusion I'm assuming because of the VRAM speed (and this is all on-card, nothing swapping from system memory). The 4090 itself only really used 300~400w most of the time because of this.

If you consider 600w for the entire system, that's only 6kWh/1M token, for me 6kWh @0.2USD/kWh is 1.2USD/1M tokens.

And that's without the power efficiency improvements that an H100 has over the 4090. So I think 2$/1M should be achievable once you combine the efficiencies of H100s+batching, etc. Since LLM's generally dwarf the network delay anyway, you could host in places like washington for dirt cheap prices (their residential prices are almost half of what I used for calculations)

So this specific model ships with a default config of 2 experts per token.

So you need roughly two loaded in memory per token. Roughly the speed and memory of a 13B per token.

Only issues is that's per-token. 2 experts are choosen per token, which means if they aren't the same ones as the last token, you need to load them into memory.

So yeah to not be disk limited you'd need roughly 8 times the memory and it would run at the speed of a 13B model.

~~~Note on quantization, iirc smaller models lose more performance when quantized vs larger models. So this would be the speed of a 4bit 13B model but with the penalty from a 4bit 7B model.~~~ Actually I have zero idea how quantization scales for MoE, I imagine it has the penalty I mentioned but that's pure speculation.

in as little as 16GB of RAM with room to spare.

I don't think that's the case, for full speed you still need (5B*8)/2+2~fewB overhead.

I think the experts chosen per-token? That means that yes you technically only need two in VRAM memory+router/overhead per token, but you'll have to constantly be loading in different experts unless you can fit them all, which would still be terrible for performance.

So you'll still be PCIE/RAM speed limited unless you can fit all of the experts into memory (or get really lucky and only need two experts).

Previous State-of-the-Art: [...] The number of parameters in the LSTM layers of these models vary from 2 million to 151 million.

We present model architectures in which a MoE with up to 137 billion parameters

Back in 2017 most models were well under 1B, GPT2 (2019) was one of the first "big" non-MOE models at 1.5B in size. People weren't sure how well/much they would scale.

The CoralAI TPU has a mere 8 MB of SRAM in 2019!

GPT3 was 175B in 2020.

Now nearly all LLM's are at minimum 1B, but dense 70B is now common.

After the Unity re-licensing fiasco, and OpenAI yanking old models,

is there any protections in place to allow the use of StableLM-Zephyr-3B indefinitely or will you be able to just deny any continuing access to models?

Refact has a decent 1.6B model that I think is better

https://huggingface.co/smallcloudai/Refact-1_6B-fim

No it's not (according to their benchmarks).

Zephyr-7B-B still beats it in most benchmarks but it's close.

This model is almost Zephyr-7B-B performance at 3B size which is a lot better for inference requirements.

Hardware: StableLM Zephyr 3B was trained on the Stability AI cluster across 8 nodes with 8 A100 80GBs GPUs for each nodes.

I might be missing it but do they say the number of training tokens that was used to train this?

This would help with efforts like TinyLlama in trying to figure out how well the scaling works with training tokens vs parameter size and challenging the chinchilla model.

My favorite with microsoft was just a year or two ago (not sure about now) - there was something like a 63 character limit for the login password.

Obviously they didn't tell me this, and of course they allowed me to set my password to it without complaining.

From why I could tell they just truncated it with no warning. Setting it below 60 characters worked no problem.