newhouseb

A more extreme version of the ADC/DAC projects mentioned here is my (old) project that built a Bluetooth transceiver using just an FPGA and an antenna (and a low pass filter if you don't want the FCC coming after you): https://github.com/newhouseb/onebitbt

Tl;dr: if you blast in/out the right 5GHz binary digital signal from a SERDES block on an FPGA connected to a short wire you can talk to your phone!

The short but unfulfilling answer is that this is because the DOCSIS standard historically has allocated a much broader frequency range for DL than UL. Unlike other forms of communication (like cell) that can use similar frequency ranges for transmit and receive, DOCSIS tends to slice something like < 70Mhz for UL and >70Mhz to 1Ghz for DL (I'm probably remembering the details incorrectly, trust Google over me!). Switching the frequency ranges often requires different circuitry and therefore different hardware.

I would guess that unlike fiber -- rarely saturated in a consumer context -- people have ~always wanted more than what cable can provide and thus the operators needed to be strategic about the allocation of bandwidth between DL and UL, hence the asymmetry.

Huge fan of RASP et al. If you enjoy this space, might be fun to take a glance at some of my work on HandCrafted Transformers [1] wherein I hand-pick the weights in a transformer model to do long-handed addition similar to how humans learn to do it in gradeshcool.

[1] https://colab.research.google.com/github/newhouseb/handcraft...

I'm a little skeptical of the paper in question: https://arxiv.org/pdf/2301.00250.pdf

If you look at the training and test set info:

We report results on two protocols: (1) Same layout: We train on the training set in all 16 spatial layouts, and test on remaining frames. Following [31], we randomly select 80% of the samples to be our training set, and the rest to be our testing set. The training and testing samples are different in the person’s location and pose, but share the same person’s identities and background. This is a reasonable assumption since the WiFi device is usually installed in a fixed location. (2) Different layout: We train on 15 spatial layouts and test on 1 unseen spatial layout. The unseen layout is in the classroom scenarios.

Depending on how they selected various frames -- let's just say it was random -- the model could have learned something to effect of "this RF pattern is most similar to these two other readings I'm familiar with" (from the surrounding frames) and can therefore just interpolate between the resulting poses associated with those RF patterns (that the model has compressed/memorized into trained weights).

If you look at the meshes between the image ground truth and the paper's results, you'll see that they are strikingly similar. I find this also suspect because WiFi-band RF interacts a lot more with water than with clothes and so you would expect the outline/mesh to get the "meat-bag" parts of you correct but not be able to guess the contours of baggy clothes. That is... unless it has memorized them from the training set.

Right, there are many folks (dozens of us!) yelling about logit processors and building them into various frameworks.

The mostly widely accessible form of this is probably BNF grammar biasing in llama.cpp: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...

You still charge through the USB-C / Thunderbolt ports (and use a USB-C to USB-C cable with the provided power brick to do so). The MagSafe plug is just for convenience (and perhaps slightly higher wattage than is normally specced for USB-C on the beefier MBPs).

An example from an earlier comment of mine on a different thread (assuming I've understood correctly):

let's say we had a grammar that had a key "healthy" with values "very_unhealthy" or "moderately_healthy." For broccoli, the LLM might intend to say "very_healthy" and choose "very" but then be pigeonholed into saying "very_unhealthy" because it's the only valid completion according to the grammar.

That said, you can use beam search to more or less solve this problem by evaluating the joint probability of all tokens in each branch of the grammar and picking the one with the highest probability (you might need some more nuance for free-form strings where the LLM can do whatever it wants and be "valid").

I think this is likely a consequence of a couple of factors:

1. Fancy token selection w/in batches (read: beam search) is probably fairly hard to implement at scale without a significant loss in GPU utilization. Normally you can batch up a bunch of parallel generations and just push them all through the LLM at once because every generated token (of similar prompt size + some padding perhaps) takes a predictable time. If you stick a parser in between every token that can take variable time then your batch is slowed by the most complex grammar of the bunch.

2. OpenAI appears to work under the thesis articulated in the Bitter Lesson [i] that more compute (either via fine-tuning or bigger models) is the least foolish way to achieve improved capabilities hence their approach of function-calling just being... a fine tuned model.

[i] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Also important to call out that anytime you have a freeform string it's pretty much an open invitation for the LLM to go completely haywire and run off into all sorts of weird tangents. So these methods are best used with other heuristics to bias sampling once you get to free-form text territory (i.e. a repetition penalty etc)

The way LLMs work is they output probabilities for every _token_, so you don't really need to backtrack you can just always pick a token that matches the provided grammar.

That said, you might want to do something like (backtracking) beam-search which uses various heuristics to simultaneously explore multiple different paths because the semantic information may not be front-loaded, i.e. let's say we had a grammar that had a key "healthy" with values "very_unhealthy" or "moderately_healthy." For broccoli, the LLM might intend to say "very_healthy" and choose "very" but then be pigeonholed into saying "very_unhealthy" because it's the only valid completion according to the grammar.

That said, there are a lot of shortcuts you can take to make this fairly efficient thanks to the autoregressive nature of (most modern) LLMs. You only need to regenerate / recompute from where you want to backtrack from.

I think you mean "run as part of the sampler," the tokenizer (and tokenization) is fixed for a given model. The sampler blob would basically:

1. Modify the output token probabilities to fit any arbitrary use case

2. Perhaps do trigger some sort of backtracking / beam-search

(I'm not Grant but we've chatted on twitter and built similar things)

I've been trying to wrap my head around this and my layman understanding is that there's an assumption (but maybe not baked into any requirements/standard) that use of the hardware key is locked behind either a biometric check (FaceID/TouchID/etc) or password. In other words, there might be an implicit second factor baked in to the passkey itself.

Modern LTE/5G provides pretty accurate location out of the box without GPS so if you really wanted to hide you would really need to just shut off the entire modem. Fortunately, the PinePhone has hardware switches to do this!

This is great! I built a house and DIY'ed this because our site was 2 hours away from where we were living at the time.

It was clear (our) GC was not used to this because they were constantly telling us that things were happening when they very clearly weren't (thanks to the live video we had from to the site).

AidKit | Remote | Full-time | https://aidkit.org | $130k-180k + equity | Social Impact / GovTech | TypeScript

AidKit runs the largest guaranteed income programs in the country. We replace convoluted workflows of glued together spreadsheets, bank portals and human tedium with one unified platform to allow organizations that do good, to do so faster and more effectively. AidKit is in many ways a technological petri dish to explore how we can bring the leverage of a bespoke software team to those without a software background through domain modeling and low-code tooling.

We're looking for an Engineering Manager to help AidKit's engineering organization grow from a tight knit team of folks always jumping in to do what it takes to one where everyone has the space and support needed to do their most leveraged work. Full post here: https://aidkit-inc.rippling-ats.com/job/637223/engineering-m...

We're bootstrapped and profitable.

If you're interested, please reach out to me at ben@aidkit.org!

But only for the whole generation. So if you want to constrain things one token at a time (as you would to force things to follow a grammar) you have to make fresh calls and only request one token which makes things more or less impractical if you want true guarantees. A few months ago I built this anyway to suss out how much more expensive it was [1]

[1] https://github.com/newhouseb/clownfish#so-how-do-i-use-this-...

Good point. Backtracking is certainly possible but it is probably tricky to parallelize at scale if you're trying to coalesce and slam through a bunch of concurrent (unrelated) requests with minimal pre-emption.

As someone also building constrained decoders against JSON [1], I was hopeful to see the same but I note the following from their documentation:

  The model can choose to call a function; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may generate invalid JSON or hallucinate parameters).

[1] https://github.com/newhouseb/clownfish

[2] https://platform.openai.com/docs/guides/gpt/function-calling

Yeah, they used to be called SmartyStreets and then dropped the Streets. Not the greatest branding call.

Quota management is indeed nonsensical. We serve large cash assistance programs that have stampedes of people applying all at once where we needed decent geocoding to determine eligibility. We were exceeding the default 50qps quota. I asked nicely (to double it, I think) and explained our use case and they said... no.

So... we left and switched to Smarty and haven't looked back since. We spend tens of thousands on geocoding annually. Really mind-boggling behavior from GCP.

Thinking about this more, softmax is also a form of normalization that could likely contribute to this phenomenon.

Oh good catch! The author defines a layer norm layer but then... comments it out in the actual implementation (I missed the fact that it was commented out). So that answers my second question of what happens without it.

Anecdotally in my own interpretability work (without layer norm), my models also learn rotations fairly frequently. I attributed this to the way I was doing positional embeddings (as rotations), but perhaps there's more to it.

The algorithm learned here actually makes a lot of sense when you spend more time understanding how transformers typically work.

Namely: once you include layer normalization your model is more or less forced to find ways to represent absolute quantities in a way that won't be normalized away and a great way to achieve this is to... store things as rotations of a unit tensor! With that as your primitive, it's fairly natural to rotate around a circle to compute modular addition.

I'd be curious to explore if a different algorithm is learned if one were to stop normalizing at various points. I wouldn't be surprised if a large hurdle to mechanistic interpretability turns out to be that the models have learned complicated rotations in non-obvious coordinate spaces that are tricky to identify after the fact.

Where do you draw the line? Is type checking in any domain harmful because it acts a crutch for your mental model of how your code works? One could similarly extrapolate this to any static analysis in any language.

Interesting, do you have any references or examples? Some brief googling around hasn't found anything like this. The fact that overloading was an issue makes me think that TF1 was doing something different because Typescript generic type parameters allow you to do "overloading" galore (by only specifying constraints rather than enumerating every possible call format).

It's a great question. I don't really have a horse in this race as long as whatever wins is maximally ergonomic. I think as long as the DSL is Turing complete such that you could "compute" on tensor shapes then we win. That said, it's very easy to build a type system that isn't so flexible (see most other languages) so I think it'd have to likely be a focus of the DSL from the get go.

We're talking about the streaming API which streams generated text token by token, not the normal one-shot API. I have no insider knowledge but would agree with your intuition on the normal API.

This is a great thread, thanks! Somehow I missed it when looking for prior art.

When I initially started implementing this I was hung up on similar concerns. For example in GPT2/PotatoGPT the MLP player is 4x the width of the residual stream. I went down a rabbit hole of addition and multiplication in Typescript types (the type system is Turing complete, so it's technically possible!) and after crashing my TS language server a bunch I switched tacticts.

Where I ended up was to use symbolic equivalence, which turned out to be more ergonomic anyway, i.e.

  type Multiply<A extends number, B extends number> = 
    number & { label: `${A} * ${B}` }
  const Multiply = <A extends number, B extends number>(a: A, b: B) => 
    a * b as Multiply<A, B>;

  tensor([
    params.EmbeddingDimensions, // This is a literal with known size
    Multiply(4, params.EmbeddingDimensions)] as const)

  Tensor<readonly [768, Multiply<4, 768>]>

  tensor([Var(tokens.length, 'Sequence Length'), 
          Multiply<4, Var(tokens.length, 'Sequence Length')>])

  Tensor<readonly [
     Var<'Sequence Length'>, 
     Multiply<4, Var<'Sequence Length'>>]> 

The downside to this approach is that typescript won't know that Multiply<4, Var<'A'>> is equivalent to Multiply<Var<'A'>, 4> but in practice I haven't found this to be a problem.

Finally, on more complicated operators/functions that compose dimensions from different variables Typescript is also very capable, albeit not the most ergonomic. You can check my code for matrix multiplication and Seb's writeup for another example of a zip function).

I'm excited about this for probably different reasons than most: I think Typescript could be a more ergonomic way to develop ML models than Python because you can automatically infer and check tensor dimensions while you are writing code! Compare this to the mess of comments you usually see writing pytorch telling you that x is of shape [x, y, z].

  // An empty 3x4 matrix
  const tensorA = tensor([3, 4])
  
  // An empty 4x5 matrix
  const tensorB = tensor([4, 5])

  const good = multiplyMatrix(tensorA, tensorB);
        ^
        Inferred type is Tensor<readonly [3, 5]>
  
  const bad = multiplyMatrix(tensorB, tensorA);
                             ^^^^^^^
                             Argument of type 'Tensor<readonly [4, 5]>' is not 
                             assignable to parameter of type '[never, "Differing 
                             types", 3 | 5]'.(2345)

[1] https://github.com/newhouseb/potatogpt

[2] https://sebinsua.com/type-safe-tensors

[3] https://t.co/gUzzTl4AAN

This is slick -- It's not explicitly documented anywhere but I hope OpenAI has the necessary callbacks to terminate generation when the API stream is killed rather than continuing in the background until another termination condition happens? I suppose one could check this via looking at API usage when a stream is killed early.