madisonmay

LLMs are not inherently non-deterministic during inference. I don't believe non-determinism implies lack of abstraction. Abstraction is simply hiding detail to manage complexity.

pypdfium2 is a great choice and a solid piece of software!

You might want to look into https://github.com/VikParuchuri/surya as an alternative to tesseract. Yes, it's associated with a commercial company, but as you long as you aren't a company with 5M in ARR or $5M in funding it's free to use.

This is an excellent use case for LLM fine-tuning, purely because of the ease of generating a massive dataset of input / output pairs from public C code

It's more like saying "I've upgraded to 128GB of RAM, I'll never use my disk again".

See figure-2

Why the decision to license as GPL?

Thanks, I hate it.

Coding aid for unittests. Debugging aid for languages / frameworks I'm not particularly familiar with. Work that requires reformatting. Translating from rough drafts to more polished / professional language. Learning more about domains I don't have much expertise in where I need specific conceptual questions answered.

Whether or not to split is more a measure of whether or not these two concepts are likely to split down the road than whether or not share similarity today.

Imperfect systems are still useful, and any sufficiently complex system is imperfect.

Interestingly it sounds like offloading could be made quite efficient in a batch setting if you primarily care about throughput rather than latency. Though I guess for most current LLM applications latency is quite important.

Often it might be viable to implement prediction w/o necessarily implementing training (especially if there are published weights or a reference implementation). Not viable for papers where the key contribution is a change to the pre-training objective / training methodology / optimizer, but useful for papers where the key contribution is architectural.

Guessing data security constraints -- I'm likely in a similar boat.

I'm getting a variety of CORS errors in console. Maybe this helps:

``` Access to XMLHttpRequest at 'https://api.doqs.dev/v1/organization' from origin 'https://app.doqs.dev' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. ```

Awesome idea, but website seems unstable. Wasn't able to login after sign-up :/

For more efficient einsum, see projects like https://optimized-einsum.readthedocs.io/en/stable/path_findi....

I suppose so, but perhaps trying to prevent the spaghettification has some positive benefits in terms of DRY + code structure.

Wojciech stated this pretty explicitly on his Gradient Dissent podcast a few months back.

Sweet! Looking forward to it.

@nloui any chance you're willing to share your dataset? Would be fun to replicate this with GPT-2 fine-tuning instead of a Markov chain.

It's also possible it's part of a docker build step or similar. Even if they're aren't downloading models at run time they may be loading s3 if their pytorch-transformers lib docker cache gets invalidated frequently.

This is a rather painful read and a relatively shallow article on CNN's vs. RNNs. Potentially useful for someone new to the field but it's a very grab bag take. The reality is much less well defined than CNNs = eyes, RNNs = ears and mouth. To start, CNNs have seen quite a bit of success in NLP applications.

I also very much like https://www.figure-eight.com/data-for-everyone/. It's not optimized for search but it's an excellent repository of high quality datasets.

Or as an alternative -- give them something legitimately beneficial to your company to work on, pay them standard consulting rates. This puts the impetus on the company to have fairly high certainty the candidate is qualified before the take-home, and the candidate gets compensated for their time even if a hire isn't made.

Well put. The NLP community seems to be slowly standardizing on language models trained on large unlabeled corpora as an analogue to pre-trained ImageNet models (ELMo, ULMFit, "Improving Language Understanding by Generative Pre-training"). The gradient publication does a good job of detailing this: https://thegradient.pub/nlp-imagenet/.

Recent research is finally checking off a few important boxes that are required for widespread applicability:

- Minimal configuration required

Aside from tweaking the language modeling loss coefficient language model finetuning seems to "just work". ULMFiT's approach also requires minimal configuration.

- Reasonable training times

You can finetune these transformer models on a few hundred examples in 10 minutes on a single GPU.

- Beneficial with very small amounts of labeled training data

This approach consistently beats out the use of pretrained word/document embeddings at ~200 training examples. Will be posting some benchmarks on two dozen classification tasks in the near future.

There are a few remaining conditions that I think need to be met before this kind of approach sees widespread use:

- Reasonable inference times

Inference is still rather slow because of model complexity.

- Reasonable memory consumption

Transfer learning is typically well suited to personalization tasks because of limited training data requirements, but large memory footprints mean that it's hard to swap out models for different users on the fly.

The critical thing to note is that although the initial language model takes a month to train on 8 GPUs, fine-tuning the language model for a specific task is much much cheaper (a few hours on a single GPU).

Initially embeddings of OOV terms are initialized to mean embeddings, and then the language model fine-tuning step allows the model to adjust these values.

See [18] here: https://github.com/fastai/fastai/blob/master/courses/dl2/imd....

Yes absolutely want to do comparisons to ELMo.

Perhaps even more interesting than comparison would be modifications to ULMFit to incorporate good ideas from the AllenNLP ELMo paper.

The learned weighting of representation layers seems like a decent candidate, as does giving the model flexibility to use something other than a concatenated [mean / max / last state] representation of final LSTM output layer (as is the case in some of ELMo's task models). I'm personally curious about using an attention mechanism in conjunction with something like ELMo's gamma task parameter (regularizer) for learning a weighted combination of outputs but haven't been able to get things to function well in practice.

The dataset the ELMo model is trained might also be preferable to WIKI 103 for practical English tasks, although you lose the nice multilingual benefits you get from working with WIKI 103.

In general it seems like the format described in the ELMo paper is simply not designed to work at very low N because the weights of the (often complex) task models used in ELMo's benchmarks are learned entirely for scratch. That's not possible without a decent amount of labeled training data.

Anyhow, thought the paper was very well put together, definitely an enjoyable read. Hope yourself and Sebastian collaborate on future papers, as good things certainly came of this one!

In Knuth's words: "Premature optimization is the root of all evil". Although I guess in this context, you're emphasizing that it's more valuable to solve the right problem poorly than it is to solve the wrong problem well.

Like most language, meaning is dependent on context. But in general it seems to be good practice to steer clear from pronouns associated with a particular gender when there are other perfectly good terms to be used (In this context, "people", "folks", or even "focused".)