luckyt

I went for NextAuth - the use case was relatively simple, and I wanted maximum control.

Clerk has quite a few dark patterns in their free tier, eg: if your app is on Clerk free tier, all your users will be forced to log out and re-login every 7 days (and they try to obfuscate this fact until you're locked in). For this reason, I've recently had to migrate away from them - I'm really glad there are alternatives.

There are several issues that make the KV cache as-is unsuitable for caching across requests. First, it requires the cached tokens to be in the exact same position in the sentence, this means it's mainly only useful for autoregressive generation where the prefix is always the same. Second, it is extremely big, so without some sort of compression, the cost to store it between requests and the time required to transfer the data to the GPU will outweigh any compute savings.

Yeah, I had a similar experience with Chroma DB. On paper, it checked all my boxes. But yea, it's alpha software with the first non-prerelease version only coming out in July 2023 (so it's 3 months old).

I ran into some dumb issues during install like the SQLite version being incorrect, and there wasn't much guidance on how to fix these problems, so gave up after struggling for a few hours. Switched to PGVector which was much simpler to setup. I hope Chroma DB improves, but I wouldn't recommend it for now.

It would be great to see more innovation like AI in DAW tools, but there are some challenges. The main constraint is it needs to process in real time, allowing just a few ms to process a sample. Very few neural methods can work with that constraint, without it, they can't fit into the standard DAW workflow where you string together many plugins, each processing the signal in real time.

There are some AI tools that work outside the main workflow, like for mastering after you're done with the DAW. But it's quite difficult to improve and bring new ideas beyond the typical signal processing modules without completely revamping the current workflow.

I had the same thought, but overall, there is probably an order of magnitude more people using LLMs in applications or fine-tuning them compared to those trying to pretrain LLMs from scratch.

I guess this goes to show how challenging it can be to implement transformer neural networks correctly. There are so many ways in which you can make mistakes at various steps, and there is no surefire way of knowing, you'll just have a slightly worse performance than you would've gotten otherwise. And in many cases, if you make a change to the network, either intentionally or not, the network adapts to it and there are many examples of different variants of the architecture performing similarly once trained. (though, in these cases, one might ask if it really matters if you match the original or not?)

One method I've seen people do to identify these types of mistakes is by precisely matching model outputs with a reference implementation. HuggingFace does this with tiny-random models: these models have randomized weights, but the output is expected to match exactly, if not, then it's an indicator of a bug. But this approach only works for bugs that arise during inference, detecting issues in data processing, optimizers, or anything that only happens during training is more challenging.

I hope we find a path to at least fine-tuning medium sized models for prices that aren't outrageous

It's not that bad; there are lots of things you can do with a hobbyist budget. For example, a consumer GPU with 12 or 24 GB VRAM costs $1000-2000 and can let you run many models and do fine-tuning on them. The next step up, for fine-tuning larger models, is to rent an instance on vast.ai or something similar for a few hours with a 4-8 GPU instance, which will set you back maybe $200—still within the range of a hobbyist budget. Many academic fine-tuning efforts, like Stanford Alpaca, cost a few hundred dollars to fine-tune. It's only when you want to pretrain a large language model from scratch that you need thousands of GPUs and millions in funding.

Yea, ONNX runtime is mostly used for inference. The requirements for training and inference differ quite a lot: training requires a library that can calculate gradients for the back propagation, loop over large datasets, split the model across multiple GPUs, etc. During inference you need to run a quantized version of the model on a specific target hardware, whether it be CPU, GPU, or mobile. So typically you will use one library for training, and convert it to a different library for deployment.

I found it helpful to start with CUDA on numba since it lets you write GPU kernels in python. Assuming you're like most ML engineers and you're more familiar with python than C++, this allows you to separately learn CUDA concepts from also learning C++ at the same time. There's also a set of GPU puzzles for beginners [1] using to get started with numba CUDA.

[1] https://github.com/srush/GPU-Puzzles

Yea, this was my experience too when I tried it out last week for my side project. It's easy to get started, but it's quite complex and disorganized and poorly documented. There are usually several ways to do things (which is by design, since it's meant to give you flexibility of either going with the default or customizing).

The main problem is the documentation is too disorganized, it's hard to figure out what even is the default and what are the configuration options, documentation is spread over a bunch of tutorials, reference pages, and blog posts by the founder. Sometimes the example code doesn't quite work because the library is changing so quickly.

We'll see if the community can figure out the best set of useful abstractions for this domain -- right now LlamaIndex is a mess and makes building things harder instead of easier and it's probably simpler to roll your own solution from scratch. However, the founders seem pretty smart, so hopefully with some time, they'll improve it and make it more usable.

Hmm, so is your conclusion that dog walking startups are not venture scale? But there are several dog walking startups that have received VC funding, and at least one that's IPO'ed. I agree with all your points that makes these types of gig economy businesses difficult to be profitable, but they similarly apply to many businesses like food delivery -- so this can't be right. (unless you believe none of these gig economy apps are successful, which some people argue, but is a bit of a stretch imo).

Yeah, it's not clear how to estimate the market size or TAM of an idea even if there already is a market, like take random examples of a dog walking or podcasting app. You can easily find sources that claim the dog walking or podcast business are $1 billion dollar markets. But how does that translate to evaluating growth in a startup in this business, and how big can it realistically grow?

I think the author's point broadly still holds -- you can get further with more engineering resources and data, whether you're using 2015 era models or 2023 retrieval-augmented LLMs and fine-tuning. Just that now you can accomplish a lot more quickly with a ChatGPT prompt.

There's definitely a lot of opportunities for technical debt in machine learning projects that don't exist in usual software development, which makes careful design decisions even more important. Reminds me of this paper, which talks about these issues and ways to avoid them: https://research.google.com/pubs/archive/43146.pdf

It's really interesting that XGBoost failed when ran on a dataset that had no noise. I've also seen a similar thing occur with the Adam optimizer when training neural networks on perfect synthetic data [1]. Always interesting to take a dive down and understand why this is happening -- it gives you a glimpse into the internals of your algorithms.

[1] https://datascience.stackexchange.com/questions/25024/strang...

Ah, I failed to consider that. The original post is correct.

This is incorrect: there's no reason to expect that X and Y will each appear 3 times in 6 trials if their probabilities are equal. If all 3 measurements of X are smaller than all 3 measurements of Y, then you have X < Y with confidence 1 - 1/8 or 87.5% confidence. You'd need at least 5 measurements to be 95% confident.

Yes, ideally you would carefully read the paper and evaluate its methodology, evaluations, and results, but this requires you to be knowledgeable in the field. That's why peer review exists -- some experts in the field read it and determine if it passes the bar, so you don't have to.

Citation count is useful for determining the relative importance of published papers. 10-100 citations is ok, 100+ citations means it made a significant contribution, 1000+ citations means it's a landmark paper in the field and probably worth reading. In the case of this paper, it's very new so citation count is not very meaningful, but it's a useful heuristic to evaluate most papers.

You're absolutely right. A good way to tell if a paper is legit is search it on Google Scholar and see how many citations it has: this paper has none. If it's an arXiv preprint (not in a peer-reviewed journal) and also a low number of citations, definitely take it with a grain of salt.

There's the Transit app, which tells you how long you have to wait for the next train. If more people used it, people would know about delays before entering the platform, and only take the train if they absolutely have to.

Ladders sound hard for neural networks. IIRC the original AlphaGo had a ladder-solver hardcoded that would determine if a ladder position is winning or losing and feed this bit into the neural network.

I disagree that CS deals only with discrete math. Machine learning makes heavy use of linear algebra, with a decent amount of calculus to model statistical phenomena. All of this uses continuous variables, not discrete.

Can I breed a dog that looks like the Metamask logo dog?

Well, unless you're doing original research, everything you write will be trivial to somebody.

Correct, there is only one branch

The day Uber goes public there will be a mass exodus of employees waiting for years to leave the company. I wonder if Uber will stay private permanently to prevent this?

Is it actually feasible to learn Japanese to a professional proficiency level as a foreigner? I've studied Japanese for several years, and can speak Japanese at an intermediate conversational level, but I'm still a long way off from communicating at a native level. So it seems like a tall order, especially given that you're busy with research and can't dedicate time to learning the language.

driver is not chatty: 0 minutes wasted

driver is chatty: 30 minutes wasted

Not the most social person, eh?

It doesn't seem to like the Sicilian Defense (1.e4 c5), which is the most popular opening by human players. I wonder if this will change opening theory?