abstractcontrol

Dumb people aren't going to be doing the work of the smart people ever, no matter how tech evolves. The actual implication of AI being on the level of sentient beings instead of hyped-up token predictors is that the era of humanity is over. If AIs were literally capable of doing what CEOs want them to, we'd have far bigger problems to deal with than employment.

Why don't they just pass the time into the RNG in order to randomize it instead of using fixed seeds?

Yes, this is what I mean. It's not that talent doesn't exist, but it's never the case that people on the top of their domain work less hard than others. In fact, it's the opposite. People don't have talent, the talent has them.

Probably one of the most ego-crushing realizations (if you're a nerd) is to discover that there are people out there MUCH more talented and higher performing than what you'll ever be, but with none of the obsession or pride. In other profession that's not really a topic. You can be a top performer in other professions, without a deep interest, clock out 4 daily, and never think about work outside work.

You could clock out, but I don't think the top performers ever stop thinking about work. Everything you've written here has to be wrong.

The job market is somewhat of a crapshot. A dice roll if you will. I got rejected from all kinds of place before I got a highly paid one from a Reddit post of all places. I got rejected from all kinds of places that only paid like 1/2 or 1/3 of what that job was paying, and I wasn't even looking for it at that time. It's been over for over half a year, and I wouldn't be able to replicate that sort of performance at will. Though at this point, I am not even looking for one and just working on my own projects since I have a lot saved up and barely spend anything.

One thing that I understand at my age of nearly 39 now is that success comes pretty hard, and luck plays a large role in it. As programmers all we can do is build and develop our skills, even if the world doesn't validate our efforts.

You're asking yourself what you are doing wrong, but you should be asking yourself what your goal is and focus on that. Is it really to just work for other people?

What can you do for yourself?

A stock market daytrading system, I am live coding it on my Youtube channel: https://www.youtube.com/playlist?list=PL04PGV4cTuIXoK6yBAFzh...

Opus has been amazingly useful at answering various statistics question that I had for it, and my current idea is a nested auction market theory inspired model. My biggest discovery is that replacing time with volume on the x axis (on a chart) and putting the bar duration on the bottom panel instead of volume normalizes the price movements and makes some of the profitable setups I've seen described in tape reading/price ladder trading courses actually visible on naked charts. A great insight I've gleamed is that variance should be proportional to volume instead of time or trade count. When plotted, it has the effect of expanding high volume areas, and compressing low volatility ones, which exposes trending price action much more readily. It honestly amazing, it's making me think that I could actually win at the trading game.

My impression is that LLM users are the kind of people that HATED that their questions on StackOverflow got closed because it was duplicated.

Lol, who doesn't hate that?

One takeaway from the book is that trend following strategies are really difficult to follow. Jesse Livermore had a 3 yearlong losing streak from 2011 - 2014 despite him following his rules. After the events of the book, he went short in 1929 and was reportedly worth over 100m in that time, a huge amount. Then he lost it all in the strongly mean reverting markets of the 1930s where his trend following strategy didn't work.

He was a problem gambler, but I think if we looked at top poker players of today, they'd all have some love the gamble in them. Jesse had godly tape reading skills that allowed him to beat the bucket shops at the start of his career.

After being kicked out of the bucket shops, he should have just become a floor trader and in all likelihood, he'd have had lower highs but would have fared a lot better overall. A lot of the trading cliches like cutting trading losses quickly, letting profits run, averaging up rather than down originate from this book. There is a reason people still talk about it 100 years after its publication. It's a good contender for the best trading book of all time.

Investment Strategy: Organizations should invest more in computing infrastructure than in complex algorithmic development.

Competitive Advantage: The winners in AI won’t be those with the cleverest algorithms, but those who can effectively harness the most compute power.

Career Focus: As AI engineers, our value lies not in crafting perfect algorithms but in building systems that can effectively leverage massive computational resources. That is a fundamental shift in mental models of how to build software.

I think the author has a fundamental misconception what making best use of computational resources requires. It's algorithms. His recommendation boils down to not do the one thing that would allow us to make the best use of computational resources.

His assumptions would only be correct if all the best algorithms were already known, which is clearly not the case at present.

Rich Sutton said something similar, but when he said it, he was thinking of old engineering intensive approaches, so it made sense in the context in which he said it and for the audience he directed it at. It was hardly groundbreaking either, the people whom he wrote the article for all thought the same thing already.

People like the author of this article don't understand the context and are taking his words as gospel. There is no reason not to think that there won't be different machine learning methods to supplant the current ones, and it's certain they won't be found by people who are convinced that algorithmic development is useless.

Can't find it either.

The strengths and weaknesses of the algorithmic niche that artificial NNs are in hasn't changed a bit since a decade ago. They are still bad at anything I'd want to actually use them for that you'd imagine actual AI would be good at. The only thing that has changed is people's perception. LLMs found a market fit, but if you notice, compared to last decade where we had Deepmind and OpenAI competing at actual AI in games like Go and Starcraft, they've pretty much given up on that in favor on hyping text predictors. For anybody in the field, it should be an obvious bubble.

Underneath it all, there is some hope that an innovation might come about to keep the wave going, and indeed, a new branch of ML being discovered could revolutionize AI and actually be worthy of the hype that LLMs have now, but that has nothing to do with the LLM craze.

It's cool that we have them, and I also appreciate what Stable Diffusion has brought to the world, but in terms of how much LLMs influenced me, they only shorted the time it takes for me to read the documentation.

I don't think that machines cannot be more intelligent than humans. I don't think that the fact that they use linear algebra and mathematical functions makes the computers inferior to humans. I just think that the current algorithms suck. I want better algos so we can have actual AI instead of this trash.

To me, the current LLMs aren't qualitatively different from the char RNNs that Karpathy showcased all the way back in 2015. They've gotten a lot more useful, but that is about it. Current LLMs will have as much to do with GAI as computer games have to do with NNs. Which is to say, games were necessary to develop GPUs which were then used to train NNs, and current LLMs are necessary to incentivize even more powerful hardware to come into existence, but there isn't much gratitude involved in that process.

I've thought about adding record row polymorphism to Spiral, but I am not familiar with it and couldn't figure out how to make it work well in the presence of generics.

Staged FP in Spiral: https://www.youtube.com/playlist?list=PL04PGV4cTuIVP50-B_1sc...

Some of the stuff in this playlist might be relevant to you, though it is mostly about programming GPUs in a functional language that compiles to Cuda. The author (me) sometimes works on the language during the video, either fixing bugs or adding new features.

What's a Net/60 basis? I am having trouble understanding how often you were paid. Every month or so?

Edit: Nwm, I saw you worked for 90 days without pay. Ack.

https://www.youtube.com/playlist?list=PL04PGV4cTuIVP50-B_1sc...

Staged Functional Programming In Spiral

I am doing a fully fused ML GPU library along with a poker game to run it on in my own programming language that I've worked on for many years. Currently, right at this very moment, I am trying to optimize compilation times along with register usage by doing more on the heap, so I am creating a reference counting Cuda backend for Spiral.

Both the ML library and the poker game are designed to run completely on GPU for the sake of getting large speedups.

Once I am done with this and have trained the agent, I'll test it out on play money sites, and if that doesn't get it eaten by the rake, with real money.

I am doing fairly sophisticated functional programming in the videos, the kind you could only do in the Spiral language. Many parts of the series involve me working and improving the language itself in F#.

Yes.

NNs for example are (mostly) a sequence of matrix multiplication operations, and GPUs are very good at those. Much better than CPUs. AI is hot at the moment, and Nvidia is producing the kind of hardware that can run large models efficiently which is why it's a 2 trillion-dollar company right now.

However, in the Spiral series, I aim to go beyond just making an ML library for running NN models and break new ground.

Newer GPUs actually support dynamic memory allocation, recursion, and the GPU threads have their own stacks, so you could in fact treat them as sequential devices and write games and simulators directly on them. I think once I finish the NL Holdem game, I'll be able to get over 100x fold improvements by running the whole program on the GPU versus the old approach of writing the sequential part on a CPU and only using the GPU to accelerate a NN model powering the computer agents.

I am not sure if this is a good answer, but this is how GPU programming would be helpful to me. It all comes down to performance.

The problem with programming them is that the program you are trying to speed up needs to be specially structured, so it utilizes the full capacity of the device.

Never tried those, so I couldn't say. I guess it would.

Even so, creating all the abstractions needed to implement even regular matrix multiplication in Spiral in a generic fashion took me two months, so I'd consider that good enough exercise.

You could do it a lot faster by specializing for specific matrix sizes, like in the Cuda examples repo by Nvidia, but then you'd miss the opportunity to do the tensor magic that I did in the playlist.

For a deep dive, maybe take a look at the Spiral matrix multiplication playlist: https://www.youtube.com/playlist?list=PL04PGV4cTuIWT_NXvvZsn...

I spent 2 months implementing a matmult kernel in Spiral and optimizing it.

Groq sells their dev kit for $20k even though a single LPU is useless.

I find this a very questionable business decision.

I have quite a lot of concurrency so I think my ideal hardware is a whole lot of little CPU cores with decent cache and matmul intrinsics

Back in 2015 I thought this would be the dominant model in 2022. I thought that the AI startups challenging Nvidia would be about that. Instead, they all targetted inference instead of programmability. I thought that a Tenstorrent hardware would be about what you are talking about - lots of tiny cores, local memory, message passing between them, AI/matmult intrinsics.

I've been hyped about Tenstorrent for a long time, but now that it is finally coming out with something, I can see that the Grayskulls are very overpriced. And if you look at the docs for their low-level kernel programming, you will see that Tensix cores can only have four registers, have no register spilling, and also don't support function calls. What would one be able to program with that?

It would have been interesting had the Grayskull cards been released in 2018. But in 2024 I have no idea what the company wants to do with them. It's over five years behind what I was expecting.

My expectations for how the AI hardware wave would unfold were fit for another world entirely. If this is the best the challengers can do, the most we can hope for is that they depress Nvidia's margins somewhat so we can buy its cards cheaper in the future. As we go towards the Singularity, I've gone from expecting revolutionary new hardware from AI startups to hoping Nvidia can keep making GPUs faster and more programmable.

Ironically, that latter thing is one trend that I missed, and going from Maxwell cards to the last generation, the GPUs have gained a lot in terms of how general purpose they are. The range of domains they can be used for is definitely going up as time goes on. I thought that AI chips would be necessary for this, and that GPUs would remain as toys, but it has been the other way around.

Considering the system only has a single H100, why would it be that performant?

BoM?

Yet many startups and existing designers anticipated this demand correctly, years in advance, and they are all still kinda struggling. Nvidia is massively supply constrained. AI customers would be buying up MI250s, CS-2s, IPUs, Tenstorrent accelerators, Gaudi 2s and so on en masse if they wanted to... But they are not, and its not going to get any easier once the supply catches up.

Can you order any of these devices online as a regular person? Anybody can order a $300 Nvidia GPU and program it. This is the reason why deep learning originated on the GPUs. Forget those other AI accelerators, even if you bought something like a consumer grade AMD GPU, you couldn't program it because it's restricted. The reason why Nvidia's competitors are struggling is because their hardware is either too expensive or hard to buy.

Thanks for posting this.

Right now, I am grappling with RSI and I haven't been able to program for more than a few days in the past month. I am not even typing this, but using the Voice Access feature of Windows 11 in order to input this. I ordered an ergonomic keyboard (Glove80) and I am waiting for it. I also have an ergonomic mouse and even got an ergonomic chair. But I know that regardless of the case, I won't be able to program. for at least a few months. Until my hand recovers. If it does at all.

I am definitely going to check out this extension. Quick question: does it work on any language or just something like Javascript?

It should be a big deal though.

If they can get the manufacturing process for them worked out. I really do wish memristors turned out to be a success, HP hyped them so massively in the 2011-2015 era. What happened was that the material they had was susceptible to rusting, so what seemed like a good initial yield would become unusable some months down the road.

I google for memristors sometimes, and all the activity regarding them is still confined to the lab unfortunately.

The PRs are really good but there's no way to talk to the actual developer working behind those PRs.

I'd really like an avenue to get into the US market as a remote worker, but am being unfairly treated by this job market. It is a pity as I am both a highly skilled programmer and have nearly a decade of experience. I'd consider this service if it could serve to showcase my skills, but if I am not going to get any credit for doing the work personally, there doesn't seem to be much point to it.

'Autism' when applied as a slur to everything and everyone has grown to mean 'talent.' It is very rare to see it referred to the actual medical condition these days.

https://deepai.org/publication/search-based-regular-expressi...

Here is the full text.

Regexps aren't even Turing complete as far as I know, if whatever they have in their paper works for arbitrary programs it would be shocking. I'll give it a read.

*Edit*: The algorithm in the paper is a DP like algorithm for building regexes. They use a matrix, and it has all the potential strings to be checked on one axis, and all the potential regex programs on the other axis, and in-between values (the actual matrix values) are booleans saying whether the string matches the program. The algorithm builds the matrix iteratively.

I haven't understood how regex evaluation is done, probably directly, but obviously this algorithm is only for checking whether a particular regex program matches an output rather than general purpose synthesis.

We'll have to wait for AI chips to really scale genetic programming, GPUs won't cut it.