Cieric

Just in case anyone else decides to write along like the article suggests. I chose to use the live.lean-lang.org link, but it seems to default to a newer version of lean where the simp[xor] actually returns both sides still wrapped in the lambdas so the next simp[add_comm] will actually fail. The way to get around it is changing the version to v4.32.0 which the article doesn't seem to mention.

Yes, thanks for saying it more concisely/clearly than I did. Just cause I understand something doesn't mean I'm good at explaining it.

As someone who is in the rendering space for work. Having a higher framerate does help, but in a weird way. Basically the start of the frame rendering is what mostly dictates where objects are rendered. By getting a higher framerate the position of objects that you see in game are much closer to their "real" position. So it's less about seeing more frames at that point and more about seeing the most up to date information possible. Technically it could be possible to render the frame in sync with the framerate and just offset the rendering so it finishes right before it's pushed to the screen, but if you're slightly wrong you'll get really bad stuttering and the execution time of gpus and the cpu submitting the work isn't really deterministic.

Being fully stopped at a stop light isn't putting my or anyone else's life in danger. If I have a phone in my hand the car is not moving. I also finished the comment before the light turned green so the phone also was no longer in my hand when I needed to start moving.

Let me know if you want to hear anything specific about it. It kind of works, so it's not something I recommend doing if it can be avoided, but as Roxxik pointed out there is much room for improvement since this was just a naive just get it to run experiment.

On the technical details of mmap I agree (at least while single threaded which is how I believe I'm running it), but making it async does sound like an interesting method for speeding it up. My only goal with my testing was get as large of a model as possible running on a computer that "can't run it." I'm going to have to actually read through the code and figure out how it really works to really make any good optimizations, but as before this was just experiments with using and LLM (claude + codex) to just get it running. Since if they couldn't get it running I'm not sure if I would have wanted to spend time trying to get it working myself.

I also know I did some things that would actually make the perf worse to, like I believe I also had AI mmap the KV Cache to make sure to runs under any circumstance. For actual optimizations based on what I currently know, I'm probably going to try and get the llm running under my igpu on my laptop with persistent shader that has some kind of inbuilt request mechanism. That way the weights that are loaded can be used as fast as possible.

For the expert prediction, I assume I could use the medusa paper as kind of a kick off point for that since I'm already using it to try and predict the next 4 tokens. Doing verification on those 4 tokens is about as much as I can do though since it started to thrash on loading the experts. So some method of predicting even more tokens, but then batching together those with the same experts would probably yield slightly better results in this weird case.

Note: All of my tests have been around programming since that's the use case I'm interested in. I don't actually know if this would preform well in other cases (and anything more broad than that I assume would be slower.)

Yeah I'll see what I can transfer over from my llama.cpp work. As before I'm not too experienced with llm work, but I have a lot of experiments I'm trying out. So I'll make a PR if I get any interesting results.

I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.

To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)

In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.

Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.

[1] https://arxiv.org/abs/2401.10774

I think I can understand that reasoning. I do have a tendency to explore, especially at the start since it gives me a view of what the site can already do, cause I wouldn't want to run into issues with my own projects without knowing of the possible limitations ahead of time.

If anything since the keyboard shortcut already exists, you could always put it in little cursor menu at the bottom of the screen. But yeah up to you, I look forward to your progress.

Seems interesting an I'll take more of a peek after work, but one thing that stood out to me is the only way back to the home page after navigating to a repo is the back button. Going back to the home page via the back button also doesn't retain that "new" was selected. But I agree with others, I do like the simplicity of the site.

I can't say I know how they actually did it, but taking a look at the trailer I can point out that it looks like the spaces are confined and your character is on rails. I'm mainly going off of the instant direction changes that don't appear to be 45 degrees off from the camera direction. Once it's constrained down to a single line/path you could do some wild things like cube mapping a video, where the position in the video is tied to the characters position. I can't say I know how they would take that video though, my best guess there is the scenes are constructed in 3d software, just it was to expensive for real time rendering.

I actually do this whenever I add a youtube channel, but some shorts still sneak through. Just as a warning to whoever tries it themselves.

All good, and agreed the fork author is in the wrong here. I just like keeping the facts straight since that can inform better arguments and discussions.

An example give by donho is "SomeProject : a macOS port of Notepad++" so it seems like the name can be used which will make it appear in searches. It just has to be clearly something else.

The only place I see that is from a user "LiEnby" not "aletik", and none of aletik's existing messages are edited. All replies I see with the message are also to LiEnby. I don't agree with aletik's slow response in any way, but I don't think your claim is correct either. Do you have anything to prove that this was said by aletik?

Neither did I, feel free to email me or message me on discord and I can try to download the video and send it to you.

Here is a video that I actually just watched recently on that exact thing.

https://youtu.be/jfLAaGtNc7U

I've been interested in doing something similar in the past, but I could really never solve issues like domain squatting and stopping individuals from claiming every name possible. Do you have a place where you keep these plans or have discussions around it? Or even just a place where I could get updates if anything does come of it?

As someone from the US I would suggest viewing both as adversarial. I don't really trust my own government, but if I was born abroad I would trust them even less.

He's avoiding LLVM which is a compiler framework. Not LLMs as has been stated a few times in the comments already.

I don't have a lot of faith there, but that's mainly due to my experience being correcting peoples assumption that all gpus waves are 32 lanes. I might be biased there specifically since it's my job to fix those issues though.

Now that I could agree with, the only place where hiccups have started to occur are with wave intrinsics where you can share data between thread in a wave without halting execution. I'm not sure disallowing it would be the best idea as it cuts out possible optimizations, but outright allowing it without the user knowing the number of lanes can cause it's own problems. My job is the fun time of fixing issues in other peoples code related to all of this. I have no stakes in rust though, I'd rather write a custom spirv compiler.

I'm not even sure a 32 wide array would be good either since on AMD warps are 64 wide. I wouldn't go fully towards auto vectorization with though.

Door knob seems to return a bunch of chess pieces and analog sticks, I did get a few doors and a few knobs, but nothing I recognized as a door knob. I didn't spend a super long time looking though so that might have been the problem there. Also suggestion, giving the ability to link to specific models might be useful (I tried a few things on mobile, url doesn't change and searching for a specific model ID doesn't return the model).

From the last time I looked at these recompilation projects, they take the assembly and basically convert each opcode back into an llvm instruction and then recompiled from there. This comes with a lot of caveats though, last time I looked at it we still needed a function map of some kind to tell us where all the functions are and you still needed to replace chunks of the generated code afterwards for things like rendering, sound, controller input, and just about anything that would interact with the world outside of the cpu.

Edit: After some reading on the github page, it seems they are translating it to c++ instead of using llvm directly, but the original idea still holds. They aren't decompiling it to c++ that looks like original source code, it more like they're converting it to c++ that emulates the processor and gets statically compiled.

So it's not really just a drop and go replacement like it sounds like it'd be, but it has so far enabled the recompilation of multiple n64 games. This seems like an extension into the ps2 space.

Side note: The ps2 is a 32bit console with a 64bit alu (and some 128bit simd)[1]. So a lot of the weird tricks from the 8bit days weren't really used here. Not that there aren't weird tricks, just things like using undocumented instructions and racing the beam are less prevalent. I could be wrong here, I was growing up at this time not making games. All of this is just from research I've done in the past. Someone feel free to correct me if I'm wrong.

[1] https://en.wikipedia.org/wiki/PlayStation_2_technical_specif...

Well you could already do doom for about 6 months now [1]. I haven't tested the nvidia side, but it ran okay on my RX 7700S in my framework laptop.

[1] https://github.com/jhuber6/doomgeneric

I'm not even sure it's just instruction support that's the problem with the RV2. I bought one since I thought it would be cool to write a bare metal os for it (especially after I found the AI results to be so bad.) But the lack of documentation has been making it very hard to get anything actually up and running. The best I've got is compiling their custom u-boot and linux repos, and even those come with some problems.

I think in this case browser is meant as a place to browse, e.g. the Google Play store is an app browser. I don't hear it used that way often anymore, but it at least sounds familiar.

I'm not sure I'm gonna grab another OrangePi board again. I was happy to grab the RV2 just to experiment around with, but I didn't realize that the linux kernel they provided to build their ubuntu distro doesn't actually build properly. I got it to build after throwing a version of ubuntu onto an unused pc, but then it didn't matter the options I selected for the build (like gui options) it seemed like the gui just didn't exist at all in the final binary. I've yet to try and build a 3rd party os with support since I spent so much time just trying to get the official distro to work properly.

I think you mean MS/AMD since this is for hardware with AMD parts in it.