Live coding a vi for CP/M from scratch

http://cowlark.com/2019-06-28-cpm-vi/

by ingve • 7 years ago

320 67 7 years ago

The source is very straightforward and readable. I suspect part of the reason why is because live-coding forces you to eschew additional complexity/abstraction in favour of the simplest solution.

That said, I'm not sure if the additional complexity of a gap buffer is worth it --- a lot of editors in the early days of DOS used a single buffer, effectively memmove()'ing the tail part when you inserted characters, and there was no real noticeable lag even when entering characters at the beginning of a "large" (i.e. >100KB) file. On a CP/M system with 64K of total address space at most, it just doesn't take all that long to go through all of memory. As an advantage, cursor movement code is greatly simplified and doesn't involve modifying the buffers.

david-given8• 7 years ago

The 8086 could move memory _way_ faster than the 8080 could! Also the ACK's memmove/memcpy is truly dreadful. I need to replace it with hand-tooled assembler ASAP. It's tougher than it looks, though; the 8080 only has two register pairs once you've allocated one for the frame pointer...

It also doesn't help cursor movement as much as you'd think; editors don't address characters very much, instead preferring to address lines. For that you still need to scan forwards or backwards looking for line breaks. It would let you use memchr and memrchr to find them, though.

teddyh• 7 years ago

A description and discussion of different implementation strategies can be found here:

https://www.finseth.com/craft/#c6.4

ncmncm3• 7 years ago

As he says, TECO used a gap buffer in the early 60s, and he would like to know why authors of later editors made such bad choices.

My guess is that they took CS classes, and actually thought things taught there were good ideas and not desperate dodges for when the simple thing won't work.

kragen6• 7 years ago

I don't know the 8080 instruction set, but the memcpy in https://en.m.wikipedia.org/wiki/Intel_8080#Example_code has 7 instructions in its inner loop and copies 2 bytes per iteration, which is probably about 20 microseconds at 2MHz and thus 640 milliseconds to copy 64K. That's a really long time to handle a keystroke, since they may be coming in every 100 ms or less. You can probably unroll the loop productively to some extent but you probably can't speed it up by the necessary order of magnitude.

Gap buffers, by contrast, should keep everything under a millisecond except search and moving the gap a long distance.

Many gap-buffer editors don't move the gap whenever the cursor moves; they wait until you modify something. But that does seem like it would be more complicated. I'm not sure how much simpler it can get than the gap-buffer case.

tom_• 7 years ago

Even a Z80 would struggle to keep up: you're looking at something like 17-18 T-states/byte for a useful memcpy, which is ~215 KBytes/sec at 4 MHz.

Suppose your document is 32 KBytes, and you're inserting chars at the start. It's going to take 0.2 seconds per keypress! You'd notice.

(0.2 seconds per keypress = 300 keypresses per second. Assume 6 keypresses per word (5 letters and a space), and that's 50 wpm. You probably type faster than that.)

ohazi• 7 years ago

Nine straight hours. Holy crap. I'm sure we've all had marathon sessions from time to time, but maintaining enough composure to keep the livestream going for that long (without breaks?) sounds extremely challenging.

abvr• 7 years ago

Absolute madlad.

ur-whale• 7 years ago

Actually, it depends on your age.

ambicapter• 7 years ago

GeoHotz has a ten hour livestream of him coding up SLAM.

kazinator4• 7 years ago

Nine straight hours. Holy crap.

Day in the life of a full-time software dev.

saagarjha• 7 years ago

Do you not eat lunch or get up to go for walks?

closeparen2• 7 years ago

Six of those hours (on a good day) would be spent in meetings, answering emails, helping interns, giving code reviews for colleagues, writing documents, reviewing documents, getting lunch/snacks/coffee, waiting in line for the bathroom or elevator, and attempting to recover focus when you return to your desk after any of the above.

microwavecamera• 7 years ago

Minus someone or in this case everyone looking over your shoulder the entire time while doing something that's pretty much never been done before. I don't care where you work, this is impressive.

onion2k• 7 years ago

In my 22 or so years of writing software professionally I don't think there's ever been a day where I or anyone I've worked with actually coded for 9 straight hours. This is not typical.

slater2• 7 years ago

Speaking of live coding, does anyone remember a certain longhaired Russian guy (I think he works/worked for Yandex?) who used to do live-coding streams?

karlmcguire• 7 years ago

Bisqwit?

mindw0rk• 7 years ago

Bisqwit is neither russian (afaik he is from Finland) nor long-haired

protoloc• 7 years ago

Maybe you are referring to this guy? https://www.youtube.com/channel/UCJAVLOqT6Mgn_YD5lAxxkUA

slater2• 7 years ago

Yep, that was the guy! Seems to have stopped, alas.

snazz• 7 years ago

Recording yourself for YouTube has to help, but I’m really interested in what the author used to stay focused and productive for an entire nine hours. I can barely make it a half hour before getting distracted by something.

airstrike• 7 years ago

I wouldn't rule out Adderall, though to be clear I'm not suggesting others take it

cmarschner• 7 years ago

I think the secret is adding a livestream.

telotortium• 7 years ago

He did take a dinner and a tea break, so he wasn't working completely straight

david-given8• 7 years ago

There are a couple of short (a few seconds only) edits where I just zoned out thinking about something, but the big breaks are clearly labelled. Also, check out the clock at the bottom right corner!

I started this at about 14:00 and finished at about 01:30, so about eleven to twelve hours wall clock time. And yes, if I hadn't been doing the recording I'd have given up and gone to bed. Also I, er, had to get up at 07:00 the next morning to go hiking.

comboy2• 7 years ago

Do the thing that is more interesting to you than anything else.

fernandotakai• 7 years ago

for me, at least, even if something is one of the most interesting things ever (be it videogames, coding or even watching series), i just can't do 9h straight.

even for 2~3h it's hard to keep my concentration.

fsiefken• 7 years ago

But Why... I know the answer and the site gives a good explanation. But still, practically speaking? Who cares about CP/M, perhaps because it's the only useful OS running on 8 bit processors? Does CP/M have something DOS doesn't have? Or is it impossible to run DOS on the Z80?

soneil• 7 years ago

I think that is a large part of it, yes. There's been a good few kit computers in the last few years - the rc2014 being my favourite (not because it's particularly amazing, but because it's designed to be understandable and digestable), and cp/m is the most approachable (roughly dos-like) OS available for them. (There's also Fuzix, which seems like a monster accomplishment in itself - but I haven't tried it myself. I rather suspect that running a unix-like on an 8bit just ends up feeling like a very slow unix, rather than anything special)

(and no, msdos doesn't run on most 8bits. The Intel 8080 was 8bit. The Z80 expanded on that with additional instructions, and cp/m lives in this branch of the family tree. The 8086 gained 16bit instructions, and branches off in an entirely different direction birthing the x86 family tree, which is where you'll find msdos & most variants)

david-given8• 7 years ago

Fuzix is amazing. I did a port to the MSP430, a Texas Instruments single-board computer with 66kB of RAM; it feels just like a Unix, despite having no MMU and no multitasking... http://cowlark.com/2015-10-27-fuzix

I find CP/M interesting because it's provably the minimum viable product for an operating system. It was the personal computer desktop operating system that really worked. It came with full documentation, a full development systems (an assembler and a debugger), and a full set of tooling to allow you to port CP/M to new systems. It was portable, too, and you could run the same program unmodified on nearly any hardware which supported CP/M. So it was this great open-ended system, cheap enough to get into the hands of ordinary people, simple enough that you could understand it all, and yet sophisticated enough that, with enough time, you could achieve _anything_ on it...

The CP/M kernel is 3.5kB of 8080 machine code. I think each frame of that video consumed more space than that.

Zardoz842• 7 years ago

note : MS-DOS & clones only work on x86. However CP/M was ported to different CPUs, including 68000 and x86. The CP/M x86 later evolved to DR-DOS.

z3phyr• 7 years ago

Also, it is in line with the hacker spirit.

pmarin• 7 years ago

Weird question considering the title of this news forum.

UncleSlacky• 7 years ago

Probably a noob question, but why not just compile vi on a CP/M C compiler? After a quick search, I found this being discussed on comp.os.cpm just last year: https://groups.google.com/forum/#!topic/comp.os.cpm/lfuEr0hR...

It's hosted here: https://github.com/udo-munk/s

csb6• 7 years ago

I’m pretty sure they are doing this partly for educational/fun reasons

david-given8• 7 years ago

There are plenty of CP/M C compilers (I'm planning on including BDS C into cpmish at some point: https://en.wikipedia.org/wiki/BDS_C It's written in hand-tooled assembly!) but with the exception of Hitech C they're all Small C or K&R C. Hitech C works fine, but it's extremely slow and somewhat quirky. CP/M's really a bit small for an ANSI C compiler. (Also Hitech C only targets Z80, not 8080.)

The main reasons for doing this cross-compiled were: that's what I had; life was too short; I wanted something for the cpmish cross-compilation environment _anyway_.

Making cpmish self-hosted would be awesome, though.

Hitech C used to be a free-as-in-beer download from htsoft's website, but it vanished about twelve years ago. I wonder if they'd be open to open sourcing it...

Zardoz842• 7 years ago

HiTech not was bought by Microchip ?

cschep• 7 years ago

This type of content is why I come to Hacker News, brilliant!

dancek• 7 years ago

I don't like watching videos, but the list of milestones and timestamps is great. Text editors are one of those things that would be nice to know how to implement, but that never was high enough on my priority list. Looking at this, I feel like I've significantly overestimated how hard it is to write the UI layer. Granted, I'm going to be maybe much slower, but still this seems like a doable project even if I don't have that much time.

Why don't we have a list of nice hacker projects somewhere? With an estimate of skill and effort required, relative to other projects?

david-given8• 7 years ago

Hello, author here.

Eeeaggh. Yes, this took a while. I started after lunch and finished sometime after midnight (look at the clock in the bottom right hand corner of the screen). I'm still kicking myself at some of the dead ends I went through.

The result (even after I cleaned it up after the stream) is still a bit slow. The biggest issue is that it redraws the entire line every time you press a key in insert mode; fixing this is moderately straightforward, as I have all the information to know where the first modified character is, but it'll require some refactoring.

The biggest think I want next is a yank buffer, but the only realistic option is to store it on disk, which would be painfully slow. Being able to do repeated inserts would be nice, too.

I actually know Bram Moolenaar; maybe I should ask how he did it...

kragen6• 7 years ago

Bravo! You should be super proud! On a 9600 baud terminal, a whole line is 77 milliseconds, so it's going to feel a bit slow even if you only redraw half the line, and at 2400 baud it would be painful. Insert-character escape sequences are going to make a big difference.

I’m no expert, but I think the Kaypro doesn't have a real serial port between the terminal and the computer the way a lot of CP/M machines did (Altair and IMSAI usually, H-8 and H-88 and H-89 always, while the Osborne used a memory-mapped screen like the Apple). I don't know how fast it can repaint the screen.

I don't think firing the loud solenoids on the floppy drive and waiting a second or two every time you delete something (or (p)ut it) is going to be a good user experience; the yank buffer probably needs to be in RAM to be usable. You could maybe prompt the user whether to discard the deletion if the text being deleted is over a threshold of size.

rbanffy2• 7 years ago

Altairs and IMSAIs didn't have standard terminals. You could add an S-100 video board with video output and keyboard input (usually something that would output ASCII) or you could attach a real terminal over a serial line on an S-100 board (which is kind of nice if you want to be far from the computer but not that useful for a desktop). Many CP/M desktops used the "bus attached" terminal (because it saves components). A common arrangement (and not only for CP/M machines) would be a CRT controller managing its own memory where the program/OS would address the video memory through the microprocessor's IO channels. This was slow compared to having video driven from main memory (like the Apple II) but CRTCs did a lot the hardware designer wouldn't need to worry about, it was many orders of magnitude faster than a 9600 bps serial terminal, and allowed moving terminal emulation to software/BIOS.

The Apple II had a couple "terminal" boards more or less like what I described that allowed it to display 80 and 132-column video. Their memory and registers would be manipulated through memory access as the 6502 didn't have dedicated IO functions. One very common was the Videx Videoterm, that used a 6845 just like the IBM CGA and MDA boards (but with a much nicer VT-100-like font). The Microsoft CP/M SoftCard emulates a SOROQ IQ-120 terminal in software.

mistaken• 7 years ago

Another improvement: for large files the editor could read / write the file in chunks and maintain only the changes in memory.

catoc• 7 years ago

How I wish I had the time to do these kind of projects... I have a wife and two daughters - that I absolutely adore - but I so crave the times when I could sit down and spend 9 hours coding. So count me envious, of your skills and freedom. Good for you! :-)

zerr• 7 years ago

Do you think using a proper IDE and an intelligent code completion would make things bit faster?

kristianp• 7 years ago

This guy has some incredible hacks, such as the mandelbrot for BBC Micro (in 6502 assembly) that uses a lookup table for squares of numbers, and takes advantage of the connectedness of the colours:

"It turns out that Mandelbrot and Julia sets have the property that any area which is completely bounded by a colour contains only that colour. (Because it’s simply connected.) (No, I don’t understand a word of that paper.) So the program recursively traces out the bounds of boxes, and if all sides of the box are the same colour it fills it and doesn’t recurse in."

http://cowlark.com/2018-05-26-bogomandel/index.html

Also a bootloader for the GPU of the rasperry Pi: http://cowlark.com/piface/

david-given8• 7 years ago

Thanks very much! But someone else worked out the maths and the base kernel --- I just did the implementation (and loads of other people chimed in on microoptimisations). I _do_ think it may be the fastest 8-bit Mandelbrot in the world, although some of the stuff the C64 people have done is completely insane, so I'd want to check there first.

This is all mine, though: http://cowlark.com/2013-10-19-insane-make

ladybro• 7 years ago

Hey, I wrapped + annotated the video to make your timestamps clickable: https://use.mindstamp.io/video/QUqevzIz?notes=1

thanatropism• 7 years ago

Hi,

So — why not Wordgrinder?

Live coding a vi for CP/M from scratch

Related Stories