Inside the Windows Console

https://blogs.msdn.microsoft.com/commandline/2018/07/10/windows-command-line-inside-the-windows-console/

by pmarin • 8 years ago

276 249 8 years ago

blogs.msdn.microsoft.com

The Windows Console code-base is currently (July 2018) almost 30 years old ... older, in fact, than the developers who now work on it!

Which really shows, especially since you still can't use a TTF-font without editing the registry.

flabbergast• 8 years ago

Yeah, this 30 years old code-base.. besides it is questionable only dev's below 30yr old are working on it (age discrimination, or accidental?), hardly anything changed. If I want to copy something I still have to do the ancient mark and copy sequence from the right click menu! What are they working on then? Still fixing bugs?

nitinreddy88• 8 years ago

Powershell. All of these shortcomings were fixed in it. Currently its also supported in Windows/*nix

ocdtrekkie4• 8 years ago

This is also not true as of Windows 10 build 16299/Fall Creator's Update (what I'm currently using), and may not be true of an earlier release or two. Ctrl+C and Ctrl+V both work just fine in cmd. I found evidence it was considered an "experimental feature" available in the Command Prompt's Properties dialogue back in 2015, it looks like it's been folded into the default behavior at this point.

Microsoft's done a lot of work on legacy components of Windows like cmd and notepad in the last few years, assumptions about behavior that has worked the same for the last twenty years may no longer be valid. ;)

zadjii17• 8 years ago

Have you tried enabling QuickEdit mode in the Properties (right click on the titlebar)? Quick edit lets you use the mouse for selecting, and in Insider's releases we've even added support for Ctrl+Shift+C/V for copy paste (even in wsl!).

We've been working on not only tons of bugs, but adding plenty of new features, including support for VT sequences, 256/RGB color, improving our unicode support, enhanced accessibility support.

The fact that all of our devs are under 30 is merely coincidence :)

pjmlp4• 8 years ago

What about learning to use the OS?

Quick edit for UNIX style copy-paste with the mouse goes all the way back to Windows 9X.

The only issue is that it was always disabled by default, until they changed to be enabled by default with the latest set of changes.

Spending a couple of seconds after logging in to enable it, for the very first time on a new system, was never a big deal.

zadjii17• 8 years ago

That's just flat out wrong: https://i.imgur.com/ae1yyXI.png

We added support for TTF fonts a few releases ago.

sun413• 8 years ago

Wow, so that's fixed after almost 30 years! Flat out amazing, what a team!

userbinator4• 8 years ago

...and by "few releases ago", you mean Windows 2000?

http://unixwiz.net/images/console-setup3.gif

(Personally, I don't care for TTF since I prefer the contrast and sharpness of a good pixel font, like this one: http://www.hassings.dk/lars/fonts.html )

thomastjeffery3• 8 years ago

All I know is that the last time I installed Windows 10, I still had to edit the registry for it to find DejaVu Sans Mono.

If that has changed recently, that's fantastic!

angelsl• 8 years ago

I'm not sure why the author is so excited! But it makes the article a bit difficult to read!

zadjii17• 8 years ago

That's Rich for you - He's pretty passionate about everything we do :)

Stratoscope• 8 years ago

I want to work with more people like Rich! Whatever they may lack in English convention, they more than make up for it in enthusiasm!

wila• 8 years ago

Thank you, love the series. Nothing wrong with the writing style.

Keep up the good work, it is great to see the console getting gradual improvements!

dcgudeman• 8 years ago

His excitement makes it hard for you to read? I guess next time he should try to come across bored when he describes his work.

bitcrazed24• 8 years ago

<img src="Eyore_looking_glum.jpg /> ;) :D

Traubenfuchs• 8 years ago

Yes, it's surprisingly interruptive. I kept going back rereading a sentence, wondering about what makes it so special.

gmueckl3• 8 years ago

So they have been working on uograding this since 2014? So far, the visible results seem a bit underwhelming for the invested time and effort. How big is the team? What am I missing here?

zadjii17• 8 years ago

Well, the team is small, we're up to three whole devs now. Since 2014 we've been doing a lot of work o help bring our VT implementation up to par (to help enable WSL), as well as implementing new features such as 256/RGB color support, improved accessibility features, and we're even releasing a pty-like API for Windows.

Not only that, but since the team was founded we discovered that conhost was not the only console implementation in Windows, though it's the only one available on Desktop. There was a pretty massive undertaking to unify all the separate console implementations on different SKUs under one application, so that all of them would be improved at the same time. It didn't have a terrible lot of impact on the everyday developer, but it was important to help internal engineering efforts.

gmueckl3• 8 years ago

Thank you for explaining! Please don't take my initial post as criticism of your work. There is just this visible disconnect between approximately 4 years of work and comparatively simple features that have been shown off so far. So there must be a story there.

I was not aware of Windows having had different console implementations in the past. Can you elaborate on that point? How did that come to pass?

Also,I am curious about one other thing: the Windows Console is notorious for having extremely bad perfomance when displaying lots of text. Last time I checked I could only get around 100 lines of text per second on Windows while Linux terminal emulations can show more than 10 times that amount. This may seem petty, but some programs are slowed down by this in practice. Can we expect improvements in that area as well?

Karliss• 8 years ago

Pty-like API sounds great. I assume you are referring to what's discussed here https://github.com/Microsoft/console/issues/57 .

komali2• 8 years ago

Are you able to speak at all to the labor assignment? Microsoft is big, why only put 3 engineers on this project?

slededit2• 8 years ago

He can't say it publicly but the code behind the console is crap. Not just from a modern perspective looking back, the original devs hated it too. The book Show Stopper! Has some colorful words about it.

And since it's so core to the system and deeply intertwined you can't just go in and start arbitrarily changing things. Everyone has been too scared to touch it for the last 20 years.

This is painstaking careful work. I don't think throwing more people at it would really help too much.

My hat is off to you and your team for finally fixing this.

russellbeattie• 8 years ago

I don't use Windows mostly because the console experience is so bad. I'm sure there are many others like me. Your superiors should realize they have lost generations of developers because of this and dedicate the resources needed to fix it. How they haven't realized this yet, astounds me.

erikbye6• 8 years ago

Could you fix proper emitting of tab escape codes so completion in REPLs, etc., work properly?

icc973• 8 years ago

Please ignore the parent you replied to. The changes you've put into the console are so fundamentally important - and brilliant. Being able to snap a window and have the code flow properly and fill the window completely is great.

In the same way that the Linux desktop has fast caught up with the Windows desktop, the Windows terminal is fast catching up with the Linux terminal.

The stuff your changing most people will never notice, but you've done more for the console in 3 years than was done in the previous 20 as far as I can tell.

It's getting very close to where I can actually use Vim full time at a DOS prompt - it's actually waiting for Vim to catch up with the 24-bit colour changes that you made [0]. Plus there's the nicety of bold/italic fonts but I'd imagine that might not occur for a while.

[0] https://github.com/vim/vim/issues/1270

mschaef• 8 years ago

Thanks for your work on this... this is going to make a big difference for lots of folks. (Myself included.)

dingo_bat• 8 years ago

Actually the cmd now is vastly improved over the one in win7. I cannot go back to that after using the latest one. The changes are definitely not underwhelming.

oldgradstudent3• 8 years ago

The article seems to ignore the other big problem with the Windows console. It is painfully slow compare to Linux and Mac terminal programs. Especially when scrolling large amounts of text.

izacus2• 8 years ago

Is that true for other UIs like conemu?

oldgradstudent3• 8 years ago

Good question. The article says that

Alas, the story here is not a good one: There ARE some great 3rd party Consoles (and server apps) for Windows (e.g. ConEmu/Cmder, Console2/ConsoleZ, Hyper, Visual Studio Code, OpenSSH, etc.), but they have to jump through extraordinary hoops to act like a normal Console would!
For example, 3rd party Consoles have to launch a Command-Line app off-screen at, for example, (-32000,-32000). They then have to send keystrokes to the off-screen Console, and screen-scrape the off-screen Console's text contents and re-draw them on their own UI!!

I'd guess that the third party emulators might perform worse, but someone need to test it empirically.

GordonS2• 8 years ago

I regularly use both Windows and Linux, and I can't say that I've ever noticed a performance problem?

ShinTakuya• 8 years ago

The terminals you use in Linux must either be painfully slow or you must be very insensitive to latency issues because the latency on Windows console is really high. When I type something I expect it to appear immediately (<10ms), not in 20-50ms. I'm just guessing the timings here though but the point is that there's definitely a difference.

A lot of basic commands in general run more slowly on Windows but I don't think that's the terminal's fault so much as it's lack of optimisation on the command itself.

iforgotpassword2• 8 years ago

Write a simple program that outputs 10k lines of text, then compile and run on Linux and Windows. It's night and day.

Const-me• 8 years ago

I’ve just run “tree /f” from the root of my drive. It printed huge amounts of text but the scrolling remained instant.

I think your big problem is not necessarily with Windows console, could be with the command-line apps you’re using in it.

mwcampbell• 8 years ago

Are you using a high-DPI display? In my experience, scrolling in the console is much slower on a high-DPI display.

oldgradstudent3• 8 years ago

Yes, but I use HiDPI displays on Linux and macOS as well.

zadjii17• 8 years ago

I would agree that our perf isn't the best. Almost all of that comes down to our rendering stack - something like 90% of our CPU time is spent rendering the window. This is because at the moment we're stuck with GDI, and IIRC we have to kernel-thunk every row of text we render. It's not a great situation.

Stay tuned for updates in this area though. Improving this experience is one of our top priorities.

delfinom• 8 years ago

GDI?!

Throw that bad boy into the abandoned WPF framework and hardware accelerate it! (Without the sandbox shengigans of UWP in the way)

zerr• 8 years ago

The Windows Console code-base is currently (July 2018) almost 30 years old ... older, in fact, than the developers who now work on it!

It smells like ageism... Did you lay off the original "old" devs?

EDIT: :)

jhall1468• 8 years ago

I'm hoping this is a joke. Most developers don't stay at the same company on the same team for 30 years...

zadjii17• 8 years ago

Lol any original devs retired a long time ago, can you imagine what $MSFT shares from 1989 are worth now?

drblast• 8 years ago

Don't even have to imagine.

  NASDAQ: MSFT
  $104.40

sun413• 8 years ago

What the heck is this related to shares? You mean to suggest all original devs are millionaire, and that being a millionaire means you don't code anymore? And you, poor guy, missed out?

skookumchuck2• 8 years ago

With the recent surge in MSFT, it must be pretty gratifying for the stock options.

PeCaN• 8 years ago

Surely they have something better for developers with 30 years of experience to do than work on Windows Console.

It's, uh, not the only thing Microsoft works on.

m_st• 8 years ago

I tried using PowerShell or Hyper. But these are just so slow to start. So I'm always coming back to cmd.exe. It is fast and serves my basic needs just perfect.

When I need more, I just type 'bash'. And bash also starts faster than PowerShell or Hyper.

So I don't see why they now change WIN+X and Explorer to use PowerShell rather than cmd.exe.

Keep improving cmd.exe. But please don't make it bloated.

ilikehurdles• 8 years ago

Do you have something custom going on in powershell? I’m not a power user of windows shells but it opens very quickly for me. So much so that I never thought about the speed of either terminal.

userbinator4• 8 years ago

"very quickly" is subjective, but compared to cmd it is noticeably slower even in a fresh install of Windows --- the few times I'm forced to actually use it to configure some things.

nailer3• 8 years ago

Powershell 5 is slow.

Powershell 6 preview is much faster (not as fast as bash but does more)

nailer3• 8 years ago

PS you can download it here

https://github.com/PowerShell/PowerShell/releases

electricEmu• 8 years ago

I'm not sure how often you start and stop your shells, but start time is seldom a concern.

I enjoy you chose Hyper.js, which uses electron. I'm not sure what speed you were expecting there. You might try ConEmu https://conemu.github.io/.

dagw• 8 years ago

I love ConEmu, but it's still pretty slow starting up

oblio9• 8 years ago

cmd.exe is not getting any improvements, for better or for worse.

And Powershell Core (Powershell 6) is quite snappy. Try it out. They changed the name to pwsh.exe, you can have them in parallel in your PATH.

philo233• 8 years ago

There's actually an option in Settings to swap between Powershell and cmd.exe in the Win + X menu.

On top of that you can actually go in and customise it quite a bit to your own liking as it's just a folder inside your local app data: C:\Users\YourWindowsUsername\AppData\Local\Microsoft\Windows\WinX

shawnz• 8 years ago

Unfortunately it has no effect on the menu in Windows Explorer which also dropped cmd for PowerShell recently.

AnIdiotOnTheNet3• 8 years ago

I think your experience is atypical. I use powershell everyday and I've not noticed any startup time difference between it and cmd. Are you sure you don't have some gigantic import in your profile.ps1? Are you launching the really terrible and definitely slow as balls to launch Powershell IDE?

Frankly, given how superior powershell is as a language (both to cmd and bash) it is beyond me why anyone would consider a small startup cost difference a dealbreaker anyway.

Macha• 8 years ago

I've got no powershell customizations, and an i5-4670k. I've also experienced Powershell having a very noticeably slow startup. It's about on par with my heavily modified zsh setup on a 2015 MBP for startup time (the same zsh setup starts up much faster on the same desktop).

It's not a dealbreaker, but it's definitely an annoyance.

gobio2• 8 years ago

I think it's pretty typical: powershell is slower to start up than cmd. On my machine, i7, 16GB RAM, Windows 10, the fastest is cmd (instantaneous), then powershell (1s) and bash WSL (1s), then supper slow default install of git-bash (12s!). I cannot use git-bash because of that.

tootie• 8 years ago

I just keep one or two shells open all day.

bitcrazed24• 8 years ago

Hi there! Author of the post that's the subject of this thread here :)

Two questions:

1. Does your machine use a spinning disk for its system drive, or SSD? If the former, that's slowing you down more any anything else. STRONGLY recommend moving to SSD system drive if so. 2. Have you tried `powershell /noprofile` from Run dialog or Cmd? If you're still seeing PowerShell take > 600ms to load and start, likely something else is up.

If you're expecting PowerShell to load faster than ~600ms, your expectations are likely a little out of whack - Cmd loads so fast because it's tiny and - relatively speaking - as dumb as a stump. PowerShell is a .NET language, and so loads the CLR, so incurs some overhead vs. Cmd, but it does A TON more that'll save you more time and effort over-all than any amount of time saving vs. starting Cmd.

HTH.

bitcrazed24• 8 years ago

Oh … and re. Hyper - its awesome and beautiful, but it's an Electron app which means you're first loading and starting Electron base, then Chromium's JS & HTML engines, then loading A TON of JavaScript which has to be tokenized, parsed, compiled and executed every time you run it.

This doesn't come cheap. Electron consumes & fills ~100MB working set just to get rolling, plus several more tens of MB to load, compile and execute.

For example, on my 16GB Core-i7 Surface Pro 4, starting a single Hyper instance connected to Cmd, requires 134MB. Starting Console takes < 1s, and consumes 6MB.

romwell2• 8 years ago

Well, re:

1. Does your machine use a spinning disk for its system drive, or SSD?

This really shouldn't be a question regarding the performance of a terminal window.

If cmd.exe starts fast regardless, then so should Powershell if it is going to replace it.

And on that note, on a new laptop with an SSD I can see a difference between Powershell and cmd.exe execution-to-prompt-ready time (however small that latency is).

As long as that's the case, Powershell simply isn't a viable replacement for cmd.exe

tempodox• 8 years ago

So, MS has finally realized they're 40 years behind the civilized world. I'm curious to see how long it will take them to catch up.

0xFFFF0000• 8 years ago

Not sure what you mean... PowerShell Console is one of the most modern and advanced consoles there is and it's from Microsoft (compared to old non object oriented and pipelined stuff like cmd here in the article) - but it's still good to see that they haven't forgotten about the good old cmd.exe :)

ygra8• 8 years ago

You're mistaking the shell for the console. As the article quite clearly points out, there's a vast difference between both, even though when you look at a console application (such as a shell) you kind of always see both.

gaius• 8 years ago

So, MS has finally realized they're 40 years behind the civilized world

Yes they should be emulating a teletype like enlightened folk!

oblio9• 8 years ago

You mean you don't actually print your output?!? Heathen!

ziotom78• 8 years ago

Very appropriate comment. Last year I was tinkering with the implementation of a TUI (Text User Interface), and I wanted it to be cross-platform. I started developing it under Windows, using the Console API (sorely lacking in features, but at least it is just an API to be called from C). I had to abandon the project once I realized that it would have been too hard to harmonize this approach with the weird way you are supposed to program a Unix console.

Let's consider the difference between calling an hypotetical system API to change the text color and a raw sequence of bytes encoding. In Unix, to set the color to be used for writing characters you have to send an ESC character, some more bytes, and then an ASCII representation of the R, G, and B components (assuming your virtual terminal supports this), separated by a colon/semicolon. The terminal will then read this sequence, parse the three numbers and move the character. This approach is not precisely efficient, especially if you consider that today it is not unlikely to have full-screen terminal applications with a large resolution. For a 200x60 terminal like the one I am using right now, to color every character with a different color requires sending 500 Kb of data, compared to the 36 Kb required by a 3-byte representation of each RGB tuples.

And if you are trying to read from a terminal, every time you receive an ESC key, you are always in doubt whether the user pressed ESC, or if it is the start of an ANSI sequence. The usual trick here is to check if there is a character already available immediately after you detect an ESC byte coming from standard input: if there is, you can assume it's the beginning of a sequence.

Admittedly the virtual-terminal approach makes remote connections (through SSH) simpler than an API (you just have to send a bunch of bytes through a socket), but I wonder why in 2018 we are still relying on the old and quirky mechanism of "virtual consoles" when real consoles have long disappeared.

Edit: small fixes.

JdeBP4• 8 years ago

More specifically, a TeleType Model 37 from 1967. None of this modern ECMA-48 stuff from 1976. (-:

* https://news.ycombinator.com/item?id=17557575

* https://news.ycombinator.com/item?id=15779194

chungy2• 8 years ago

Technical errors regarding how UTF encodings work:

* UTF-8 is at maximum four bytes per codepoint (which aren't characters, since multiple codepoints can make up a single character... see combining diaretics or ZWJ emoji things). "additional 1-4 bytes" should be "additional 1-3 bytes" (Historically, UTF-8 did support 5- and 6-byte sequences, and if you read up on it, the implementation is obvious, but that was before Unicode settled on U+10FFFF[1] being the maximal codepoint for backwards compatibility reasons with UCS-2...)

* It says UTF-16 extends UCS-2 (which is correct) by incorporating a 4-bit encoding... which probably was intended to be a "4-byte encoding" instead. The surrogate pairs reserve their first six bits for an additional 10 bits making up half of the codepoint being stored (whose value is later added to by 0x10000 to get the actual codepoint). Also, it provides 16 additional planes to the BMP. 17 total including the BMP.

[1] Actually, U+10FFFD is the maximum legal codepoint, the last two codepoints on every plane are reserved and declared as noncharacters. Few parsers are really so strict.

jwilk• 8 years ago

Actually, U+10FFFD is the maximum legal codepoint, the last two codepoints on every plane are reserved and declared as noncharacters. Few parsers are really so strict.

This is what Unicode 11.0, §2.4 says:

Noncharacter code points are reserved for internal use, such as for sentinel values. They have well-formed representations in Unicode encoding forms and survive conversions between encoding forms. This allows sentinel values to be preserved internally across Unicode encoding forms, even though they are not designed to be used in open interchange.

It would wrong for a general-purpose parser to reject these codepoints.

chungy2• 8 years ago

Hmm, alright. I have seen parsers reject them, presumably from misinterpreting the standard.

Thanks for the correction :)

bitcrazed24• 8 years ago

Hi - Windows Console PM and author of the post here :)

You're absolutely correct.I made some fixes, adjustments, and clarifications to the post this afternoon - hope they're a little more accurate.

We'll be digging into this area in much more detail in future posts.

tofflos• 8 years ago

I love reading about the work that's being put into old apps like Console and Notepad. Keep it up!

dvfjsdhgfv2• 8 years ago

I love Cmder, it's superior to cmd.exe in all possible ways, but from time to time it just stops working. Now I understand better why.

ygra8• 8 years ago

cmd is the shell, Cmder (which is just a wrapper around ConEmu) is the console/terminal. Both have little to do with each other, apart from that one needs the other to show a window.

EnderMB• 8 years ago

I haven't done any Windows dev since last year, but I've often wondered why more .NET devs don't use Cmder. I've always been a fan of a full-screen console with split panes (tmux style), and other developers have looked at me like I'm some sort of wizard, despite using fairly basic tech.

Sure, it takes a little while to get used to Powershell, but once you get over the hump it's really useful knowledge to have, especially if you have anything to do with devops or deployments.

bitcrazed24• 8 years ago

G'ah - PLEASE read the posts about which you're commenting - Cmder is a terminal/Console, not a shell.

Cmd.exe is a (headless) command-line shell.

A shell accepts input from a terminal/Console and emits text to its connected terminal/Console to be displayed back to the user.

All this is covered in some detail in the posts thus far.

dvfjsdhgfv2• 8 years ago

That's quite clear, I have no idea why people assume I confuse the two. I just said the article helped me to understand why Cmder has certain problems.

slinzer• 8 years ago

Too many exclamation marks.

bitcrazed24• 8 years ago

Fixed! Let me know if you prefer the updated version I posted this afternoon.

Thanks for the feedback :)

frankohn2• 8 years ago

I like the post, it seems like they are good guys doing a good work on an unrewarding application.

It is interesting the insight thet are giving with this article: - MS is improving on Windows Console only because of WSL - the design of the Windows Console was flawed since the beginning and remained unchanged to something like 30 years. Flaws includes: - using Console API instead of sending characters / cannot work remotely - the shell and the terminal are not clearly separated: the OS will open automatically a console for some kind of executables / cannot attach a shell to an arbitrary console

It seems to me that it shows a few things.

First, unix design was better as it was more flexible and lasted over time and computers evolution. Windows' design was sort of wrong since the beginning but they didn't change anything and developers were left to use one of their horrible APIs. This is similar to the difference between X Window and Win32 GUI API.

Second, MS does not care about developers except for the user of their flagship product Visual Studio. They put 30 years of developer effort solely on Visual Studio and nothing else. Compare with Apple, they at least cared of implementing an excellent terminal and contributed greatly to the wonderful LLVM compiler infrastructure.

So, okay some improvements might lands on Windows 10 at some moment but I personally don't care. At work we still use Win 7 and I think we will keep using it for a long time. The Windows Console is just horrible to use and propram for and a testament to the MS awful software and APIs that developers are forced to use because of Windows dominance.

ygra8• 8 years ago

I didn't get the impression that they consider the Console API in itself a flaw. An API for this is much more robust than an in-band text API that's part of your output. For example, no Windows console application has to even think about whether printing a user-provided string can be a problem. They do note that it's an impediment for porting software to Windows, as well as the problem that the API isn't able to be implemented by any other application, e.g. ConEmu. Those applications have to rely on scraping a hidden console window to work instead of just being an alternative console implementation.

I read it not so much as a testament to the perceived superiority of the Unix model, but rather that with WSL users expect certain things to work more Unix-like, just that they don't work like that right now.

repolfx• 8 years ago

The most obvious problem of their design is that there's no serialisation mechanism for their API calls, so apps don't remote over low bandwidth connections. Which is one of the big advantages of terminal apps to begin with.

Microsoft did objects better later on with COM and DCOM: you can imagine a better console API if COM had come first, but even then, DCOM network protocol was not good and had numerous problems that would have meant ssh still beat it. But trying to do OO API in 1989 with their tech stack wasn't a good idea.

oblio9• 8 years ago

X Window has a good design? Only because it allowed competing tool kits on top of it ("Provide mechanism rather than policy. In particular, place user interface policy in the clients' hands.")

Besides the fact that the X libs are almost universally reviled and nobody used them as soon as they could (Motif, Qt, Gtk, etc.). The core concept, that of network part is actively being abandoned right now, see the Wayland efforts.

X Window won because its competitors were proprietary, basically. And then it dragged on because every application was compatible with it.

Win32 is also horrible, but you can kind of get why it was that way: the hardware was abysmal when it was conceived and backwards compatibility was super, super important in the early and cutthroat desktop market.

barrkel2• 8 years ago

I don't think Win32 UI API is terrible. It's basically message passing object orientation before its time.

The API idioms are clumsy to code against, but OTOH it's much more viable to avoid linking libc into your app, whereas in Unix land, the only portable API is defined largely in terms of a C worldview. And some things are just leagues better, like signals vs structured exception handling.

frankohn2• 8 years ago

X Window has a good design? Only because it allowed competing tool kits on top of it ("Provide mechanism rather than policy. In particular, place user interface policy in the clients' hands.")

Yes only this simple thing made a much better design. It can be used to provide a modern and beautiful desktop environment even 35 years after its creation.

Besides the fact that the X libs are almost universally reviled and nobody used them as soon as they could (Motif, Qt, Gtk, etc.).

You are not supposed to use Xlibs directly to program user interface. They just provide the primitive APIs to create windows, process events and draw on the screen. It is normal to use a toolkit like Qt that use Xlib as a backend.

I agree that programming Xlib directly is not very sexy but it sort of make sense and is yet better than Win32 programming.

The core concept, that of network part is actively being abandoned right now, see the Wayland efforts.

The fact that this is a good choice is still to be demonstrated.

Win32 is also horrible, but you can kind of get why it was that way: the hardware was abysmal when it was conceived and backwards compatibility was super, super important in the early and cutthroat desktop market.

Yes, you can excuse them for creating Win32 at the time but they have no excuse for not providing qnything better after 30 years. Oh yes, you can create beautiful UI if you only use Visual Studio, you get use to all the warts and limitations, if you forget cross-platform, it will works only on windows, if you use visual studio installer and ship all the zillions of DLLs that you may need with the most baroque distribution system.

On windows I am still looking for a decent way of programming UIs without embracing all the MS bullshit.

msla6• 8 years ago

X Window has a good design?

X does. "X Window" might not, but it probably isn't in widespread use.

barrkel2• 8 years ago

I don't think anyone who has suffered fixing their termcap, terminfo and stty settings would think Unix has much to boast about in the terminal space. Even today, whole swathes of key combinations have no distinct escape sequences, forcing a different experience for apps like Emacs that can run both native and in terminal.

FWIW, I worked at Borland back in the day, and it simply isn't the case that MS only cared about VS users.

zadjii17• 8 years ago

We're certainly not improving the console only because of WSL - WSL has highlighted a lot of functionality that we could improve upon, yes, but we're improving the console because the console needs improving.

I'd say that we're conceding that the linux commandline application model (having two streams of characters) is more portable and cross-platform compatible. The Windows Console API isn't particularly well designed and it's incompatible with *nix terminals, so instead of us trying to force our shitty API on them, we'll just adopt theirs.

I certainly care about developers. I know everyone on my team does. I know that there's not a tom of goodwill between Microsoft and external developers, but there are a lot of people at Microsoft who are working really hard to try and fix that.

cryptonector6• 8 years ago

So... I think the Windows model is superior in these respects:

     - object handles >> file descriptors
     - security descriptors >> {owner, group, mode, [acl]}
     - access tokens >> [gs]et[e]?[ug]id(), [sg]etgroups(), initgroups()

But everything-is-a-file makes things generic, which means they can be remoted (as you put it), which means I/O redirection in shells is trivially transparent to the programs being run , etcetera. This is a very big deal.

Now, not everything is really a file on Unix. There are ioctls for some things -- lots even. And the limited and non-standardized representation of processes as files is obnoxious, but whatever, it works.

The Windows model of console as ioctl-/API-based is simply an utter disaster of calamitous proportions. It's the reason Unix users/devs hate the Windows experience, and it has held Windows back in some ways. The success of Windows as a whole has papered this over, and it's probably the reason that the console got no love from MSFT for thirty years, but the problems are real.

Another thing that is shockingly bad on Windows is the C run-time. The lack of a single C run-time DLL, the fact that the C run-time is statically linked into every DLL, and the total lack of love that MSFT has shown the C run-time, is atrocious. Doing portable code development, where by portable one means "runs on various Unix and Unix-like OSes _and_ Windows", is rather hard, and the hard part is always Windows. Invariably this involves writing C code with lots of #ifdef WIN32 spaghetti. The only good thing about this is that it has forced some memory management discipline on library APIs that might otherwise not have happened.

If there is any chance that you could get MSFT to fund a similar project for revamping the C run-time, that would be fantastic. Because there are so many discrete portions of the C run-time to fix up, such a project could probably scale to more than three developers -- perhaps three core developers, two developers to improve a variety of individual and peripheral functions, and two test engineers. Please mate.

msla6• 8 years ago

X Window

logcat• 8 years ago

There are ways to get much better terminal experience for typical web dev https://blog.ropnop.com/configuring-a-pretty-and-usable-term... (also works for tilix)

cbolton• 8 years ago

This a very candid and informative look at the Windows console and it comes from Microsoft! I'm impressed. Congrats to the writer and thanks for posting!

sevensor• 8 years ago

This is of a piece with Microsoft's GitHub acquisition -- seems to be part of a program to win developer mindshare. And it's working; I was impressed by this post, both tone and technical content. Windows having never really been suitable for the server, and rapidly losing its grip on the home user, it seems totally sensible to push the "windows is a good platform for people doing serious work" angle. It's all Windows has left.

bitcrazed24• 8 years ago

Hi. Windows Console PM and Author of the post here :)

Thanks for your kind words. Glad you liked it - check our blog often - LOTS more posts coming on this subject!

megaman223• 8 years ago

Yikes, the anti-Windows crowd is strident today. Yeah, cmd.exe is crap, and if you're expecting to use a Unix workflow rather than learn how things work in Windows, you're going to have a bad time. I'm not sure that religiously sticking to the baggage and constraints of obsolete hardware from fifty years ago is really a great ideal either...

But you just can't change existing shit willynilly in Windows, with the kind of install base and legacy it has, or you'll break things in a way that has been historically unacceptable in Redmond.

vetinari3• 8 years ago

I'm not sure that religiously sticking to the baggage and constraints of obsolete hardware
But you just can't change existing shit willynilly in Windows, with the kind of install base and legacy it has,

You do see the irony, right?

tom_3• 8 years ago

I think his (?) point is that if you don't have the backwards-compatibility goals of Windows, you can afford to be - and perhaps should be - a lot more adventurous. Which is something POSIX systems have summarily failed to be in this respect, for all that they've managed to polish that turd to an acceptable shine.

gigatexal2• 8 years ago

They should just replace it with the power shell prompt that wraps the command line.

gigatexal2• 8 years ago

Too many fans of the command line from the 90s. Oh well.

nailer3• 8 years ago

As a nix person who now spends his entire day in the Windows terminal, I wrote a guide to setting up the console and pwsh for maximum productivity:

https://github.com/mikemaccana/powershell-profile/blob/maste...

7kmph• 8 years ago

Who cares anyway

lagadu2• 8 years ago

Well, you cared enough to post. Now imagine all the people who actually enjoy reading insights about developing software that we've all used.

deejbee• 8 years ago

One thing I've always missed is tabbed consoles. I've use Cmder (http://cmder.net/ I have no affiliation) for quite a while now and has been totally awesome as it provides a tabbed container window with any mixture of cmd.exe, bash shell, launching as Admin and best of all it persists all your windows after reboot.

neilsimp1• 8 years ago

Another Cmder lover here. Although when the Windows 10 "Sets" UI comes out and default Powershell/conhost has tabs built in, it may be time to just stick with that. We'll see...

duncanawoods• 8 years ago

Pretty unhappy Cmder user here.

Sooo many bugs. Currently have dash.exe crashes on start-up, fatal race conditions generating certain types of error output, performance becomes unusable in git folders with lots of files because of how it updates the auto-text in the prompt and it has the configuration/settings from hell but they just blame ConEmu.

I just want a tabbed console that will remember the tab locations and command history between sessions. This would seem obvious thing but no.

Serow225• 8 years ago

https://blogs.msdn.microsoft.com/commandline/2018/04/13/tabb...

zadjii17• 8 years ago

Trust us, the pain on not having a tabbed console is felt here in the console team too. It's our most requested feature, but implementing it has always ended up a lot harder than expected. Fortunately we're working with the Tabbed Sets team to help nail that experience for the console.

bitcrazed24• 8 years ago

Hi - Windows Console PM and author of the Command-Line posts here.

We hear ya. Tabs are VERY high up our to-do list, but to get there, we had a TON of internal engineering and modernization to do to wrangle the ~30 year old codebase into a shape where we could efficiently and effectively work on UX features.

We're now nearing the completion of the core re-engineering effort and will start to turn to more UX-visible features in the next few releases.

Bear with us - TONS of exciting stuff coming soon!

OliverJones• 8 years ago

Very interesting. I learned a lot.

The author mentions "re-re-re-implementing" command line history as a disadvantage of using shells in a dumb terminal emulator.

But, with respect, I think he's wrong about that. Command line history is a valuable productivity enhancer, and tying the details of its operation to the terminal rather than the shell restricts functionality and flexibility. For example, witness the differences between git bash (mingw) and cmd on Windows.

Inbuilt OS stuff should be as simple as possible. It doesn't have to be precisely /dev/tty - like, but it should be simple. Partly to avoid bloat, and partly because it's much less flexible than userland stuff.

Even when RAM was worth twice its weight in gold, various UNIX shells weren't heavy. Now? not worth worrying about.

I admit I came up as a UNIX user so I favor sh-like history. But still, Windows terminal history just isn't as good. (I wish it didn't deduplicate repeated commands and lose them when they're edited.)

laumars5• 8 years ago

There was a few points the other raised which I disagreed with and that was one of them. I completely agree with you that shell history should be managed by the shell itself.

I also didn't agree with the points about Console API. I mean sure, he's right that ANSI / VT escape sequences are naff, but they are the standard so we're stuck with them. However as much as I love the idea of a Console API, the fact that they only work on local host is just inexcusable. And I don't say this because "hindsight is 20/20", there was already decades of proven example that remote consoles are worth having before the console API was even written. Why they couldn't have created is a network API (ala rsh) is beyond me. But we are where we are.

Some of the examples given about OOP was just plain wrong too. You can't bunch 20 years of object oriented langaues together and say "they all quickly followed". 20 years is not quick; not in real terms and most definitely not in IT terms.

What surprised me the most when reading that article is that NT was originally command line driven with a GUI added for release. It's amazing just how bad the console sucks even without taking the evolution of NT into account; and now that I know NT was originally command line oriented, well I'm just shocked.

If there was anything good that can be taken from the article, it's that it sounds like they are working on better terminal / shell separation which will enable developers to write better terminal emulators and better alternative shells. Plus potentially allow for a networked shell. My next request after that would be to write a unified way of splitting ARGS because their current method of passing all command line parameters as a single string is just terrible (yes, I get that's DOS legacy and impossible to change due to backwards compatibility, but isn't it about time Windows had a standard API for splitting that ARGS string so, going forward, programs can be written to split parameters in a consistent way meaning command line users know exactly when they can and cannot use quotation marks, spaces and escaped characters, etc.

cryptonector6• 8 years ago

Hear hear!

Terminal escapes... you know what, they may seem ridiculous, but they work with a read/write API instead of an ioctl API, and that makes all the difference. And they work with serial ports and terminals. From day 1 terminals were read/write-oriented, not ioctl-oriented, and the concept of escapes originates with the terminal vendors, not with Unix.

Regarding command-line parsing, there are ways to move to a Unix-style argument array in a way that is backwards-compatible. It could just be an attribute of the executable to exec(), that it wants a proper argument vector. Shell escaping in Windows is just not-sane.

bitcrazed24• 8 years ago

Hey Laumars - Console PM and author of the post here:

Fair points in part, though …

Remember that things like Remote Desktop obviated the pressing need for command-line --> remote command-line access for many years. RDP quickly evolved into a very efficient way of remoting the entire desktop experience, not just Command-Line - something that was particularly necessary until PowerShell started to mature due to Windows' heavy GUI tool influence (for better and worse).

Don't despair though - there's a VERY interesting post coming soon on this subject, that I think you'll enjoy! ;)

I summarized 20 years of history: Simula arrived in '68, Smalltalk in '72 and then, other than research efforts, relatively little until CFront/C++ around '85. After that there was a new variant of C++ or new object oriented language almost every 8-12 months ever since - ObjectBASIC, Modula-2, ObjectPascal, Delphi, Java, Python, C#, Ruby, etc. It was practically a language explosion.

And, of course NT started as a command-line OS. All new OS' start that way it's the simplest UX to get running. The Windows GUI wasn't slapped on at the end though - it arrived relatively quickly since Microsoft was able to leverage much of the experience of having build Windows 3.x/9x etc. previously.

Command-line args are not all passed as a single string. In Windows, args are separated by space. If you use C/C++ and others, you get argv and argc, or in C# you get an array of space separated values.

Alas, when NT came along it was seen as EXTREMELY important to make it easy to run/port MS-DOS scripts and tools to encourage migration and adoption. By this time, so many tools used so many ways of specifying args and values … / vs. \ vs. - vs. none, /o=foo, /o foo, etc. … that it was practically impossible to both support backward compat while rationalizing usage.

PowerShell, though, has done much to make args MUCH more consistent, self-documenting, etc.

This said, yes, it'd be AWESOME if args could be rationalized and handled in a more consistent manner. Had a fascinating discussion about just this the other day. Stay tuned to our blog ;)

cryptonector6• 8 years ago

I really don't understand why I would want the terminal to be aware of command-line history. That seems completely wrong. On Unix one can have a terminal used by a non-shell application -- what would a terminal's command-line history mean in that case?

bitcrazed24• 8 years ago

Hey Oliver - Console PM and author of the above post here.

Glad you learned lots - makes writing it worthwhile ;)

I didn't mean to imply that command history is not a productivity enhancement - it absolutely IS!

What I was saying was that in the absence of a system/terminal-provided command history mechanism, every shell has to implement its own. Cmd needs it. PowerShell needs it. Bash needs it. Zsh … fish … yaddayaddayadda need command-history.

So if every shell (and many apps) need a command history buffer, why not provide command-history as a service from the terminal. There are MANY amazing and valuable things one could do if one could consistently log/store/serialize command-history for later replay, analysis, etc.

I agree 1000% that Windows' command history is FAR from good, but I think there are some pretty awesome things we could do to make it so.

Bear with us on this ;)

jscholes• 8 years ago

Found the article content quite interesting but the tone of the writing (emojis and exclamation marks everywhere) really put me off. Sounded forced and added nothing.

degenerate• 8 years ago

It reminded me of a very excited Steve Ballmer: https://www.youtube.com/watch?v=HMhrRovP9qA

bitcrazed24• 8 years ago

Fair comment - I agree and so removed a bunch of !'s and emoji for you, though I did leave a few in to convey some of my (and the team's) excitement - we're genuinely buzzed to be able to massively improve the Console.

Hope you find the latest updated version more palatable.

xvilka• 8 years ago

Hopefully, support for bidirectional text will be added soon. Microsoft needs to lead the way of handling it in console programs. Even in UNIX systems, only a few terminal emulators support bidirectional text properly.

zadjii17• 8 years ago

Could you go submit this to our UserVoice?

https://wpdev.uservoice.com/forums/266908-command-prompt-con...

I'd say that bi-directional text is a pretty low priority at the moment. We just haven't really heard any asks for it, and like you said it's poorly supported on other platforms (if at all).

cryptonector6• 8 years ago

I really like the point that an API-centric console does not remote well (think ssh).

In general I prefer the WIN32 object-handle model to the Unix file-descriptor model, and I much prefer the WIN32 security descriptor and access token model to the Unix haphazard security model, but very clearly, the everything-is-a-file model is superior at least for remote access: it's easier to design and implement protocols for that since the protocols don't need to know anything about the remote resources' nature. Thus the Windows console cannot be exported via SSH, not without writing a layer that converts text reads/writes into ioctls, because it's not remotely file-like. That the Windows team had to and did write that layer really is fantastic news.

iforgotpassword2• 8 years ago

Oops! That page can’t be found. It looks like nothing was found at this location. Maybe try a search?

Is this just me? Late to the party?

In case this isn't temporary: https://web.archive.org/web/20180717165103/https://blogs.msd...

bitcrazed24• 8 years ago

LOL - nothing nefarious - made a few corrections and improvements to the post - aaaaand managed to bork the publishing process which took a few mins to re-generate and refresh the cache!

Ugh!

Should be back up and available now!

zaat3• 8 years ago

I'm late to the party, but maybe the console guys are still lurking. If so, than first and foremost I wanted to thank you for the amazing walk you guys took the console so far.

That said, as someone who uses your product for hours on daily basis, I have two pain points that I would love to have fixed, if that possible at all:

1. selection of text and deletion - works in PowerShell, doesn't work in cmd. Is this curable?

2. Customizing the console is pain. I don't mind doing it once but it is impossible to save theme and copy it between computers. It is also impossible to export registry keys - not only do they differ for various consoles, the color settings are not saved there. The only way I found so far for achieving this is copying the shortcut for the specific console - the colors are saved to the .lnk file in binary form. Seriously, can we have at least color exportable themes in a format that can be edited with a text editor?

bitcrazed24• 8 years ago

Hey @zaat. Thanks for your kind words :D

Alas, deleting text is an operation controlled via the shell. PowerShell's (awesome) PSReadLine does it for PowerShell, but introducing that kind of behavioral change is unlikely to be possible in Cmd because, as pointed out in the posts, Cmd is in maintenance mode, is extremely fragile, and it's key MO is backward compat, which includes behaviors because many tools, and scripts depend on behaviors (even broken ones) as "Features" ;)

zaat3• 8 years ago

Thank you for answering. Any chances on number 2, especially color themes?

zadjii17• 8 years ago

I think rich (@bitcrazed) already talked a bit about #1, so I'll share a bit on #2.

I agree, the current properties story is INSANE. I don't get how whoever wrote it even came up with it - it's likely a frankencode system that we're stuck supporting for now. I can't tell you how many times I've had to explain how the console settings work, and how your settings can be different depending on how you launch the console. It's been a pain point I've had a particular eye on since I started here.

It's a powerful system that is very prone to "you moved my cheese" syndrome - It's REALLY hard to change it into a sane system without breaking someone's workflow.

That being said, I have a bone to pick with that system, and I think I might have some spare cycles to finally get to it soon :)

PS: While we're on the subject of colors, lemme quick throw out (Colortool)[https://github.com/Microsoft/console/tree/master/tools/Color...]. It can be really helpful for quickly changing the color scheme of the console. It's not great for everything, but I've found it super helpful.

zaat3• 8 years ago

That's look promising. Thank you very much!

Inside the Windows Console

Related Stories