VSCode-WASM: Implement a first version of a WebShell

https://github.com/microsoft/vscode-wasm/commit/0cdb9997d83bae116bb7693ba6837ffa812425a7

by tech234a • 3 years ago

90 31 3 years ago

rektide2• 3 years ago

Getting a FS going on the web has been such a brutal slog for me. There's some pretty decent existing projects lole lightning-fs but it feels like the web APIs have been changing & changing & I keep wanting newer.

Very cool to see this nicely tackled. Still poking around but seems integrative with wasi, which hopefully grows closer & closer with good fast web APIs. There's been a ton of criticism I think justly so that neither the wasi nor the web fs teams have spent much time trying to figure out how to work together, & so there's real peril possible too, but I think wasi is a much better much less hacked out model & is probably the "right" real target. Having build in object capability sandbox security puts it lightyears beyond what the web spec folk have been up to, and the rest of their fs model is also much much better built out.

Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web!

Already a very solid start building out coreutils. There's a bunch of similar ish efforts out there already, building JS impls of core utils. For example, posix-tools, https://github.com/shama/posix-tools

jakear2• 3 years ago

running on the web the target so going to be primarily in-memory for a long time

What do you mean by this? FS implementations I've seen (including lightning-fs) use IndexedDB for the backing store, which is not "in-memory". I authored VS Code's web file system driver and it also uses IndexedDB, which I didn't see evidence of being memory-constrained. Furthermore, there's the File System Access API for web which is also not in-memory (I also authored VS Code's driver for that...)

rektide2• 3 years ago

Color me impressed. I have been aware of indexeddb drivers, and I didn't investigate Vs code's driver (looked briefly through the PR but was short on time). I don't know why I don't credit that route enough.

File System Access API just feels way to limited. There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be when the web api has so many limitations.

cxr• 3 years ago

Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web!

Despite the constraints of the browser sandbox and the performance penalties, when you consider all the environments where people have actually used coreutils to great effect, it's undeniable that people were getting real work done in the 90s and early 2000s on strictly worse systems; if you tried to simulate many of those experiences entirely in-browser today, then you still wind up at an advantage (even with modest hardware, like, say, a 5-year old Chromebook) compared to what people were actually getting by with at the time.

The biggest stopping force is really the premature fatalism that follows from the comparison to programs that run with the system linker/loader, have unmitigated access on the host, and the freedom ("pleasure"?) of running memory-unsafe code, or so on. I'll further stipulate that programs that are written in such a way that they require any one of these things are fundamentally broken (at least for most values of x).

syrusakbary3• 3 years ago

The goal of the extension is great, as it allows developers to use WASI easily when creating new extensions. Mad props to the team that worked on it.

But is unfortunate that they used literally the name of the most famous extension for WebAssembly in VSCode (vscode-wasm), which has been around already for a few years, as it might confuse people on what is what:

https://marketplace.visualstudio.com/items?itemName=dtsvet.v... (113k installations)

Repo here: https://github.com/wasmerio/vscode-wasm

nl• 3 years ago

allows developers to use WASI easily when creating new extensions

What is WASI?

re-thc• 3 years ago

API for Webassembly. Common interface.

nindalf• 3 years ago

WASI stands for WebAssembly System Interface. It's an API designed by the Wasmtime project that provides access to several operating-system-like features, including files and filesystems, Berkeley sockets, clocks, and random numbers, that we'll be proposing for standardization.

cjbprime• 3 years ago

Am not expert, but here's the understanding I've picked up:

When you use JavaScript through Node, your JS gets the ability to open files and make network sockets and so on, because there are a set of APIs implemented by Node and exposed to your JS code, like so:

JS <-> native APIs <-> Node implementation of platform-specific operations

WASI applies to WebAssembly instead of JS. So again if we're on Node:

wasm <-> native APIs (WASI) <-> Node implementation of platform-specific operations

But WASI is much more general than Node's APIs, and meant to be used by different engines in different environments (browsers, native bindings, etc).

tcper• 3 years ago

Simplely, it's wasm's POSIX

capableweb2• 3 years ago

But is unfortunate that they used literally the name of the most famous extension for WebAssembly in VSCode (vscode-wasm), which has been around already for a few years, as it might confuse people on what is what:

At this point, where Microsoft frequently ends up naming something identically to something else already existing (and sometimes quite popular) I'm done believing it's "unfortunate" and not on purpose. Once or twice maybe it's some unfortunate accident, but at this frequency?

duped2• 3 years ago

It's more likely that they just don't care. There's a certain cavalier attitude from the VS Code project in general.

jeroenhd• 3 years ago

It's similar to how Google chose "Go" for their language despite the existence of another programming language named the same.

Big companies don't care unless you have a trademark.

syrusakbary3• 3 years ago

Let's give them the benefit of the doubt, and let next actions speak for themselves.

I've created an issue in their repo, so they are fully aware of the issue and they have the opportunity to amend it :)

https://github.com/microsoft/vscode-wasm/issues/62

yewenjie• 3 years ago

It is kind of sad that Monaco (VS Code's editor as a JS web component) still doesn't support tree-sitter.

debugnik4• 3 years ago

Tree-sitter grammars take more effort than textmate ones for the most basic highlighting, but provide less accurate parse trees than a compiler could use for more advanced features and highlighting. I understand what they were going for but I don't "get" the trade-off.

At least as long as the parse trees are inherently worse: Even ignoring context-dependent grammars, last time I checked tree-sitter parsers had many edge cases in which they'd accept invalid syntax but not mark the parse trees as bad, which makes them unsuitable for compiler use.

DannyBee3• 3 years ago

Using compilers for highlighting basically doesn't happen because they are too slow. They are used for progressive improvement as a result (depending how fast, it can take at least 5-10 seconds post-update to get a result. For slower languages, minutes).

I'm not sure what "more effort" means - for lots of languages, it's less effort than textmate because people spend lots of hacks trying to work around textmate to get it to highlight even slightly complex languages right. It's nesting mechanism is complicated, etc.

Tree-sitter is faster than textmate on initial parse in lots of cases, and update a lot faster than textmate (since they can be incrementally updated and textmate can't).

They are not meant as compiler replacements.

Again, compilers are slow, and rarely incremental. Using tree sitter for folding/highlighting/etc, and compilers for additional progressive type info, is the right path.

Textmate is worse than useless for anything but super simple languages.

I say this as a person who has written about 15 vscode language servers for all sorts of varied languages, from CNC g-code to you name it.

I even implemented incremental parsing/lexing for ANTLR's typescript port (and then java port), as a comparison to tree-sitter ;) It can be made as fast, but tree-sitter is clearly winning the current popularity contest ;)

I'd say for those languages, textmate work was the worst part of all of them, and getting textmate to highlight properly required more work than writing proper lexers/parsers. I still fix more bugs in textmate grammars than anywhere else.

Part of the reason is that it requires fairly complex regexps. Part of it is getting it to stop trigger on things it shouldn't, because you have to emulate a lot of state. Again, you start to spend a lot of time working around it, rather than working with it. Meanwhile, yes, the upfront work of a lexer in tree-sitter is sometimes more. But it is much easier to get it to work, to rely on it working, etc.

In the end, all my language servers were easier to implement in tree-sitter, and some don't even have compilers (like g-code), but have control flow/blocks/etc. tree-sitter handled all language duties there.

Hopefully this helps you "get" the tradeoff.

Anyone who has written enough vscode language servers would tell you the same thing. I have yet to meet someone in the community who writes language servers who is excited or happy with textmate vs tree-sitter.

Where do you think the push came from?

duped2• 3 years ago

Tree sitter grammars generate a complete/concrete syntax tree with incremental updates, they necessarily have to be able to represent invalid source text (because users will type it!).

If you're using a tree-sitter grammar as the input to a compiler (which is not what its designed for) you need to write a pass to convert from CST to AST, as you normally would.

JoelMcCracken• 3 years ago

Can anyone explain what this is? all of the comments seem to be talking about different things, and when I look at code though it looks to me like they're implementing a JS based terminal?

pipes• 3 years ago

Can anyone explain what this is?

csense• 3 years ago

So... what is a WebShell, and how does it differ from, say, what you get if you compile a "lightweight" shell like Ash to a WASM / WASI target?

VSCode-WASM: Implement a first version of a WebShell

Related Stories