Oils 0.17.0 – YSH Is Becoming Real

https://www.oilshell.org/blog/2023/08/release-0.17.0.html

by chubot • 3 years ago

66 20 3 years ago

Oil looks pretty incredible and I think both the work done and the dedication to backwards compatibility are admirable.

That being said it seems like Oil is polishing the shiniest turd. I never liked Unix shells or shells much in general and strongly prefer to type out commands in a script (usually Python or Perl) and then run that script. It takes longer but I actually trust the result to work and I’ll understand it two months from now. Also when you do that you get all the benefits of using your riced-out, LSP-enabled editor. Ofc I still use a terminal for apt and other such programs, but not munging data.

It seems like if you wanted to make a modern shell you would work on improving the interface and not so much the internals. Like in the SerenityOS terminal emulator, how everything is clickable. I’ve always wanted an easy way to “export” a shell session to a script, including env variables and whatnot. Maybe types in YSH will enable better terminal emulators, I don’t know.

aidenn0• 3 years ago

While OSH aims to be strictly better than Bash, YSH aims to be strictly better than "shell written in Python[1]." If there is something shell-like that you think Python is superior to YSH for, then please give some examples before the language hits 1.0 so they can be taken into account.

1: Obviously a large general purpose program that only occasionally needs to run an external program is probably going to be better in Python due to the library ecosystem alone. I'm talking about those Python programs that use primarily some combination of pathlib, os, sys, and subprocess and exist only because of the deficiencies (perceived and real) of shell.

chubot4• 3 years ago

(author here) Funny thing is I also type out most of my shell into a script file. I generally don't create huge commands interactively. I elaborated on that here:

https://lobste.rs/s/pcuppu/what_helps_people_get_comfortable...

Basically I want to be able to use all the power of Unix without remembering it all -- shell is a great tool for that.

Someone made a LSP plugin for Oils awhile back, and I expect if enough people use it, those tools will continue to improve, and new ones will appear. They will be MUCH easier to produce for Oils than shell, because we have a principled code representation. (Some tools will be inside the project, and some outside, as usual)

---

On the UI side, there are also people exploring shell GUIs via the "headless shell" -- something no other shell has -- but we can use more people to help.

screenshots: https://www.oilshell.org/blog/2023/06/release-0.16.0.html#he...

i.e. Join #shell-gui on Zulip; don't assume it will get done without your input or effort ... as mentioned in this blog post, the biggest risk is that we don't have enough people helping!

subjectsigma3• 3 years ago

I can't contribute directly to open source because of my employer. But the pages you linked are very interesting. I may be interested in doing beta testing or something, so I'll join Zulip and say hi. Thanks!

predictabl32• 3 years ago

Couldn't agree more and at this point, virtually every new mainstay tool I adopt is written in Zig or Rust and I can't see myself migrating to a shell written in C/C++. Especially when something like nushell so elegantly avoids the situation you're describing. (I use nushell as my daily driver shell, and bang out scripts in it on an hourly basis. It's a blessing to use, though still missing an essential "shell" feature or two)

chubot4• 3 years ago

There's only 7K lines of C++ in Oils, mentioned in this post, and we intentionally keep it down. (It's for the garbage collector and OS bindings -- basically the unsafe parts.)

The rest is typed Python (with algebraic data types), which more common and simpler than Zig or Rust.

It's not as fast as Zig or Rust, but

1. The shell is the same speed when it's running external processes (and we do measure)

2. I believe it can be as fast as bash's hand-written C, even for code that doesn't do I/O (which isn't common but exists). We're perhaps ~50% the speed now, being "hilariously unoptimized".

This release mentions a big optimization, but the bigger point is that we have leverage via metaprogramming. We can do transformations on the entire interpreter (e.g. "boxless" transformation)

3. It means we only have 40K lines of Python, compared to 142K lines of C in bash. If you add YSH, we will only have around 50K lines, compared to probably 200K lines if you wrote it by hand

So the whole program is closer to an executable spec -- there is less room for bugs to hide.

It's easier to optimize 50K lines that is known to be correct (with thousands of tests), than 200K lines of code

---

Our binary is about 1.5 MB of executable code with no dependencies other than libc and (optional) GNU readline.

On the other hand, shells in Rust seem to have a big tree of dependencies and end up at 20 - 50 MB. It's not the size I'm concerned about, so much as the stability

As mentioned before, our approach is unusual, but I think the "usual" approaches haven't worked well. That said, if someone wrote a compatible shell from scratch in Zig or Rust (with limited dependencies), I think that would be also be valuable contribution

---

Oils is also aimed at the system level, and is an "exterior" shell that uses processes and file descriptors. Comparison with PowerShell, Elvish, nushell here:

https://www.oilshell.org/blog/2023/06/ysh-design.html#survey...

i.e. "shell" is a bit overloaded, and in some sense those tools are "one level higher".

LispSporks22• 3 years ago

Is it still missing job control?

b5n• 3 years ago

Also when you do that you get all the benefits of using your riced-out, LSP-enabled editor

https://github.com/bash-lsp/bash-language-server

https://github.com/koalaman/shellcheck

https://github.com/idank/explainshell

em-bee• 3 years ago

i agree with that. i almost never create scripts though, because i work on multiple computers at once, and copying and adapting commandline snippets is easier than scripts that don't necesarily work the same on every machine. so i like improvements in the interactive experience like fish mostly focuses on. but if i were to write a script i would not use fish either.

that said, a language that focuses on shell scripts could be a good idea because there is not much work done yet in that space

e28eta• 3 years ago

your riced-out, LSP-enabled editor

It’s my belief that “riced out” has negative racial overtones, and isn’t an appropriate adjective to be using. Wikipedia has some history on this related term: https://en.wikipedia.org/wiki/Rice_burner

subjectsigma3• 3 years ago

Been browsing tech forums for years and years and I’ve only ever heard “rice” used as a compliment to refer to software that’s really aesthetically pleasing. Didn’t mean to offend, had no idea it was even a term borrowed from car culture.

paddim8• 3 years ago

Elk shell is quite different from traditional shells. More like Python. https://elk.strct.net

JNRowe3• 3 years ago

I really want this to succeed¹, but I struggle to use it. How are people that are testing/using it coping without the normal niceties of their regular setup? I'm wondering if there are just some tricks I'm missing.

Off the top of my head, things I'm thinking of include:

* Just a minute ago realising that without zsh globs I have to remember find syntax, and then have to figure out how to sort the results(probably decorate →sort(1) → undecorate). My laziness led me to just C-d and execute my task with zsh.

* No context sensitive completion; I don't realise how much I rely on this until it is gone. Using bash-completion gets some completion, but it is a big step backwards.

* No programmable input, you can fake some with ~/.inputrc hacks but only basic text support(no macros). (Edit to add: The guard name is "oil" not "osh", if you're looking to do this yourself.)

There is a document in the wiki², but it doesn't really cover much. I'd be interested in seeing the configs of people who are using it.

Should say I really like the idea of hijacking '#!/bin/sh' to inject osh, it feels like a really good way to round out testing of scripts you'd otherwise forget about. I wonder if a LD_PRELOAD'able solution that would work outside osh would be a good idea.

¹ Strong déjà vu on this comment, but if it is real it is because I truly like the goals set out for oil.

² https://github.com/oilshell/oil/wiki/How-To-Test-OSH

chubot4• 3 years ago

(author here) This is good feedback

1) I think you mean stuff like **/*.py from zsh, which bash now emulates with shopt -s globstar.

This has come up several times, and we should have it. But I'm actually not entirely sure how ** behaves -- if someone can write an exhaustive shell script that shows what it does (e.g. a bunch of mkdir and touch commands and output), we can copy it in our spec tests:

https://github.com/oilshell/oil/tree/master/spec

https://www.oilshell.org/release/0.17.0/test/spec.wwz/osh-py...

and then implement it. Generally once the behavior is spec'd out, it's not too hard to implement

2) Completion has a long history in the project -- we emulate bash completion very faithfully, but yeah I've realized that's not enough

This comment elaborates on a different plan -- I think something like "fishcomp" to emulate fish would be both useful and pretty feasible:

https://lobste.rs/s/fdqbhk/oils_0_17_0_ysh_is_becoming_real#...

We can definitely use help with that, and it's relatively self-contained.

3) What exactly do you mean by "programmable input" ? We support GNU readline so everything there should work (maybe the ./configure detection didn't find it for you?)

---

But there is a bigger reason for the "struggle" -- the main focus of the project now is YSH, the language with data tYpes, influenced by pYthon, JavaScript, Lisp, etc.

That is, a shell is both a language and an interactive UI, and I've long emphasized the language part, e.g. in the FAQ - http://www.oilshell.org/blog/2021/01/why-a-new-shell.html

The theory was/is that the language is not only important for the system level tasks, but also a good foundation for the UI, e.g. like Emacs is written in Emacs Lisp

So I think structured data (in the form of languages/protocols) in YSH is really the killer feature of Oils, that will hopefully bring many other people into the project, who can help with filling out the interactive features you mention, e.g. polishing completion.

I wrote FIVE posts about YSH in June: https://www.oilshell.org/blog/2023/06/release-0.16.0.html#so...

As mentioned in this post, the biggest risk is that we don't have enough help. All the technical pieces are there now, and the risks are about project management, not the code.

I appreciate everyone who contributed while the codebase was undergoing this big transformation to native code -- this was kind of risky and the most unusual part of the project, but I think it paid off!

(We also have big ideas for the interactive shell in the form of a headless protocol for GUIs, but again need help -- https://www.oilshell.org/blog/2023/06/release-0.16.0.html#he... )

johnjameson99• 3 years ago

Instead of going with the strained datatYpes and pYthon reasoning, perhaps you could just say that YSH(and I call it y shell in my head) is WHY you are doing all of this in the first place. The end goal so to speak.

JNRowe3• 3 years ago

1) In the specific instance I hit when I wrote that comment I wanted a variant of `**/*(.Om)`, which gives me files sorted by mtime. I think I'd probably have ended up with a `find ... | xargs stat -c '%Y-%n' | sort -n | sed "s,^[0-9]\+-,,"` construct if I'd kept at it, but I'm sure there are some more reasonable ways I'm struggling to think of right now with my extended glob blinkers on.

Yeah, I'd go the fish route if I had to do the work too ;) It is a huge step up from bash-completion, whilst still being easy to understand.

zsh's completion is unbelievably powerful, but also exceedingly complex once you move beyond basic file completion. Anything beyond even the most basic example tends to be littered with other zsh-isms too(said with the love of someone who submits such monstrosities to projects on a semi-regular basis).

[Edit: I just spotted a patch I submitted to bash-completion almost twenty years ago², I've been chasing this "perfect tooling" tail for a long time.]

zsh allows fully customisable interactions by defining custom widgets within ZLE¹. Like many things with zsh they're super powerful once you work your way through the learning curve.

I have added a few `$if oil` blocks to my ~/.inputrc to workaround a few of the edit sequences I like to have available, but beyond the built-in readline functions and basic mappings readline feels quite limited to someone accustomed to zle. I'd just have to get used to some of the other functionality needing to be external tooling, and extracting things out of the shell would probably make a lot of sense anyway.

Also -- if you made it this far -- let me reiterate how much I appreciate the work you're doing. I've learnt heaps reading through both your posts in the blog and your comments here!

¹ https://zsh.sourceforge.io/Doc/Release/Zsh-Line-Editor.html#...

² https://github.com/oilshell/bash-completion/commit/24ab16438...

saghm• 3 years ago

without zsh globs I have to remember find syntax

My "solution" to this is using https://github.com/sharkdp/fd (even when in zsh and having glob support). I'm not sure if using a tool that's not present by default would be suitable for your use cases, but if you're considering alternate shells, I suspect you might be

JNRowe3• 3 years ago

[My bad, I wasn't very clear in my comment.]

I do use `fd` in simple cases as it is sooooo fast, but it has quite limited functionality compared to zsh's extended glob syntax¹. I must add that `fd` is one-billion times more readable when you're flipping back through your history or extracting a short script with `fc`.

¹ https://zsh.sourceforge.io/Doc/Release/Expansion.html#Filena...

Oils 0.17.0 – YSH Is Becoming Real

Related Stories