Rye: Flask author's new project for Python bootstrapping

by BiteCode_dev • 3 years ago

208 45 3 years ago

The project is very new and unstable. Don't expect it to solve the whole thing today. But the concept is sound.

How's support for PyTorch/Jax/Tensorflow-GPU? Poetry struggles when resolving cross platform bundled driver dependencies. So far only conda/mamba is able to handle it performantly.

qbasic_forever2• 3 years ago

If you can pip install those libs and they work on your platform then it looks like this should work--it's just a nice opinionated wrapper around pip, virtual env and other Python tooling. If you need conda installs because those libs have specially compiled native depencies (boatloads of fortran code, etc) then you're going to run into trouble and probably should stick with conda.

KRAKRISMOTT2• 3 years ago

Another issue is that PyTorch installation requires the dependency resolver to scrape the dependency host's HTML like a RSS reader because pypi doesn't automatically store metadata.

https://github.com/python-poetry/poetry/issues/6409

gaganyaan• 3 years ago

The workspace support looks really interesting, it's something that I'm really missing in Poetry right now.

alberth• 3 years ago

Off topic: it’s too bad Armin is doing so much in Rust, instead of Nim.

Would be great to have a recognizable Python developer, using Nim.

coldtea3• 3 years ago

The point is not to prop up Nim. I'm not sure why it would be desirable even, aside from "but I like Nim".

The point is to improve Python's ecosystem. If they do it using a more established language than Nim, with great traction, that's for the better.

atoav• 3 years ago

Aside from a very subjective "I like Nim to succeed" there is literally no rational reason to pick one over the other (except, maybe the other way around, because Rust has wider adoption).

So communicating that thought as if it was a generally accepted/objective statement on Armin's project isn't going to give you a lot of love.

smitty1e2• 3 years ago

My hope is that the ideas on offer inform a PEP that gets into 3.13.

Python's packaging/project management story isn't among its selling points at the moment.

coldtea3• 3 years ago

That's quite the understatement...

smitty1e2• 3 years ago

Mild >>> wild.

darkteflon• 3 years ago

This looks very nice.

I’ve been quite happy with Hatch, which I adopted about 3 months ago on the strength of the PyPA’s recommendation. I sort of like where they’re coming from philosophically - sticking as close as possible to standards - and find the API pretty nice. I’ve never used Poetry so couldn’t compare.

One concern I have is that it seems to largely be the work of a solo dev who also has other commitments. Absolutely nothing wrong with that in principle, of course, but I wonder how sustainable it is. I have just noticed that PyPA owns the repo - not sure if that was always the case or if it’s a recent development, but does bode well.

thatxliner• 3 years ago

It wasn’t always the case. It used to lie at ofek/hatch

BiteCode_dev2• 3 years ago

It's more than those tools as it provides python itself.

fbdab103• 3 years ago

Has a similar look to the Posy/PyBi idea[0]. PyBi is also currently under consideration as PEP711[1].

[0] https://discuss.python.org/t/announce-pybi-and-posy/23021

[1] https://peps.python.org/pep-0711/

techdragon• 3 years ago

Neat and it doesn’t fall into the same trap as many others trying to build similar tools in python itself by being built with rust there’s no bootstrapping issues with managing the python environment the tool to manage python environments needs.

However it’s very opinionated and ignores several things I like to have control over (and have needed to use in the past) and most importantly I want to be able to compile my own python binary and pyenv makes it pretty painless. So while I’m impressed and think it looks good it’s a 100% no for me just because I have on multiple occasions had to compile ahead for testing, or apply a tiny patch python to build on an unsupported os such as a too old or new macOS…

I’m not the target audience but unfortunately I suspect sue to how well known he is, this will lead to adoption and popularity and while I think this tool is good for him and other people I don’t think python needs more opinionated management tool projects getting wildly popular… I don’t want to see a repeat of the pipenv saga

the_mitsuhiko2• 3 years ago

Author here.

However it’s very opinionated and ignores several things I like to have control over (and have needed to use in the past) and most importantly I want to be able to compile my own python binary and pyenv makes it pretty painless.

For what it's worth I do agree with this, and in fact before I published this to GitHub, rye was capable of doing that. I somewhat intentionally stripped it down a bit to better explain and showcase what I wish that experience looked like.

In an ideal world, we would not have to use custom Python installations because a tool like rye could automatically fetch it. However there are always cases where you need (eg: local python builds for development etc.) so I will definitely want to bring this back.

The reason I never put this on GitHub is really that I do not want to build the next thing that just splits the ecosystem apart. But as of recent, there have been some developments that make me somewhat optimistic that there is a chance, Python packaging could be fixed once and for all, and this is supposed to be a bit of a teaser of what could be.

santiagobasulto• 3 years ago

Armin, thanks for rye and all you've done throughout the years.

I'm curious, what are these developments you've mentioned that make you optimistic?

hgatrd• 3 years ago

It is clever to use Rust so the tool won't break with each new Python release.

sverhagen2• 3 years ago

Isn't it also declaring complete bankruptcy as far as Python's own ecosystem is concerned, that you'd have to go to Rust to fix this?

tsunagatta• 3 years ago

Maybe if a tool like this is successful enough it could “bootstrap” the Python ecosystem into being as reliable :P

rtpg2• 3 years ago

not really? It's not like CPython is written in Python either. Almost every language that has some notion of virtual environments writes a good chunk as shell scripts for maximum environment compatibility.

Sometimes you want to use the right tool for the job, and sometimes the scripting language that has a lot of environment-based configuration will not be a good fit for an "installer" that needs to handle a lot of environments cleanly.

bsaul• 3 years ago

And to add to your point: if even rust is preferable to python to write those kind of script-like tools, what does it says about the future of python ? (Yes, i know, ML, etc)

HelloNurse2• 3 years ago

Not quite, as noted in other comments compiling the tool to a self-contained and unchangeable executable is a very effective way to avoid bootstrapping issues where to set up a Python environment (to do something useful) you need to setup a Python environment first (to run Python tools).

coldtea3• 3 years ago

It's not like regular tools need to break with each new Python realease. It's mostly endless tinkering, attention deficit, and bad choices.

Heck, even within the same Python release, I never understood why we need to be in the 2000+ update of pip, for example, and it needs to update itself almost every month or so. It's a ducking dependency installer, just formalize the design and protocols already and keep it at that.

(For context, I never had that issue with npm. Not to any same-order-of-magnitude comparable level at least).

nmalx• 3 years ago

Python/pip criticism is downvoted again. The Python propaganda arm is strong as ever. No other language relies on marketing and censorship that heavily.

Ingaz• 3 years ago

I encountered problems with dependencies in python projects only when AI/ML libraries were used pytorch/scikit-learn/etc

NPM on the other hand: it broke in every project and in trivial cases. E.g. upgrade of postgresql lib.

I hope that deno/smth other will make possible to forget npm finally

dpoljak• 3 years ago

Love the self-consciousness of it all, even having a "Should this exist" with an XKCD that jumped right to my mind as soon as I read the title.

Honestly, as someone who has to periodically jump into Python to do some scraping and data wrangling I dearly hope we land on something that is sane and maintainable in the near future. I'm just so tired of (ana)conda, poetry, pyenv, and even pip itself.

andrewshadura2• 3 years ago

I wish it supported setup.cfg

danpalmer2• 3 years ago

Out of interest... why? From what I can tell the community really seems to be adopting pyproject.toml as the standard and most tools are supporting it as their primary or only configuration option now. A few are using setup.cfg, but that seems to be for legacy reasons rather than an active decision. I'm not a huge fan of TOML as a configuration format personally, but it's at least more capable than cfg files. I'm really keen to know why you feel cfg support would be useful.

andrewshadura2• 3 years ago

To be honest, I feel like setup.cfg hasn’t been given a chance to prove itself useful. Just as it was made possible to use setup.cfg only (without setup.py), it’s been almost immediately taken over by pyproject.toml. However, for those who actually migrated their projects to setup.cfg, migrating once again doesn’t bring that many benefits (but e.g. loses compat with flake8).

Besides, setup.cfg’s syntax is slightly more elegant, or at least I think so :)

By the way, why do you think it is more capable? As far as I know, setup.cfg has some feature toml (intentionally) doesn’t support (not that they’re useful in this context anyway).

snowe20104• 3 years ago

Is there seriously another standard being set up in the Python community? When will it end? That’s not a rhetorical question, I’m seriously asking when the community is going to actually solve the tooling problem without making everything worse with each successive update.

r9295• 3 years ago

Using a highly opinionated package manager with barely any tests? Supply chain attacks exploiting package managers are actually a thing to be extremely scared of. https://news.ycombinator.com/item?id=34836275

epage• 3 years ago

No Core Proprietary Stuff: Rye (with the exception of it's own tool section in the pyproject.toml) uses standardized keys. That means it uses regular requirements as you would expect. It also does not use a proprietary lock file and use pip-tools behind the scenes.

The use of `pip-tools` to generate a `requirements.txt` as a lockfile is a big negative for me. Last I looked, they do not generate a cross-platform lockfile. In theory, you could still use `requirements.txt` for a cross-platform lockfile; it just might be a mess for doing incremental updates like when adding a new dependency.

Rye: Flask author's new project for Python bootstrapping

Related Stories