PEP 594 – Removing dead batteries from Python's standard library

https://www.python.org/dev/peps/pep-0594/

by jorshman • 7 years ago

401 222 7 years ago

python.org

avar• 7 years ago

Relevant discussion 4 days ago, "Python's batteries are leaking": https://news.ycombinator.com/item?id=19948642

jstimpfle6• 7 years ago

https://docs.python.org/3/library/fileinput.html

I've never had a strong opinion about "batteries included", but boy there are some weird ones in there...

ehsankia2• 7 years ago

Especially when you start looking at the code for some of these

https://github.com/python/cpython/blob/master/Lib/imghdr.py

Znafon2• 7 years ago

To be fair, it dates from 1992. I don't think PyPi existed at the time, it's rather impressive the Core team supported this for so long.

tomjakubowski• 7 years ago

Ruby includes something similar, ARGF. It's pretty useful over there. https://thoughtbot.com/blog/rubys-argf

jstimpfle6• 7 years ago

Well, maybe fileinput is meant to be used for making shell one-liners using the -m switch or so, like in the link you posted, or like perl/awk. Does that play nice with Python's block syntax, though?

mftrhu• 7 years ago

Why do you think fileinput is weird?

jstimpfle6• 7 years ago

The functionality is almost less than trivial. It's like, trading 3 lines of super straightforward code for one line of import statement and a library dependency.

Pretty sure it tries to emulate awk's programming model, but even if I had a use for that I'd rather write these 3 lines of code myself - so the code is actually clear without looking at documentation, and so that it can be modified easily.

ehsankia2• 7 years ago

I honestly didn't realize there was so much random stuff in stdlib.

varelaz2• 7 years ago

There are much more of it. I think around 1/3 of standard library is legacy and burden for python. I recall there was a PyCon talk about why you may not want to be a part of standard library. Release process and support of old versions is very strict and heavy, it could delay releases and API change dramatically. You can look at situation with urllib for example, there are 3 versions of them, but most popular is requests 3rd party module.

icodestuff• 7 years ago

I'm surprised at the removal of aifc. AIFF is still used somewhat in macOS.

coldtea• 7 years ago

Well, they don't remove AIFF tho, just the lib -- which presumably nobody uses...

schrijver• 7 years ago

AIFF is used widely in audio production software; for exporting uncompressed audio many application default to AIFF. It’s the most universally useful lossless format; WAV support is ubiquitous too but AIFF has better metadata. FLAC and ALAC are good alternatives today, but not on the same level of support yet. For example, for purposes of price differentiation, Pioneer reserves FLAC and ALAC support for their most expensive digital turntables.

If you’re writing a Python script that outputs an audio file, outputting to AIFF seems like a save bet. Then again I’m not sure how many people are actually using the module from the stdlib--but neither does the PEP’s author, it seems. His level of research was asking his friends on Twitter:

https://twitter.com/ChristianHeimes/status/11302577994753351...

dlbucci• 7 years ago

This seems like a good idea to me, but what are the chances that something like this actually lands? I know Guido did not seem to like this approach (to put it mildly), although I'm not sure he still has veto power having stepped down as BDFL.

Also, the changes sound like a major version bump to me, but I doubt people want that to happen after Python 2 to 3...

stOneskull• 7 years ago

can you step down from something that is for life?

cortesoft• 7 years ago

I mean they can't really stop you

knolax• 7 years ago

Kings abdicate all the time.

Macha• 7 years ago

I'm not sure Guido's argument was with the argument indicating some stdlib parts are outdated as the argument that new library features are not worth using because twisted wants to support python 2

jsmeaton• 7 years ago

Guido can not veto, but the PEP still needs to be confirmed by the steering council[0] of which Guido is a member.

[0]https://www.python.org/dev/peps/pep-8100/#results

yingw787• 7 years ago

This seems like a vast swath of things to deprecate with one PEP, but it looks like the author of the PEP talked with a large number of core developers about issues that they had with the stdlib and decided deprecated packages that way, which I think should be good for approval of the PEP.

I have a couple of questions:

- I realize there may not be a fork with this PEP implemented, but how might this impact Python's local relative build time, and how might that convert over to the build pipelines? Drastically faster build times would be really nice.

- Are Python 2 -> 3 migrations for stdlib packages mostly rewrites or mostly tacking on compatibility layers like `from __future__ import unicode_literals`? If stdlib packages were updated with Python 3 syntax, it might indicate sustained demand for said package going forward. I'm not sure.

Znafon2• 7 years ago

Are Python 2 -> 3 migrations for stdlib packages mostly rewrites or mostly tacking on compatibility layers like `from __future__ import unicode_literals`?

Packages in the stdlib don't need compatibility layer since they are always used with the Python version they were written for, it's motly incremental rewrites.

Regarding the build time, I don't think most of the time is spent testing those parts of the stdlib.

enedil2• 7 years ago

I'm not the OP, but I suspect that what was means was backporting features to py2.7, which presumably use some new Python 3 which haven't been built in the old language version.

kbumsik• 7 years ago

how might this impact Python's local relative build time

stdlib usually written in Python, not in C. So I guess it won't affect the build time much. It will reduce test time but I'm not sure.

monocasa• 7 years ago

MSI support removal seems premature. Can you do unsandboxed stuff from an AppX? Also can you install AppXs on Windows 7/8?

chungy• 7 years ago

How many people actually used Python's MSI library?

It was Windows-only (presumably a wrapper around the native tooling), and was primarily for creating Python's own installer, which apparently doesn't get built as an MSI anymore.

ptx3• 7 years ago

I use it for packaging my Python application. But it seems pretty low-level, so I suppose I could call the win32 API directly through ctypes when they remove it.

WorldMaker2• 7 years ago

You've been able to do Win32 stuff in an AppX since early in Windows 10 history, it's mostly "unsandboxed". MSIX, the "new" AppX, has Windows 7/8 support for Win32-focused installs in preview.

Semi-relatedly, Microsoft has already been exploring with the Python team MSIX deployments for the Python interpreter itself (you can even find it in the Microsoft Store now on Windows 10, and typing `python` on a command line in recent builds of Windows 10 will auto jump you to the Store if you don't have a python.exe in your PATH/installed).

WorldMaker2• 7 years ago

(Also, getting in the weeds, the "mostly unsandboxed" refers to that MSIX installs will do a few things to insure at all costs clean uninstall, including system folder virtualization/redirection, registry virtualization/redirection, and similar techniques. Windows and MSI has been doing a lot of this sort of thing under-the-hood silently since XP as a part of compatibility work, MSIX just does it more obviously/automatically/always.)

leothekim• 7 years ago

Serious question - Why put off DeprecationWarnings to 3.9? If these modules are as dead as this PEP claims, then I'd wonder if Python scripts using such modules would be actively looking to transition to 3.x in the first place.

puetzk• 7 years ago

Because the 3.8 beta (and hence 3.8 feature freeze) is next week, and there's basically no chance of completing the PEP in time to make that cutoff.

korethr• 7 years ago

Hmm, should perhaps this be worthy of a major version number bump? Yes, it's not as large or as breaking of a change as the 2.7->3.0 transition, but by removing modules from the standard library, backwards compatibility is being broken. Now, perhaps most people don't use those features anymore, but for those people who do use and rely on these modules, a major version number change would be a welcome signal that their code will no longer work with the new version.

But, after the initial pain of the 2.7->3.0 transition, I doubt we'll ever see another major version number jump, even if a logical use of version numbers would merit it.

enedil2• 7 years ago

It's also quite possible, that the people who use these features use Python 2 anyway.

mjevans• 7 years ago

Does this mean we're going to have Python 2 installed on things forever? (Particularly anything that touches NNTP servers, as there's no replacement for that lib)

Mirioron• 7 years ago

Even if they don't bump up the major version number, people are probably still going to treat it as such. I think it makes sense to bump it up.

m4634• 7 years ago

but by removing modules from the standard library, backwards compatibility is being broken.

API 101 - there is very little cost to leaving them in, but a hidden major cost to having them disappear, usually for non-developers.

By the way, "batteries included" is one of the BEST features of python.

Have you tried to fix something in your house with a "homeowner's toolkit" which is usually something like a hammer, pliers, 2 screwdrivers, a putty knife and a few more basic tools?

It is REALLY tedious, like writing a C program with a few basic tools like stdio and ctypes.

More languages need "batteries included", maybe like Perl.

I think if the cost of deploying a script is 1, deploying it + a dependency is literally something like 100x. You have to make assumptions about all the environments the script will run in, and they are usually wrong.

arcticbull2• 7 years ago

By the way, "batteries included" is one of the BEST features of python.

None of these libraries need to be nuked from existence as a result of this change. I'd wager they'll move into PyPI modules so that teams relying on them could safely continue to do so.

Have you tried to fix something in your house with a "homeowner's toolkit" which is usually something like a hammer, pliers, 2 screwdrivers, a putty knife and a few more basic tools? It is REALLY tedious, like writing a C program with a few basic tools like stdio and ctypes.

That's one extreme, but I don't think that's what's being proposed. The proposed model is closer to what Rust does today, where the core is slim, opening up new potential use cases, and the more complex functionality built on top of it is left to the community to maintain.

Take a look over some of the modules they're deprecating, like smtpd. What kind of standard library requires an SMTP daemon built in? That's akin to a homeowners toolkit including a planishing hammer [2] for some reason.

I think if the cost of deploying a script is 1, deploying it + a dependency is literally something like 100x. You have to make assumptions about all the environments the script will run in, and they are usually wrong.

Depends on how it's done, honestly. Check out this Rust "scripting" system [1]. It has full support for third-party crates.

[1] https://github.com/DanielKeep/cargo-script

[2] https://en.wikipedia.org/wiki/Planishing

kllrnohj2• 7 years ago

API 101 - there is very little cost to leaving them in

That's not true as newcomers will typically bias towards bundled modules instead of 3rd party ones. If it's well accepted that the bundled modules are bad in some way then you are encouraging further use of them by leaving them in. That's real cost, and not just on the Python maintainers side of thing.

gugagore• 7 years ago

C also doesn't have a standard package manager. I think that's the big problem.

Built-in functionality is less important if you can easily use packages.

labster• 7 years ago

By the way, "batteries included" is one of the BEST features of python. More languages need "batteries included", maybe like Perl.

Agreed with both points, the lack of batteries makes Lua quite annoying for me personally. Python and Perl are much more comfortable because you don't have to hunt down the most common operations.

Of course, Perl 6 has decided not ship batteries included, replacing batteries with a fusion reactor instead.

bigiain• 7 years ago

Have you tried to fix something in your house with a "homeowner's toolkit" which is usually something like a hammer, pliers, 2 screwdrivers, a putty knife and a few more basic tools?

Sure. But did you read the list of proposed deprecations?

It's not like that homeowners toolkit becomes any more useful when it's also got a buggy whip tool, a set of special allen-like keys that only open the case of a TRS80 or a Commodore64, and one of those picks for getting stones out of horses feet.

Most of the proposed things to take out seem obviously "right" to me. AIFF sound file support? MacOS9 binhex? CGI? _Maybe_people are still using those things, but it seems to me like they're edge-case enough to let them know now that they'll need to do "pip install aifc" or "pip install cgi" sometime after late 2021 if they update their python installation (which they can choose to put off until at least 2024 or so, just like all of us who're still sometimes using Python 2.7...)

c3534l• 7 years ago

The approval process for getting libraries okayed for production is a massive pain in the ass, and for good reason. Without a batteries included language, it'd be easier to write all those tools in-house than hope that maybe in two months I can get a library out in production, even though it's just JSON parsing or something. The "this is an official part of the language and held to high standards, guaranteed to be suitable for poduction and needing no approval beyond already being approved to use the language" makes Python way more preferable IRL than languages I love, but don't even include a random number generator in the standard library.

sago• 7 years ago

Python minor versions allow things to be removed and thus 'break backwards compatibility'. There is a process for warning of upcoming deprecation then removing it after that. For modules this is documented in PEP 4 (from 2000). Similar deprecation schedules are used by other important ecosystems such as Django.

We'd be on way more than Python 3 if the major version was bumped for each one.

nitely• 7 years ago

This. There are plenty of backward incompatible changes in every minor version bump.

rozab• 7 years ago

some of these modules have actually been deprecated since before python 3.0

anoncake14• 7 years ago

But, after the initial pain of the 2.7->3.0 transition, I doubt we'll ever see another major version number jump, even if a logical use of version numbers would merit it.

Do you think the transition would have been less painful if 3.0 had been called 2.8?

dagw2• 7 years ago

If Python 2.8 had come out and failed to run almost all code written for Python 2.7 then that probably would have been truly disastrous for the language.

mleonhard• 7 years ago

I love this. I hope more and more cruft gets removed from our languages and tools.

larkost• 7 years ago

My one objection to this is the `imp` module. As far as I can tell it is the only way (short of `sys.path` modification) to specifically load a module found at a specific path. For testing systems I use this quite extensively... Looks like I need to make some comments...

saila2• 7 years ago

You can do that with importlib too:

    from importlib.machinery import SourceFileLoader

ChrisSD• 7 years ago

That's been deprecated as well. The current docs say you should do this

    spec = importlib.util.spec_from_file_location(module_name, file_path) 
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)

https://docs.python.org/3/library/importlib.html#importing-a...

tgb• 7 years ago

I use imp.reload(my_module) all the time when %run-ing things from ipython that import my_module after editing it. What's the alternative for that use?

carreau• 7 years ago

In[1]: %load_ext autoreload In[2]: %autoreload 2

maxnoe2• 7 years ago

Using importlib.reload

saila2• 7 years ago

Just one data point here, but with the exception of the cgi module a long time ago, I haven’t used any of these modules in the past 15 years of web programming, ETL scripts, or data processing. In the keep list, I’ve used fileinput once that I can remember.

0xADEADBEE• 7 years ago

This has been a long time coming and I'm surprised it's taken until 2019. A better time to have done this would have been back at the 2-3 migration; I'm not sure the community will be able to survive another transition like that, but hopefully lessons have been learned from the previous schism. A step in the right direction for sure!

jancsika• 7 years ago

audioop is an interesting little thingy.

For example-- why does "add" add two fragments, but "mul" multiplies one fragment by a scalar value? Was the assumption that "mul" will probably just be used for attenuation?

It would be fun to build a tk-based audio app around audioop for the sole purpose of complaining about the deprecation of this module. :)

Alex3917• 7 years ago

The only thing I worry about is the uu module. Even though 99.99% of people will never use it, there are also probably literally millions if not billions of pieces of content that are encoded using that standard. I realize it's only a few lines of code, but from an archival perspective it seems like there's something distasteful about making it harder for future generations to access content that may have historical value. At least send a quick note to the library of congress or the archivist community first to get an idea of to what extent this stuff is getting used.

roblabla• 7 years ago

This module is pure python. It looks like if people rely on it, they could easily put the code in a pypi module, and maintain it there. It doesn't really make sense to keep it in the core if it has no dedicated maintainers.

takeda• 7 years ago

That code doesn't need any maintainers, it is not like UU encoding is evolving or really complex.

guitarbill• 7 years ago

That's a bit dramatic. Our knowledge of the encoding doesn't go away because Python removed support. Old versions will continue to work. The code is preserved in VCS. And finally, you/someone can release a 3rd party module to PyPI to fill the gap.

naniwaduni2• 7 years ago

The useful functionality (encoding/decoding) is covered by binascii, which uu in fact uses internally.

What the uu module proper provides is a pair of awful file-to-file interfaces. Default to spitting the decoded file directly to disk at with the name in its header is, uh, questionable.

owaislone• 7 years ago

Isn't it just being removed from the standard library? It can still exist on pypi especially if it is as important as you suggest, someone will definitely host it on pypi.

masklinn4• 7 years ago

As noted in the PEP, the uu codec itself is in `binascii` and not moving anywhere. `uu` is just a (pretty awful) high-level interface.

coldacid2• 7 years ago

Considering how many different types of data have and can be stored in IFF chunks (and their order-swapped RIFF counterparts) I'm almost insulted by the PEP author considering IFF to just be "an old audio file format".

I think my Commodore/Amiga persecution complex is acting up again.

oblio• 7 years ago

Wasn't the last Amiga sold in 1996? That's 23 (!) years ago.

coldacid2• 7 years ago

You can still buy Amiga platform hardware, although it's not the classic Commodore-era stuff but rather the modern PowerPC based stuff.

peterwwillis• 7 years ago

If we're going to lean more on PyPI, it would be nice to clean it up some. There's a lot of cruft that makes it very hard to find the right module to use to write a new project with. And it would be nice if package naming convention were more standard, and suggested extending existing modules rather than writing entirely new ones. If you want HTTP functionality, you use the "requests" package, which uses urllib3. Why not io::net::http::client, extending io::net::http, and so on?

torlakur• 7 years ago

Well, that sounds like the package names (and namespaces) that the Perl 5 community have landed on with CPAN modules :)

chubot2• 7 years ago

Since the parser module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the parser module is scheduled for removal in 3.9.

Hm interesting comment. Does anybody know what the new approach to parsing Python in 3.9 is ? I searched python-dev@ but couldn't find any references to it.

I found an interesting tidbit about Rust "switching" from LL to LR here:

https://www.reddit.com/r/ProgrammingLanguages/comments/brhdt...

And I noticed some rules in Python's grammar that are awkward in LL parsing (set and dict literals, and comprehensions).

I wonder if those things motivated the switch? They certainly work though.

why_only_15• 7 years ago

It appears that the ast module is the recommended one now.

ivoflipse2• 7 years ago

They're having the discussion on Discuss: https://discuss.python.org/t/preparing-for-new-python-parsin...

chubot2• 7 years ago

Thanks a lot! That led me to find this November thread:

https://discuss.python.org/t/switch-pythons-parsing-tech-to-...

And I posted here about it:

https://www.reddit.com/r/ProgrammingLanguages/comments/brz2y...

It looks like the set and dict literals I noticed weren't so much the motivating use cases, but even more fundamentally assignments and keyword args!

scrollaway2• 7 years ago

Huh interesting. I've been using parsimonious for cases like this, didn't know python had a stdlib for it...

nomel• 7 years ago

A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.

Python, lean and mean? Seems like an incredibly niche use case to restrict the python community to.

yjftsjthsd-h• 7 years ago

At that point I'd expect to be using micropython anyways.

jedberg• 7 years ago

My only concern with this is that they won't be made available on Pypi for those that still need it. They will pull out the code but they expect the community to package it back up and make it available. Their hope is that the community will improve it before making it available, but I suspect the reality will be that the moment it is deprecated, someone will just go through and make all of them available in Pypi exactly as is.

JeremyBanks• 7 years ago

Why is that a concern?

analognoise• 7 years ago

Again, Tkinter isn't on the hit list and I'm glad.

k4ch0w• 7 years ago

Someone linked to Amber Brown's talk and I thought this point would be important for discussion. I think there is something that isn't being considered here when it comes to relying on PyPi.

3PP(Third Party Packages) issues are responsible for a lot of application security vulnerabilities today. Every large enterprise organization has no idea what packages a developer is pulling onto their laptop and into their codebase.

As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, instead of someone who just git pushes code every night. You have no idea who is on the other end of that push too. It's too hard to keep track of changes and causes us to have to pin packages and versions that have been ok'd for use instead of a new release of Python. You have no guarantee the security engineer who reviewed the code didn't miss anything either.

jahewson• 7 years ago

There’s really two separate concepts here: review and the standard library. Why not separate concerns and have a review process that isn’t coupled to the standard library? There’s no reason why any PyPI package can’t have meaningful reviews published.

The question that you’re ultimately seeking the answer to is “what code has been reviewed and which reviewers do I trust?” - lots of ways to solve that.

anoncake14• 7 years ago

One reason for this proposal is to save manpower. Separating concerns doesn't help with that.

Kalium• 7 years ago

There’s no reason why any PyPI package can’t have meaningful reviews published

You are completely correct. There's no reason at all why any random PyPI package can't have meaningful reviews published. I would go so far as to say that this is true without any changes to the standard library or current processes at all.

With that in mind, I'm thinking about all the various packages my colleages use. I don't think I've seen published reviews for an appreciable fraction of them in any language. This suggests that perhaps enabling reviews might not be the hard problem to solve here.

This thought process also hilights to me that the major advantage of a stdlib is that you have a higher degree of assurance that its contents have seen meaningful review by multiple sets of eyes. It's not just the potential for review that matters, it's the degree of assurance.

alexchamberlain2• 7 years ago

I wonder if something like Arch Linux's main repository vs AUR would work?

zwp3• 7 years ago

“what code has been reviewed and which reviewers do I trust?”

Thank you for putting this so succinctly. There are other supply-chain problems in decentralizing control of the modules though:

Modules might only have a few people or one person responsible for them. Consequently suspect commits by a malfeasant or compromised (hacked) team member might not be noticed. Maybe this is a variant of the old saw "with enough eyes all bugs are shallow" but my gut feeling is that commits to Python core will get more inspection than those to a small 3rd party module. Today, modules get some review for free, just by being part of stdlib. Even if I know and trust @jdoe I have less assurances that they didn't get phished and their repo tweaked.

Also trusted team members/organizations may change over time. The browser extensions world is the poster child for this, where we've seen not only similarly-named malicious extensions posted to stores but also once-legitimate extensions quietly purchased and subsequently subverted by bad actors.

This is one example (adware: could've been much worse): https://www.bleepingcomputer.com/news/security/-particle-chr...

I like the idea of a review process but I have a hard time imagining a crowd-sourced system that wouldn't get gamed. We have a "dissolution of responsibility" problem: millions of companies rely on these components but have no explicit responsibility of care. Perhaps that needs to change, somehow.

tialaramex• 7 years ago

I suspect you're gravely over-estimating how much value there is in the dustier "batteries included" modules over an out of tree component for security.

The problem is that security assumptions change. If you're looking at some code in 1995 that accesses an API, it seems reasonable that it disables validation for the SSL library it brings in to access an HTTPS URL. Handling that properly would have been a huge pain, faster is better.

But in 2019 you'd be appalled, how is this thing not depending on certifi and requests? As it is it's wide open to a MitM attack and you'd get blind sided.

A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy() and in 1970s Unix I'm sure this was invaluable. But obviously today "copying strings into fixed width buffers with no terminator" is essentially never what you wanted, but people assume based on the name that strncpy() will do some other thing they do want, and insert a security defect into their code.

geezerjay• 7 years ago

A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy()

Arguably, what makes C the perfect example is not necessarily its standard library functions per se but the language's official string data type.

ivoflipse2• 7 years ago

Then maybe those large enterprise organizations should start paying for the long mature process with experienced developers

Scarblac• 7 years ago

As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers,

Yes, but do you like funding them? This is all freely downloadable open source software.

newen• 7 years ago

Plenty of people fund the Python Software Foundation. Their revenue is around 3 million dollars a year.

geofft• 7 years ago

As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, instead of someone who just git pushes code every night.

This is an appealing sentiment, but you will get hacked if you rely on it.

Python 2's standard library didn't do SSL certificate verification by default until the fairly controversial PEP 476 from late 2014, which changed the default in a point release of Python 2.7. Through 2014 (and for some time later, given how slow it is to get new upstream releases through distros onto someone's computer), the standard thing to do to use HTTPS securely was to use requests, a module developed by someone who got commit privileges taken away on another of his projects for making multiple releases a day. You were more secure with his code than with the standard library.

I think you have no more idea who's on the other end of pushes to the standard library than who's on the other end of pushes to PyPI. (Which is to say, in part, that you can equally well have an idea of both of these if you put effort into it. I've met the maintainers of several of my third-party dependencies at PyCon.) Something being in the standard library doesn't mean it has a higher class of developer behind it. There's more overhead, but it doesn't mean there's more maturity; many third-party module developers are more experienced (either in their field, if it's something like crypto, or just in general as responsible developers) than standard library developers. And if anything it means that security updates are slower and rarer because the process is more painful.

Consider the argument of the 2009 paper "Security Impact Ratings Considered Harmful" https://arxiv.org/abs/0904.4058 (disclosure: I'm a coauthor). Whether people find a vulnerability important enough to patch has little correlation in practice with how exploitable it is. And the only thing that gets regular attention is the latest development version of the code. If some module was significantly refactored or reimplemented, no upstream developer is looking for bugs in the old version. So, if you want to be safe and you haven't personally both audited and fuzzed the code you're running, you actually want to be running the latest released version of that code, regardless of whether someone stuck a CVSS on the old version yet. I've met very few companies who can upgrade to the latest Python 3.x minor release promptly when it comes out (and several who are still on Python 2!). I've met many companies who can pip install the latest versions of their dependencies without too much trouble, though.

anoncake14• 7 years ago

Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of xml. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used.

Developers don't have a compelling reason to use 3rd party modules instead of the standard library. Therefore they don't. You consider that a problem and want to encourage them to use 3rd party modules more.

In short, you want developers to use 3rd party libraries because there is no compelling reason to do so?

A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.

That's a silly reason. Just make a separate distribution with a stripped down standard library.

hoseja• 7 years ago

They want to stop supporting the old modules because to keep them in working order is almost useless and uses too much manpower.

anoncake14• 7 years ago

That's one of three reasons. https://www.python.org/dev/peps/pep-0594/#rationale

carlmr3• 7 years ago

> A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
That's a silly reason. Just make a separate distribution with a stripped down standard library.

True, you could have a std and core library. I think Rust does this for embedded. IMHO if you take away the large standard lib, except for pandas I have no more reasons to use Python. I don't like the language that much, but the std library is pretty good for scripting when you need something on a target system that only has standard Python.

int_19h2• 7 years ago

To give a specific example, there's a lot of headache that new Python users could be spared if only they used requests instead of urllib2. But they don't know that it's compelling if they never go look, and just stick to stdlib.

anoncake14• 7 years ago

Good point. However, there's a box saying that request is recommended at the top of the urllib.request docs.

https://docs.python.org/3.8/library/urllib.request.html#modu...

pfranz3• 7 years ago

I'd love if something like requests was included along with Python. But requests uses urllib3, so it's not like you could get rid of that. The main page for requests says they're working on a v3...you wouldn't want that kind of churn in the stdlib. Personally, I kind of like the structure of low-level libraries and fancier high-level libraries I use most often.

So I'm not quite sure what is the best thing to do. It sounds like requests III is a Python3 forward version. So it might make sense to ship that?

Python has decided to include packages developed outside of the stdlib before; I think json, pathlib, and maybe optparse.

jl6• 7 years ago

Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages.

I’ve been out of the Python loop for a few years, but my last impression was that packaging and distribution of Python modules was far from a solved problem. Has this changed?

carlmr3• 7 years ago

No, it's still one of the worst package management systems out there.

I'm always wondering if it has to do with the language and or just the package manager itself. Rust's cargo, Node's npm and probably quite a few others exist that work amazingly well.

alexchamberlain2• 7 years ago

What makes it the worst? Is it non-pure Python extensions, or is there something missing for pure python packages too?

bow_• 7 years ago

Why do you think it is the worst? If you don't consider Python, what would you then think is the worst?

Genuinely curious. I use Python daily and I rarely encounter problems with it. There is some getting used to in the beginning, but I went through a similar phase when I started using npm and cargo as well.

What I can say is that I had to go through a lot of experimentation myself to arrive at the tools I use now (pyenv + Poetry). And if anything, maybe the lack of one way that is adopted by everyone in the community is the problem.

motles• 7 years ago

"Amazingly well" has become 500MB node_modules directories in your source tree?

sametmax7• 7 years ago

npm is worse IMO.

First it forces the node_module in your repository, which mean you have to configure tons of tools to ignore it or you'll have a bad time.

Then there is no way to have comfortable local command, so plenty of time you gotta have sudo -g. Recent npm version now have a tool for that but the experience is meh.

Plus there is no builtin tool to manage several versions of node, so you gotta add nvm on top of that.

And then if you work on the front end, you can't even use whatever you npm installed stuff, you gotta setup the entire webpack shenaigan, making it literally the hardest stuff to setup. It's so complicated we rely on black box such as create-react-app to do the job for us.

stackzero• 7 years ago

The problem is there's no clear way to get started with setting up a python package. Every other day someone ask a question on SO about it and the first google result is outdated distutils IIRC

d0mine• 7 years ago

Have you looked at https://packaging.python.org/

sametmax7• 7 years ago

It's actually pretty ok now, espacially with .whl for pretty much everything avoiding compilation and venv being provided with recent Python. The problem is the sheer amount of bad information you find online, so that people take hours figuring how things work.

E.G:

- people telling you to sudo pip install

- people telling you to use virtualenv instead of venv

- people not telling you to do python -m

- people not telling you about setup.cfg

pas5• 7 years ago

Pipenv!

aitchnyu• 7 years ago

Some batteries are surprising https://docs.python.org/3/library/

difflib - Text diffs, even html output

textwrap - obvious no?

rlcompleter - autocomplete symbols and identifiers, used in interactive mode

pprint - for printing complex data structures with indentation

reprlib - repr with traversal depth and string size limits fraction - Fraction('-.125') becomes Fraction(-1, 8)

statistics - averages and deviances tempfile — Generate temporary files and directories

glob — Unix style pathname pattern expansion

gzip, bz2, zipfile, tarfile - obvious

configparser - ini-like format

secrets - use this instead of random for safe cryptography

sched — Event scheduler

turtle — Turtle graphics

shlex — Simple lexical analysis

webbrowser — Convenient Web-browser controller, for the antigravity module!

_fbpt• 7 years ago

LZMA is better than gzip i think.

Also secrets deserves to stay in the library, Python has hashlib and it should have a secure RNG by default.

glob too (or Path.glob).

And I've used shlex for command-line-escape parsing (it might not have been the optimal solution?)

masklinn4• 7 years ago

LZMA is better than gzip i think.

LZMA provides much better compression at much higher costs. Generally speaking it's pretty strictly better than bzip2, not necessarily gzip (DEFLATE, really).

In my experience, zstd can be considered better than gzip/deflate (almost every time I tried it, it provided as-good-or-better compression at much faster throughput).

int_19h2• 7 years ago

Turtle graphics is really handy for teaching programming. And it's important for new users to have minimal friction when setting things up, so having it in stdlib does make sense.

Spiritus• 7 years ago

Surprising how? I've used most of the "batteries" you mentioned in the past year.

sametmax7• 7 years ago

textwrap - used by Python itself for docstring.

rlcompleter - used by Python itself for the shell.

pprint - trememdously useful for debugging.

statistics - added recently because people kept rewritting mean() functions that were broken

tempfile.gettempdir(), glob("*.ext"), gzip, bz2, zipfile, and tarfile are kinda mandatory for a language you use massively for scripting

secrets - we had random. People used it for security all the time and created stupid security holes. So we added this.

webbrowser — fantastic module that open a new tab in the default browser from any code. It's one file, so for the value it provides, I'm happy it is here.

robobro• 7 years ago

noo not the cgi module :(

zestyping• 7 years ago

I was wondering about that too. CGI is still just about the easiest and most universally supported way to serve dynamic web content from a Python script.

The complaint is that the module is designed poorly, which is fair; to remove CGI support without any "batteries included" replacement seems a bit of a shame, though.

skykooler• 7 years ago

Especially as, if you're in a situation where you have to use CGI rather than a better server worker solution, you probably also don't have access to the environment to install pip packages.

mjw1007• 7 years ago

The cgi module doesn't actually contain anything CGI-specific. It dates from a time when "CGI" was a sensible shorthand for "handling http requests".

What's actually in the module is a rather clunky, but serviceable, system for processing html form data.

ptx3• 7 years ago

You can use wsgiref.handlers.CGIHandler instead, with the advantage of being able to easily move the application to some more appropriate WSGI environment later if needed.

draegtun• 7 years ago

You have to keep moving forward with the times!

Perl removed CGI from its core back in 2014 - https://perl5.git.perl.org/perl.git/commitdiff/e9fa5a80

hyperion2010• 7 years ago

This really, really, bothers me. The rationale seems absurd. Is there no more uu encoded data in the world? Throwing this code away is a recipe for a future where some data is never recoverable. Python is a swiss army knife. The imagination that pypi is going to survive forever and be accessible forever is absurd. A programming language that throws away the past is doomed to become a passing fad. If the code is old who cares as long as it still works?

masklinn4• 7 years ago

Is there no more uu encoded data in the world?

The uu codec is provided by the binascii module. The uu module is a "high-level" interface for conversion between binary and uuencoded files ("-", paths, or file-like objects).

Basically uuencode(1) / uudecode(1) reimplemented in python.

A programming language that throws away the past is doomed to become a passing fad.

That's complete nonsense.

If the code is old who cares as long as it still works?

Keeping it working is a maintenance burden. Why keep it if it's worthless?

varelaz2• 7 years ago

All of removed modules should go to PyPi no one is going to kill them completely. Python keeps a lot of legacy. I'm also pretty concerned with builtin robots.txt parser for example. There are much better PyPi libraries which are well maintained and have better API.

gtirloni• 7 years ago

It feels like this PEP should have more practical examples of how these modules are impacting the work of core Python developers. Some number of bug reports or time spent dealing with them.

noname120• 7 years ago

Dupe: https://news.ycombinator.com/item?id=19971131

kd5bjo• 7 years ago

3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process.

Why should this be fast-tracked outside the normal process? I can’t imagine any of these removals are urgent.

imglorp2• 7 years ago

Nobody talks much about the original goal of "There should be one-- and preferably only one --obvious way to do it." -- PEP 20 (!)

Back when, I lurved me some Perl TMTOWTDI as much as the next guy--plenty of uses for such a thing in a Swiss army chain saw--but then Python had a good (or at least different) answer to that by reducing the language load and focusing more on the problem.

So tell me, how many string formatters do we have now? Could we please decruft and dump some of them while we're at it? % or f' or .format or whatever, but please just pick one and get the rest out of my face.

https://www.python.org/dev/peps/pep-0020/

https://en.wikipedia.org/wiki/There%27s_more_than_one_way_to...

simonh3• 7 years ago

It's not that it's never been possible to do things other ways in Python. The obvious way to do stuff is to use the latest module or feature. The obvious way to do string formatting is now f-strings.

dhuramas• 7 years ago

In an enterprise-y environment, it's not always possible to keep upgrading to the latest. So assuming someone creates a project based on f-strings, that would be python 3.6+ only. Whereas using string.format will work across entire python 3+. Is that worth the upgrade- I'm not convinced.

imglorp2• 7 years ago

Until the newer way comes along and then there's n+1 ways.

aeturnum2• 7 years ago

how many string formatters do we have now? Could we please decruft and dump some of them while we're at it? % or f' or .format or whatever

To me, there's a difference between "one way" and "one interface." I agree we should depricate %-formatting in strings, but .format / f"" are different interfaces on the same functionality - which seems fine.

aequitas• 7 years ago

Except that f-strings are not lazy evaluated so you cannot replace something like:

  log.debug("foo bar %s", foobar)

with fstrings. And you end up with part of your team enforcing f-strings because of cleaner look, the other half .format or '%' strings and the rest annoyed when their code is constantly rejected in reviews because they don't know/care which standard to use where.

Waterluvian• 7 years ago

I think an 80/20 solution is documentation UX. Just take all these old modules and put them into an "old modules" section and preface that section with a short discussion on what this all means and what the wisdom is.

HelloNurse• 7 years ago

Many ridiculous suggestions, like the whole "data encoding" group (obsolete data formats don't need maintenance) and the various modules about old file formats (ditto).

Blackthorn• 7 years ago

They do need maintenance. Python updates cause code or tests to need updating due to syntax or other changes.

diydsp• 7 years ago

Old people stuff below:

I guess we're getting to the point where people now say things like "it was developed for TSR-80", where it is more correct to say "for the TSR-80." :)

The uu module provides uuencode format, an old binary encoding format for email from 1980.

while true, uu was also used extensively in usenet, at least throughout 1997. But as we learn later, noone has touched nntp code in 5 years! and there is no interest in porting nntplib to py3.

originally introduced for Commodore and Amiga. The format is no longer relevant.

not disagreeing. Just saying "ouch."

The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats.

Again, not disagreeing, but implies .wav is ancient. It's still in heavy use (they're keeping "wave"). And ancient is hyperbolic and slightly insulting, but I'm just going to let it go as humor in service of persuasoin. It's hard to let go sometimes when these formats provided so much joy and delight in such a pure fashion. Particularly AIFF vs WAV.

AIFF was for Amigas and Macs. Lots of great early techno used that format, while clumsy windows machines used .wav and lagged seriously in music software. Life then was more natural and computers were these shiny, productive boxes of creativity. We sat down at them for an hour at a time and expressed wonderful songs. Then got up and lived life. I don't need to mention that we now spend almost all day and all night inside a computer, at its keyboard or tablet, its screen and increasingly its headphones. Instead of augmenting our lives like a microwave, they "shelter" us.

I know the tools are objectively better now. They're more reliable, more productive and have more features. But I disagree that they provide more of what we actually need as human beings. Thinking like a designer, we need nourishing human experiences, challenges and interactions. We need opportunities to use our bodies physically, to run and cook, and especially to work, play and function together. But our health is objectively decreasing. Suicide is on the rise in several demographics. We're spending more time e-socializing so we eat more fast food. Obesity is on the rise. We can't even put the darn things down to drive.

Traditionally, human functioning meant organizing socially in groups that combined strangers' abilities. It required us to have patience and instruct each other. Increasingly, computers/apps/networks give us all the same concentrated superpowers. Now I can find any recipe I want, but I don't have anyone to cook it with. Our interactions are increasingly mediated, robotizing our communication. All this pausing caused by http-to-sql operations after every mouse click or sentence stilts us. It's almost as if our world is more black and white and lower resolution because we're experiencing so much of it through a densely mediated stack.

I'll also point out that in the old days, a library that supported five formats was a flipping miracle. But time marches on.

Having watched and been part of this whole python and internet phenomenon, experiencing it rise from nowhere and evolve from a neato thing into a mechanized, hyper-official "means of production" this is somewhat emotional for me. Guess I'm just getting old and I'll suck it up, but I just want to write these notes for people who wonder about what life was like before, or forget to wonder.

truth_seeker• 7 years ago

Very practical move!

I think few other legacy platforms Java(JDK), JS(NodeJS), Ruby etc. also in need of this kind of refactoring exercise and removing the old cruft.

PEP 594 – Removing dead batteries from Python's standard library

Related Stories