Pytz: The Fastest Footgun in the West (2018)

https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html

by djoldman • 3 years ago

195 121 3 years ago

blog.ganssle.io

I know some of the HN crowd are gonna wag their finger at me but the moment I have to deal with timestamps in Python, even for the most simplest of cases, I parachute in pandas. `pd.Timestamp` (and its friends like `pd.Timedelta`) in my opinion are the most powerful timestamp toolkit available anywhere.

To me it's very often easily worth the extra 50-150MB of install size that just works. I've used pandas for around 7 years now and it was inconsistent and buggy at the beginning. But these days, it's practically flawless. Also I'm aware that pd.Timestamp is built on top of many awesome libraries.

robertlagrant• 3 years ago

I'm not a Pandas person - is it possible to install the Pandas timestamp functionality as a smaller library, or do you need to install the whole shebang?

iknownothow2• 3 years ago

I had investigated if this was possible to do but I couldn't find a way to do it myself. It's possible that pd.Timestamp might be intricately tied interally to vectorization optimizations using numpy or others. Therefore separating out just the timestamp stuff from pandas would be a lot of work.

dagw2• 3 years ago

Python is in many ways the anti-JavaScript when it comes to packaging. Where JavaScript will have a installable package that contains a single trivial function, Python packages will bundle hundreds of functions, all vaguely related to an area, with no easy way of just installing what you actually need.

nerdponx• 3 years ago

My experience is that Pendulum does a good job of handling date/time data as a drop-in replacement for the standard library classes.

FObersteiner2• 3 years ago

pd.Timestamp is built on top of many awesome libraries

... pytz for example ;-)

orthoxerox• 3 years ago

Yes, and? Doesn't mean it inherits all the usability flaws of pytz.

marcosdumay• 3 years ago

As long as it hides the footguns, you are good to go. Pytz is not technically flawed.

irjustin2• 3 years ago

Pendulum is quite good[0]. Relatively intuitive interface.

When I left Rails, this is one area I really missed.

Time.use_zone("Singapore") { (Time.zone.now - 3.days).beginning_of_week }

This readable line in datetime utils or even pendulum, is such a pain.

[0] https://github.com/sdispater/pendulum

ttymck• 3 years ago

How does the rails example work? It must be using a global/shared state?

wussboy• 3 years ago

I've been using pendulum heavily in production for over a year now and it has made my life very much simpler. It's a great library.

jstx1• 3 years ago

Handling timestamps well was one of the motivations for building pandas in the first place (Wes McKinney started pandas while he was working at an investment management company).

arcanemachiner• 3 years ago

Django has similar functionality:

from django.utils import timezone

now = timezone.now()

later = timezone.now() + timezone.timedelta(minutes=1)

noirscape• 3 years ago

This article is correct, although slightly outdated - Python has been shipping its own timezone library called zoneinfo to solve this specific issue[0]. On the various *nix systems it just uses your systems timezone data, but it will fall back on the tzdata pypi package it that's not available (for example on Windows you need it if I'm not mistaken.)

[0] https://docs.python.org/3/library/zoneinfo.html

jwilk• 3 years ago

New in version 3.9.

lexicality• 3 years ago

Available to all in a backport: https://pypi.org/project/backports.zoneinfo/

8organicbits• 3 years ago

But 3.9 is no longer new. If you're still on 3.8 you're running out of time to upgrade as official support is ending next year.

The project has also been backported to 3.6+, so you can still use it if you're stuck on a legacy/unsupported version.

See https://www.python.org/downloads/ and https://pypi.org/project/backports.zoneinfo/

vesinisa• 3 years ago

Interesting. I got to thinking why a negative fold (e.g. -1) is not used to indicate local times that do not exist due to standard time to DST transition (moving clocks forward in spring). It is in fact discussed by the PEP: https://peps.python.org/pep-0495/#strict-invalid-time-checki...

The PEP provides instructions for detecting invalid times (that work as long as you use the standard library zoneinfo database.) In short - for invalid times, the UTC offset will be different for different fold versions (1/0) but with converse order than for times that exist doubly:

    # Non-existent wall time (std -> DST):
    >>> print(datetime(2023, 3, 26, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=0).utcoffset())
    0:00:00
    >>> print(datetime(2023, 3, 26, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=1).utcoffset())
    1:00:00
    
    # Ambigious wall time (DST -> std):
    >>> print(datetime(2023, 10, 29, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=0).utcoffset())
    1:00:00
    >>> print(datetime(2023, 10, 29, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=1).utcoffset())
    0:00:00

karmicthreat• 3 years ago

Pytz throwing an exception on ambiguous datetimes was the first footgun I discovered with it.

masklinn• 3 years ago

That is the opposite of a foot gun.

quickthrower2• 3 years ago

A foot nerf gun

sumtechguy• 3 years ago

That depends if you got it into production or not first :)

bhargav• 3 years ago

The title of this post claims that pytz is the fastest footgun in the west, by which I mean that pytz is a quite well-optimized library, and historically it has been faster than dateutil.tz

Wat. The author has no idea what a foot gun is.

Pytz is a must because using the standard library and timezone manipulations often leads to bugs. There are nuances around “naive” date times where one can easily just add a fake timezone value, not realizing that the time didn’t actually get localized to the timezone.

There are other libraries such as Arrow and Pendulum worth checking out but pytz is probably enough.

roelschroeven2• 3 years ago

Did you read the article? The author explains exactly why they think Pytz is not enough, and why they think dateutil.tz (not the standard library) is a better alternative.

pganssle3• 3 years ago

To be clear, when I wrote this article, I had not yet created the `zoneinfo` module in the standard library. All the recommendations for `dateutil.tz` apply to `zoneinfo` as well, except that `zoneinfo` is also faster than `pytz`.

Joeboy• 3 years ago

Wat. The author has no idea what a foot gun is.

In the article, "fastest" is italicized. The sentence is about whether the footgun is fast, not whether the footgun is a footgun.

Twirrim• 3 years ago

Eh? They're pointing out it's an extremely well optimised and fast library, and then pointing out a number of ways in which it's a footgun because it doesn't behave as you'd expect, e.g.

    NYC = pytz.timezone('America/New_York')
    dt = datetime(2018, 2, 14, 12, tzinfo=NYC)
    print(dt)
    # 2018-02-14 12:00:00-04:56

How is that not a footgun?

steveBK123• 3 years ago

thats horrific

pmayrgundter• 3 years ago

python, the footgun factory

benreesman• 3 years ago

Tangential, but man time zones are just really hard. Some theoretically “simple” rules seem to routinely violate human intuition, and the almanac actually changes based on laws and stuff.

Keeping everything in Zulu ms or ns and rendering into timezones as late as possible is a useful guideline as far as it carries one, but sometimes it’s not enough, this is one of a zillion reasons why e.g. calendaring is just really hard.

I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.

Cthulhu_• 3 years ago

Zulu ms or ns

Is that timestamps as in unix timestamps? That won't work with birthdates for people more than 53 years old for example; a colleague mentioned a weird issue for someone born in 1938, but that was because the timezone in that location at that time was something like GMT+1:23, then the Germans came around and standardized it to +1:20, then later it was normalized to +1:00.

tl;dr, time and date can only be accurate if time, date and location are known. Actually, some places have moved to different timezones over time as well, so you can't just say "this time has a timezone of GMT", you have to ask "what is the timezone of this location at this time/date"

Aachen2• 3 years ago

To get to/from that local time representation, you can still store it as a timestamp (to know when it actually happened) and use a timezone database that tells you what the people living there in that moment labeled that time as, right?

magicalhippo• 3 years ago

It's even harder. Is the time the user entered actually an always-local time?

As in, if I set my alarm for 08:00, I'd expect it to wake me up 08:00 local regardless if I'm in Oslo or I've traveled to say New York.

However if I make a calendar event things are less clear cut. For meetings and such it makes sense that the time is global time (hence stored in UTC), but for tasks it can be be either or, depending.

sdfghswe• 3 years ago

I really like what google calendar does, which is to allow you to display both.

comicjk• 3 years ago

I was baffled when I first entered my schedule for a conference into Google Maps, only to find when I flew across the country that all the times were wrong. I understand that it's a reasonable choice to use the local time zone at event creation, but it sure wasn't what I expected (I vaguely thought it would be like your alarm clock example, where it would move with the local time).

maicro• 3 years ago

Thank you for the examples - now I know to not try to make and publish an alarm clock or calendar app/site/feature...

cameronh90• 3 years ago

What happens exactly if you set your alarm for 1:30AM on DST changes. Does it go off twice in Autumn and skip altogether in Spring?

tshaddox4• 3 years ago

This is the part that isn’t even technically hard per se. It’s just hard because there’s literally no way to know what the computer should do because there’s no way to know what a human would even expect.

In many cases reasonable assumptions can be made, e.g. a calendar event for two people in different time zones to meet on a video call almost certainly needs to be scheduled for a proper time (or likely UTC, which is close enough since if you can ignore leap seconds). Although even then I wonder what the two attendees would expect if, between the scheduling of the event and the event itself, both of their local time zones changed their DST rules such that their UTC offset changed by +1 hours. If only one time zone changed its UTC offset we might assume that the previously agreed upon UTC time would hold, but it’s hard to say what the humans would expect if both of their time zones shifted the same way.

johnwalkr2• 3 years ago

I think it's fixed now, but DB (german transit company) used to always use your local time in its app, making it incredibly confusing to book tickets or even check on tickets you already have. I planned a trip from Japan and the train times showing up were something like 2:30 in the morning.

Another bad example is using Windows bootcamp on a mac. Windows writes local time to the RTC and MacOS writes universal time, so when switching between MacOS and Windows, it's always wrong at first.

bentcorner• 3 years ago

For meetings and such it makes sense that the time is global time (hence stored in UTC)

I think this is tricky too - if you have a 9AM meeting, if your time zone switches, should the time of your meeting change too? Most of the time, no, but if you have a meeting with other people in far away time zones, maybe you don't want it to change.

maweki• 3 years ago

For things in the past, Zulu is fine. But you won't know Zulu for a date in the future, because Timezone changes may happen.

So Zulu for the past, datetime+location for the future.

brookst2• 3 years ago

Time zones, character encoding, localization. I admire people who dig in and really understand them deeply, but as far as I’m concerned they are all exposed high voltage wires that I avoid going near.

laci37• 3 years ago

Implement everything in Zulu, then watch as the frontend ticket for proper rendering stays on the bottom of the backlog, because you always have more concerning problems than users having to do mental math on times.

dullcrisp• 3 years ago

Are you giving this as genuine advice, or are you suggesting implementing time zones early instead as a form of penance?

hprotagonist2• 3 years ago

the almanac actually changes based on laws and stuff.

some of my perennial favorites:

- in samoa, there was never a december 30th 2011.

- in indiana, time zone observance is determined per county

- in arizona, the reservation land observes dst, and the state doesn’t, and reservation lands can enclose non-reservation lands.

- lebanon recently had a legislative dispute about dst such that some entities in the nation did or did not observe dst for about a week.

cudder2• 3 years ago

- in samoa, there was never a december 30th 2011.

That was interesting, so I had to read about it. Apparently they skipped a day to be closer to Asian time zones. They did the same thing in the opposite direction in 1892 by having two 4th of Julys to be closer to American timezones. Now they kinda just walked back on that.

Interestingly I think the Python datetime approach can handle both of these with no problem. The rest of your list sounds... a lot tougher.

RobotToaster• 3 years ago

If you're storing historical dates, September 1752 only had 19 days.

ska2• 3 years ago

Tangential, but man time zones are just really hard

That can be expanded. Time and dates is just fundamentally hard.

smarx0072• 3 years ago

I think keeping time in UTC as primary data for as long as possible is a really bad idea, as explained by Jon Skeet quite a while ago: https://codeblog.jonskeet.uk/2019/03/27/storing-utc-is-not-a...

hoosieree2• 3 years ago

Non-programmers don't care about this nearly as much as we do. The prevailing attitude is "Ok sure there are edge cases, but so what? My use case is simple".

And after thinking about it a lot more, I think this attitude is actually more sensible than mine used to be. Why do we as programmers keep trying to fit a square peg into a round hole? Timezones are a political construct, not a mathematical one.

We should address edge cases by deferring to people as much as possible. Oh, you find it annoying to enter an offset? Don't worry about it, it's optional! Your dentist will expect you for your cleaning at 8:00am +/- 5 hours.

If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.

cudder2• 3 years ago

As much as I would love this too from a programmer's perspective, reading [1] convinced me that time zones are a necessary evil we just kinda have to deal with. They could be improved by a lot, though. First thing I'd do is get rid of DST everywhere.

[1] https://qntm.org/abolish

naniwaduni• 3 years ago

We should address edge cases by deferring to people as much as possible. Oh, you find it annoying to enter an offset? Don't worry about it, it's optional! Your dentist will expect you for your cleaning at 8:00am +/- 5 hours.
If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.

This is gonna go over great with the people making purchasing decisions.

banannaise• 3 years ago

Time zones are themselves a footgun. There is no actual reason for morning to start at approximately the same time everywhere. 9am can be 9am everywhere. In some places 9am can be sunrise, in some it can be sunset, and in some it can be the middle of the night. No conversions, no confusion, no resetting of watches...

We don't change the months in the southern hemisphere. Australia is cold in July, Europe is cold in January, this doesn't seem to bother us.

MajimasEyepatch• 3 years ago

It's a lot easier to remember "north and south are opposite" than it is to remember, "Oh right, 9 am is morning in England, early morning in New York, and the middle of the night in Los Angeles." And so on for every possible time of day.

Whereas it's (relatively) easy to guess, "Los Angeles is in California, that's three hours behind me in New York, so it would be 6 am there at 9 am here, so it's too early to call them yet." Not perfect, but not too bad for simple cases, and easy to do in your head most of the time.

Timezones are about standardizing _human_ time, not actual time.

chefandy• 3 years ago

There is no actual reason

There may be no technical reason for it, but having a common reference point for times of the day in terms of human experience and behavior is an actual reason. Saying you were walking down the street at 2am or 3am is immediately distinct from saying you were walking down the street at 10pm or 11pm in ways many people understand. Having to qualify it with the date and geographic location and/or sunrise or sunset patterns would be a much bigger PITA than your average person will ever experience because of time zone inconsistencies. It's just not that important for most people to be able to pinpoint a timestamp to one exact time throughout the world.

We've got GMT. Nothing is stopping anybody from using that to schedule their lives. There's a reason people don't.

sangnoir• 3 years ago

I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.

I will second you ok that: date & Time handling across timezones was one of the 2 hardest things I've ever worked on that are deceptively simple on the surface. The other was adding localization/making an app translatable to any language after shipping the first version with hard-coded text.

hermitdev• 3 years ago

Same here. Worrying about current timezones is a hard enough problem. Getting accurate historical timezones handling working is an even harder problem. Getting consistent results across Windows, Mac and Linux is an even harder problem. Needed it for proper handling of historical time series for financial data.

didip• 3 years ago

I agree. Local time with timezone is really a presentation-layer concern.

For server-side and storage layer, it's saner for everyone to simply store in UTC or epoch seconds/nanoseconds.

JediWing• 3 years ago

Working with date/times in python made me miss Joda-Time/Java time libraries. For my money it's the most well designed date/time API I've worked with.

civilized• 3 years ago

Is something like this why you can only book a ticket on Lufthansa with your correct birth date if you set your computer's time zone to German time?

Sebb767• 3 years ago

Published Mon 19 March 2018 (updated Wed 01 April 2020)

This needs a (2018) (or (2020)) @dang

mark-r5• 3 years ago

It's a bit unfair to complain that pytz doesn't use the "standard" interface, when that standard interface didn't exist when it was created. The necessary calls for syncing the time zone with the date on datetime creation were a relatively recent addition.

spott• 3 years ago

I don't think it is unfair.

If you are looking to solve a timezone problem right now, then you care about what the standard interface is right now... not "when the library was created".

mark-r5• 3 years ago

The documentation is very clear on how to use it to solve your timezone problem. The trouble comes when people expect it to "just work" without understanding how it works.

roelschroeven2• 3 years ago

Until now I never took a close look at how Python deals with timezones. Now I did, and it turns out the way Python handles arithmetic on timezone-aware datetimes is not what I expected. An example to illustrate:

    from zoneinfo import ZoneInfo
    from datetime import datetime, timedelta, timezone

    localtz = ZoneInfo('Europe/Brussels')
    utctz = timezone.utc

    a_local = datetime(2023, 3, 26, 1, 59, 59, tzinfo=localtz)  # right before change to DST
    b_local = a_local + timedelta(seconds=2)   # 2 seconds later; should be right after change to DST
    c_local = a_local + timedelta(hours=2)     # 2 hours later

    a_utc = a_local.astimezone(utctz)
    b_utc = b_local.astimezone(utctz)
    c_utc = c_local.astimezone(utctz)

    print('datetimes:')
    print(f'(A)  {a_local}  (utc: {a_local.astimezone(utctz)})')
    print(f'(B)  {b_local}  (utc: {b_local.astimezone(utctz)})')
    print(f'(C)  {c_local}  (utc: {c_local.astimezone(utctz)})')

    print('deltas:')
    print(f'(B-A)  local: {b_local-a_local}  utc: {b_utc-a_utc}')
    print(f'(C-A)  local: {c_local-a_local}  utc: {c_utc-a_utc}')

This prints:

    datetimes:
    (A)  2023-03-26 01:59:59+01:00  (utc: 2023-03-26 00:59:59+00:00)
    (B)  2023-03-26 02:00:01+01:00  (utc: 2023-03-26 01:00:01+00:00)
    (C)  2023-03-26 03:59:59+02:00  (utc: 2023-03-26 01:59:59+00:00)
    deltas:
    (B-A)  local: 0:00:02  utc: 0:00:02
    (C-A)  local: 2:00:00  utc: 1:00:00

For (B), I would expect 2023-03-26 03:00:01+02:00: that is the point in time 2 seconds after (A). The datetime Python produced doesn't even exist (because of the start of DST, time jumped from 2:00 to 3:00).

For (C), I would expect 2023-03-26 04:59:59+02:00: that is the point in time 2 hours after (A). As you can see by the times in UTC, (C) is only 1 hour later then (A) instead of 2.

Both feel really wrong.

The article compares pointer arithmetic using Python's time zone model vs pytz's time zone model, and concludes that pytz is worse because you have to normalize the result for the time zone to be adjusted. But the pytz result is correct (according to my expectations, at least), while Python's standard approach does adjust the time zone but without properly adjusting the time (and hence is incorrect, according to my expectations). It seems the author only looked at the time zone, without checking the time itself.

I can work around it by converting to UTC and back, fortunately. I had a quick look at Pendulum and it seems it has functions to do arithmetic properly.

NAHWheatCracker• 3 years ago

I've run into this at work and I handle by always add timedeltas in UTC or just design everything to work in UTC.

Unfortunately, some people dislike the former because it's 3 lines of code (convert to UTC, add delta, convert to local). I've had the same data scientist reject 3 MRs fixing this sort of issue in his code. At least it only causes a once per year error or a blip that no one notices.

Always using UTC doesn't work always depending on requirements. The worst was interacting with an API that required precisely every hour of the following day to be sent without good documentation.

stan_kirdey• 3 years ago

I've read the title and thought "oh this seems to be a weird name for a footgun. It would totally make sense if it was about timezones". Then I clicked the link.

smarx0072• 3 years ago

For a second, I thought Jack Ganssle [1] got himself a hipster domain and started to do Python.

[1] http://www.ganssle.com/tem-back.htm

nodemaker• 3 years ago

Lol I was caught by this footgun a while back. Use arrow - there I saved you a ton of time.

hoosieree2• 3 years ago

I see what you did there.

zzzeek• 3 years ago

I was going to complain that Python for all of its "batteries included" lacks a simple facility in the standard library to "please give me a timezone for EDT", but apparently one has been added in 3.9: https://docs.python.org/3/library/zoneinfo.html#module-zonei...

larkost• 3 years ago

The real problem that still is not fixed is that `datetime.date.now()` and others do not do the right thing and encode the system timezone. Rather they record a time without timezone. I have seen this cause so many problems because people are just not aware.

pganssle3• 3 years ago

Actually, naïve times are considered aware in the system time zone. There's no good way to create a `tzinfo` object that behaves like "the system time zone" and doesn't have unfortunate consequences. See this article: https://blog.ganssle.io/articles/2022/04/naive-local-datetim...

It's not an ideal situation, but given path dependence I don't think it can be improved much from here.

Too2• 3 years ago

Even worse is that utcnow() returns timezone unaware dates, when it could have easily returned dates with timezone set to utc. Leading to accidentally mix of local and utc times.

dr_kiszonka2• 3 years ago

What are your favorite datetime related Python packages? (not necessarily for tz handling)

I like dateparser for certain use cases:

https://dateparser.readthedocs.io/en/latest/

aeyes2• 3 years ago

arrow

dr_kiszonka2• 3 years ago

This arrow[0], correct? Initially, I thought you meant Apache Arrow and was confused.

https://arrow.readthedocs.io/en/latest/ (looks good!)

Too2• 3 years ago

Freezegun for unit testing.

Can stop and step time predictably. Any code using datetime functions will get the time controlled by the test, with no additional abstractions or other boilerplate needed.

Pytz: The Fastest Footgun in the West (2018)

Related Stories