New string formatting in Python

https://zerokspot.com/weblog/2015/12/31/new-string-formatting-in-python/

by eatonphil • 11 years ago

173 132 11 years ago

scrollaway7• 11 years ago

String interpolation is one of those features where I'm really confused about the direction the Python team wants to take. It's implicit and magic, and yet another non-obvious way to do do string formatting.

What's more, it's being hailed at a plus for localization which it isn't. Localizers should never, ever deal with string interpolation - anything past what .format() does is essentially untranslatable.

eru2• 11 years ago

Someone made a good argument that in order to support lots of languages, you actually need general functions not just strings to be put into .format() or interpolation.

That's because some languages have complicated changes in the text depending on eg the number of things: not just singular/plural, but more complicated. Russian is one example.

cbsmith5• 11 years ago

That's a little misleading.

In practice, what you need is a notion of a "context" for your formatting (often something that is aware of both the current i18n and l10n factors), and that context needs access to the fundamental objects that would be rendered in to a string.

Effectively, the "format string" becomes an identifier for the particular message you want to render, though often the objects themselves are definitive enough.

In that paradigm, interpolation/formatting/whatever might be the default mechanism employed by the context, but you don't want to have that as your explicit mechanism. You want one more level of indirection before you get to it.

Some platforms make the mistake of having that context be tied to the thread, out worse still, a global, and that falls apart once you have any multiplexing logic (and the way python currently works with posix locales fully captures how terribly you can do this). Either way, I'd argue out really is a different problem from interpolation, and if you are trying to solve it using interpolation, you don't understand the problem.

yeukhon• 11 years ago

Being a die-hard Python user on a daily basis, I concur. I think these core devs should not invent too many new syntax and ways to solve problems which don't need to be added to the language.

I am not sure why i18n is a big deal, let another library deal with it.

Whatever, I have the choice not using this feature after it is accepted and implemented. Their PEP discussion on email always ended up in tangential. Problem I always have with string manipulation is dealing with long string, which for coding style I'd split into multiple +, and thus using format is pretty ugly.

scrollaway7• 11 years ago

You don't have to use + to join strings on multiple lines. Here:

    >>> (" foo "
    ...  "bar.")
    ' foo bar.'

mixmastamyk26• 11 years ago

I'm a bit confused about all the doubt on such a common feature industry wide. No, it's not magic, it's a simple transformation, takes about 30 seconds to learn, and is becoming an industry standard among modern languages. See C#, Scala, JS, Swift, etc, not to mention bash and perl.

It also has little in common with i18n, the use cases differ too much. Perhaps in the future someone can figure out how to bring them together, but not today.

schmichael8• 11 years ago

All of the languages you list have been criticized for adding too many language features.

Go and Rust are two notable new languages which require explicit formatting functions as opposed to relying on special string interpolation syntax.

schmichael8• 11 years ago

Python was and still is to some diminishing degree influenced by C, C++, and Java - none of which have language syntax for interpolating strings based on variables in scope.

js8• 11 years ago

I agree with you. It is already possible to use quote and plus in the same way as the bracket in the new way, with the added advantage that the original syntax is more orthogonal.

Is it really worth updating all the Python syntax formatting and analysis code out there just to save one character on an operator? I don't think it's a good tradeoff.

sametmax6• 11 years ago

+ is a terrible operator for formating.

You got:

    price = 18.8
    date = datetime.datetime.now()

Now you want to save in a file:

    "It's 18.80 and we are the 01/04/2016"

With +:

    num, dec = str(price).split('.')
    msg = ("It's "+ num + "." + dec.ljust(2, '0') + 
          " and we are the " + date.strftime("%m/%d/%Y") + "\n")
    f.write(msg)

It is so long and ugly we have to break it on several lines to respect PEP8.

And it's a pain to write, or even worst, to read.

With print():

    num, dec = str(price).split('.')
    print("It's", num, ".", dec.ljust(2, '0'), 
          " and we are the ", date.strftime("%m/%d/%Y")), file=f)

A tiny bit better since we don't have to use that many concatenation tricks.

If we didn't have to format the float, print() would have converted it to string for us which is nice (with + you have to call str() all the time).

With "%":

    f.write("It's %.2f and we are the %s\n" % (price, date.strftime("%m/%d/%Y")))

Way better, but still hard to read.

With format():

    f.write("It's {:.2f} and we are the {:%m/%d/%Y}\n".format(price, date))

Now we are getting sometwhere...

With fstrings:

    f.write("It's {price:.2f} and we are the {date:%m/%d/%Y}\n")

fstrings have everything:

    - easier / faster to type than any other versions;
    - easier to read than any other versions;
    - as expressive as any other versions.

It's reduce work by making you think less, type faster, read faster (for you and your colleague) and it's easier to spot bugs.

The only place where you don't want fstring are for l10n.

hyperpape• 11 years ago

Why is it implicit and magic? Looking at this post, and the PEP, it seems like interpolation is actually pretty close to .format in semantics, but syntactically simpler. There are a few bits that I didn't quite follow in the PEP, so I may just be missing the issue, however.

mixmastamyk26• 11 years ago

It isn't for the most part, the GP is incorrect, except for the confusion.

saurik• 11 years ago

The important thing for localization is to allow the strings to be swapped at runtime by loading them from some kind of database in a way where the arguments are at least numbered, if not named, so as to let them appear in different orders for different languages. The interpolation syntax does not seem, to me, allow for this: it is an "implicit" syntax offered during parsing to take a constant string and immediately build it out of parts, as opposed to the "explicit" % operator or .format methods: it is extremely clear with these two models how one would load a template that has the strings in a different order, or which used a different subset of the strings.

    >>> '%(one)s %(two)s'%{"one":"hello", "two":"world"}
    'hello world'
    >>> '{one} {two}'.format(one="hello", two="world")
    'hello world'

The PEP that is looking at this problem has been deferred: at the end of the document they kind of indicate that they didn't really think about the i18n problem correctly and so don't actually have a good solution to present, and are going back to think about it more.

https://www.python.org/dev/peps/pep-0501/.

Walkman3• 11 years ago

I have mixed feeling about this. This will be the FOURTH way of formatting strings in Python. The other formattings will never go away.

orionblastar• 11 years ago

I think they keep the other formatting for backward compatibility in legacy code.

bhaak• 11 years ago

What's the fourth way? The article only mentions three.

nothrabannosir• 11 years ago

Not the op but Im guessing just using + (which calls .__str__(), iirc?)

rdtsc• 11 years ago

4th way is with templates:

   >>> from string import Template
   >>> s = Template('$who likes $what')
   >>> s.substitute(who='tim', what='kung pao')

This will probably be TIL for many people. It is a surprisingly hidden feature.

I for one, still like the "%s" % x instead of "{0}".format(x). It is simply shorter and I already know or use printf for C in other parts of the project. But with the new f"..." interpolation, I can see liking that more. I am all for being as concise as possible while still being explicit (if that makes any sense at all ;-) ).

syrusakbary• 11 years ago

Quite useful! If you want to have string interpolation in Python 2.6+ just use https://github.com/syrusakbary/interpy

kseistrup• 11 years ago

“They” could put string interpolation in the __future__ module:

from __future__ import string_interpolation

evrial• 11 years ago

Just because simply one way isn't enough.

richard_todd• 11 years ago

The motivation for PEP-0498 given in the article was the difference in verbosity between these two lines:

  "{} {}".format(a, b)
  "%s %s" % (a, b,)

That's not very convincing. I was hoping this article would make a good case for interpolated strings, since it's starting to feel like Python is having an identity crisis. Type annotations especially took me by surprise, but string interpolation is another good example of an addition that doesn't feel like Python (imho, anyway).

jdnier2• 11 years ago

The more common case for me is to name the parameters. Especially with longer variable names, this gets tedious fast. Given

    >>> very_long_var_name_1 = 'spam'
    >>> very_long_var_name_2 = 'ham'

compare

    >>> # Explicit but tedious and doesn't help readability:
    >>> print('{very_long_var_name_1}: {very_long_var_name_2}'.format(
    ...       very_long_var_name_1=very_long_var_name_1,
    ...       very_long_var_name_2=very_long_var_name_2))
    spam: ham

with

    >>> # Explicit but somehow feels dirty:
    >>> print('{very_long_var_name_1}: {very_long_var_name_2}'.format(
    ...       **locals()))
    spam: ham

and

    >>> # Still fits on one line. I think f prefix makes intent clear.
    >>> print(f'{very_long_var_name_1}: {very_long_var_name_2}')
    spam: ham

toyg2• 11 years ago

Thing is, * * locals() "feels dirty" but does exactly the same thing as f''. Except f'' does it implicitly, hiding dirt under the carpet. One day, that dirt will turn something into a bug; which is exactly why the Zen says "explicit is better than implicit".

The best way imho would be:

    vars = {'short1': very_long_var_name1, 
            'short2': very_long_var_name2}
    print('{short1} {short2}'.format(**vars))

Easy and extremely unlikely to ever include the wrong variable.

monkmartinez• 11 years ago

I'll just leave this here: https://www.python.org/dev/peps/pep-0020/

"There should be one-- and preferably only one --obvious way to do it."

There are a lot of warts on that snake, but I still really like using Python.

aexaey• 11 years ago

PEP20 sound amazing, but then when this only way end up being way too verbose/ugly, you are kind of stuck with it. Compare:

    #python
    import re
    m = re.search('(a.+)(d.+)', 'abcdef')
    if m:
      print(f"{m.group(1)},{m.group(2)}")

    #perl
    if('abcdef' =~ /(a.+)(d.+)/){
      print "$1,$2";
    }

JoshTriplett• 11 years ago

I don't know why Python re match objects don't support indexing; a __getitem__ method would allow m[1] and m[2] instead (or m['name'] for named groups).

That aside, though, I don't consider brevity alone the most critical criteria for a programming language; that way lies APL. Expressiveness, yes, but not at the expense of clarity.

eugenekolo22• 11 years ago

Preferably one. Unfortunately they put in `format`, and can't remove it now. I think moving towards f-strings is a lot better and should be embraced.

mixmastamyk26• 11 years ago

Format is still needed for non-literals and user objects, this is just an extension of it.

orf• 11 years ago

There are a couple of warts for sure, but Python is remarkably wart free. Compare it to something wart ridden like PHP or JavaScript.

mixmastamyk26• 11 years ago

"Practicality beats purity."

sametmax6• 11 years ago

keyword is "preferably".

Also "Simple is better than complex", "practicality beats purity" and "Readability counts".

The zen is not the bible, you don't get to cherrypick the stuff you want to make your case.

Plus, they are just guide lines, in the end, you have a debate in the python dev mailing list with reasonable people making their case.

sauere• 11 years ago

Not a fan. We already have 2 ways of formating strings in Python, we really should not bring in a third one.

It may be true that

    "{} {}".format(a, b)

is a bit verbose, but it is crystal clear and clean. Just remember the Python Zen: "There should be one-- and preferably only one --obvious way to do it." _and_ "Explicit is better than implicit."

mixmastamyk26• 11 years ago

It is the same thing as .format without all the noise.

Grue33• 11 years ago

No, it's the same thing as .format, except limited to literals, so basically useless (hardcoding literal strings is bad for i10n).

rshaban• 11 years ago

There should be one-- and preferably only one --obvious way to do it.

Hoping for a decision from the core team – having both f"" and .format is a pretty clear deviation from this principle.

noobermin• 11 years ago

Well, having both % and .format was already a clear deviation from it.

btmiller• 11 years ago

Though I think the communication has been pretty clear that developers should stop using % formatting in favor of .format when moving to Python 3. Not so much a deviation, just a migration. Now that there's another alternative, there definitely needs to be some clear communication about what's going to be idiomatic going forward.

Rangi42• 11 years ago

Also not a fan. The two existing methods of string formatting are sufficient:

    "%s %s %s" % (a, b, a)
    "{} {} {}".format(a, b, a)

If you want placeholders to match variable names, you can do:

    "{a} {b} {a}".format(a=a, b=b)
    "{a} {b} {a}".format(**locals())

So this is just unnecessary (especially since the "f" prefix is easier to miss than a "format" method):

    f"{a} {b} {a}"

And this is downright obfuscated—putting operators inside of string literals:

    f"{a + ' ' + b + ' ' + a}"

roywiggins• 11 years ago

The PEP says that locals() and globals() solutions were considered, and discarded:

https://www.python.org/dev/peps/pep-0498/#no-use-of-globals-...

You could argue that's not a good enough reason, but it's there.

eugenekolo22• 11 years ago

I certainly agree about your last statement, I have no clue what that would do without reading the spec. I'd guess an error, but apparently not.

soheil• 11 years ago

Am I crazy or PHP had this feature since forever? String templates can significantly improve programming speed specially for those of us who are keyboard impaired. Not having to remember how many %s' you need or if one of them should be %d and so on may to most seem just knit picking but personally it makes a day and night difference.

MereInterest3• 11 years ago

PHP has had it forever. Given PHP's reputation, I'm not sure if that is a good argument for this feature. Rather than '%s' and '%d', why not just use '{}' and '.format'? That way, each argument is converted to a string appropriately. If it becomes confusing with many arguments, it is easy to name the parameters '{myparam}'

mixmastamyk26• 11 years ago

From another post:

    "{a} {b} {a}".format(a=a, b=b)
    "{a} {b} {a}".format(**locals())

Compared to this:

    f"{a} {b} {a}"

Sorry that's about 1000% better. This should have been the one way to do it, originally. It isn't magic either, rather a simple compile-time transformation to existing format syntax. There's nothing new to remember besides a large reduction in noise.

meowface• 11 years ago

Some people deride string interpolation as not explicit enough, but I think it's extremely readable and adds clarity. Plus, with the `f` prefix, it's plenty explicit IMO.

MereInterest3• 11 years ago

String interpolation is only being added for string literals, not for generic strings. This means that you still wouldn't be able to read a template from a file, then format text in.

    with open('template.txt') as f:
        template = f.read()
    formatted = template.format(**values)

mixmastamyk26• 11 years ago

That's a good thing due to security reasons as arbitrary expressions are allowed. There are plenty of templating solutions available.

Walkman3• 11 years ago

Nothing wrong with this, because it's explicit. You could't be mo terse in this case, could you?

schmichael8• 11 years ago

I'm in the "wish it was more explicit" camp. Would a fmt function be that terrible?

    fmt("{a} {b} {c}", a=a, b=b, c=c)

If you want to save typing, maybe use :a instead of {a}. Or ?a would have made plain old ? a nice positional variant:

    fmt("?a ? ?", a=a, b, c)

The main benefit of the fmt function is that it requires no syntax changes to the language and is trivially provided by a third party library for all past versions of Python.

That being said this ship has sailed. I guess I just take a more conservative approach to syntax changes than most.

Update: a bit sad to see my votes fluctuating wildly on this post. Please don't use votes to support or disagree with me: that's not what they're for. Please vote only based on whether you find this relevant.

mixmastamyk26• 11 years ago

It is explicit, in that one must use the f'' prefix. There are no syntax changes as u'', b'', and r'' already exist. Positionals make bugs more likely.

scrollaway7• 11 years ago

There's a fantastic "more explicit" version of it: str.format. This really is just an implicit version of str.format(). I just don't understand why such a significant syntax change would make it into python when it deviates so much from their usual mantra.

pvg• 11 years ago

https://news.ycombinator.com/item?id=117171

kevin_thibedeau2• 11 years ago

This breaks gettext translations:

  _(f"English {a} words {b} here {a}")

The interpolation is done before the string is passed to gettext which can't retrieve the translated string any more.

mixmastamyk26• 11 years ago

Correct, don't use it with i18n, the same as with any other string formatting.

such_a_casual3• 11 years ago

I never understood why I had to write (count=count, apples=apples, name=name) when a logical default would suffice.

Walkman3• 11 years ago

I suppose you mean to take the variable names of the arguments without using keyword arguments? That's not possible because .format() is just a simple method call on str/unicode objects.

such_a_casual3• 11 years ago

That's not possible

That statement is against my religious beliefs. Apparently we can stick an f in front of the string, but we can't have default values that work with locals()?

makecheck• 11 years ago

If you have a dictionary "d" somewhere with the keys that should become keyword arguments then this would work:

    f(**{x:x for x in d})

pramodliv1• 11 years ago

One can also do "{count} {apples}".format(locals()) if we are playing code golf.

Well, star-star locals. I don't know how to write it according to https://news.ycombinator.com/formatdoc

jdnier2• 11 years ago

+1 for https://pyformat.info/ (mentioned in article), with some nice practical examples of equivalent %s and .format() formatting.

hisham_hm3• 11 years ago

That actually looks pretty nice but as Python 3.6 is slated for release in another 12 months you will have to wait a little longer.

...Or just do it in 28 lines of Lua:

http://hisham.hm/2016/01/04/string-interpolation-in-lua/

This is a nice showcase of how Lua's metamechanisms can be applied to do things that often require new features in other languages.

kedean2• 11 years ago

I'd argue that it's not a good thing that you can just disregard block scoping willy-nilly from within unprivileged code.

hisham_hm3• 11 years ago

If you are in a context where you are concerned about "privileged vs. unprivileged" code, you have to sandbox way more than that, and Lua provides you many mechanisms to control that (more than any other mainstream scripting language). Disabling the "debug" library which I used there for traversing block scopes is the very first thing one does when sandboxing in Lua.

c3534l• 11 years ago

At the very least, it sure looks a hell of a lot better. And using symbols and single-character abbreviations for types (in a dynamically typed language, mind you) and an overloaded % which has the extra syntactical rule that it only takes a single argument and so multiple replacements have to be done by throwing them all into a tuple is FAR more intuitive and readable than the way most languages do string formatting. Plus the alternatives, that require you bounce back and forth between the string and the variable it is replacing rather than read left to right the way humans are meant to read strings of text is a terrible design. Something as simple as string formatting should not rely on arcane and nuanced rules which are more or less arbitrary.

You've all been programming in C-like languages for far too long to realize what a horrible design string formatting is. You can argue over "explicitness" all you want, the new way is easier to learn, easier to read, makes more intuitive sense, requires learning fewer rules, ad is close enough to the format string method that they work well together.

sandGorgon2• 11 years ago

The rift between Python 3 and Python 2 seems to be a fallout of the one-true-way philosophy. In fact, it almost stands the reason that Python 3.6 ought to be Python 4 if one-true-way needs to be upheld.

If Python 3.6 is going to introduce multiple ways to do the same thing, there is no good reason to not merge Python 2 and 3 together and have both set of behavior co-exist with each other (__future__ or __past__).

In one shot, you break the Berlin wall of Python.

sametmax6• 11 years ago

Apprently you haven't being using Python 3 much. It reduces the numbers of ways to do stuff a lot.

    - class stuff(object) vs class stuff;
    - range vs xrange;
    - itertools.izip vs zip;
    - itertools.imap vs map;
    - itertools.ifilter vs filter;
    - dict.items vs dict.iteritems vs dict.viewitems;
    - dict.items vs dict.itervalues vs dict.viewvalues;
    - dict.items vs dict.iterkeys vs dict.viewkeys;
    - __cmp__ vs __eq__ + __gt__;
    - sorted(cmp) vs sorted(key);

They didn't fallout from the philosophy. They have been pragmatic and tried to balance the language design : gaining modern features vs making a robust base vs pleasing the legacy crowd. Is. Is. Very. Hard.

And yes, we would all prefer to have less way to format. Would it mean I would prefer NOT to have fstring ? Certainly not, it's a great feature. We can't live in the past because it will make us not stand perfectly to the ideal we have.

Real life is not ideal.

But merging Python 2 and 3 ?

With the string model completly reworked, that would be apocaliptic. Most people don't realize how deep the unicode change has been.

I have been dev and teaching Python 2 and 3 for years. The amount of problems linked to UnicodeDecodeError dropped by 90% after the switch.

Not because Python 2 model didn't work.

Because nobody understand text.

Most dev don't understand what text is. They just want to format string. That's what Python 3 helps to do, and it does it well.

Mixing both would be like mixing olive oil and vanilla ice. Great on their own way, but use them together all you'll get a terrible meal.

sandGorgon2• 11 years ago

at no point am I arguing whether python 3 is better or worse than python 2. That is an uninteresting conversation because the fact remains that less than 30% of all software is in python 3. Even when Google released Tensorflow, it was in Python 2 (and they employ core developers of Python).

Do you seriously think anybody is thinking of dropping python 2 support by 2020 ? that will only create a fork. None of the core frameworks have upgraded in a decade. Look at Flask for example. So yes, I havent been using Python 3 much - there has been no reason to.

the point I'm trying to make is how to get everyone on the same page. The reason Python 3 was api incompatible was because of the core tenet of one-true-way. All the functions you mentioned may be superior, but unless you give people a way to mix and match both in the same source , you will not have adoption.

Or do you think Python 3 adoption has been successful ?

bootload• 11 years ago

"PEP-0498 tries to improve this situation by offering something that has been common to other languages like Ruby, Scala and Perl for quite some time: Interpolated strings."

P3 '.format 'is fine, the only problem I have is forgetting the last ')' and vim picks this up. Is interpolation that good to introduce another way of doing things?

losvedir• 11 years ago

As an outsider to the python community and full time ruby dev, this "controversy" baffles me every time it comes up. Lightweight string interpolation is obviously better! I think you all have string Stockholm Syndrome or something.

The one counter argument that makes sense to me is that in general we shouldn't be doing easy string interpolation, since that way lies SQL injection, XSS, etc, and should instead rely on a stronger type system with binary text blobs, HtmlStrings, SqlStrings, etc, with automatic escaping into and out of the data type.

But then that's not the case with Python now. If you're only trying to stick this string inside that string in a quick and dirty manner, I totally don't understand the reticence folks have to something the way ruby does it: "Name: #{first_name}".

scrollaway7• 11 years ago

If it's obviously better, would you care to share some better arguments than "people who disagree have stockholm syndrome"?

Don't get me wrong, I'd like Python to have better, more obvious, more concise string formatting. However the last time we had this discussion, it was about str.format() and how it was going to be awesome and don't worry modulo-formatting will go away.

Turns out it did not; modulo formatting is still there because why would it be removed. This is history repeating itself - are you actually baffled that some people learn from past mistakes?

mixmastamyk26• 11 years ago

All you have to do is try it once or twice to enjoy the readability and lack of positional mismatch bugs.

None of the other techniques are going away. This time yes, no one is naive enough to think so.

pvdebbe• 11 years ago

I for one welcome the shorthand interpolation. String formatting is the thing we all do all day long (slightly exaggerated pretty much).

Now if python would begin to support immutable values by default then I'd be most content, and Python complete enough.

blubb-fish• 11 years ago

by offering something that has been common to other languages like Ruby, Scala and Perl for quite some time

So funny ... the most prominent language known for this stuff is PHP ... and missing :D

New string formatting in Python

Related Stories