Ruby-like string interpolation in Python

https://github.com/syrusakbary/interpy

by est • 11 years ago

63 50 11 years ago

github.com

est4• 11 years ago

Another gem is pyxl from Dropbox

https://github.com/dropbox/pyxl

Reminds me of JSX/E4X/XML Islands, etc.

alanfranzoni• 11 years ago

Interesting and clever approach (even though, as I see, pyxl came first at that).

Just a doubt: how do I specify source file encoding if the coding string is now hijacked for interpolation purposes? Is there a default, fixed encoding (which I hope is not iso-8859-1, python's own default)?

est4• 11 years ago

Is there a default, fixed encoding

For interpy, I think it assumes utf-8 by default.

https://github.com/syrusakbary/interpy/blob/master/interpy/c...

mkesper• 11 years ago

How is this optimized if it compiles to a + "" + b?

mesozoic• 11 years ago

It's not

andybak• 11 years ago

Are we sure?: http://stackoverflow.com/questions/3055477/how-slow-is-pytho...

rpcope12• 11 years ago

This is certainly interesting, but I think it kind of breaks with the "only one obvious way to do things," which Ruby tends not to follow (and with regards to string formatting, Python doesn't either :P ). Does this buy you anything beyond trying to ramrod Ruby syntax into Python? If this truly compiles down to byte code that's doing string concatenation in the Python VM, the built-in string formatting library tends to be a non-trivial amount faster, beyond trying to write Ruby in a language that isn't Ruby.

renox• 11 years ago

Does this buy you anything beyond trying to ramrod Ruby syntax into Python?

I'm not the poster but IMHO it fixes Python's syntax which is a case of DRY violation.

rpcope12• 11 years ago

Maybe you can elucidate what about Python's syntax is broken or not DRY?

EDIT: To me, seeing this is Python code anywhere would seem to violate principle of least astonishment, which I think is somewhat more important than being DRY, if that's even a problem here.

methodover2• 11 years ago

Well... String formatting already sorta breaks the "only one obvious way to do something" rule. I dunno about you, but we have string formatting all over the place in our code. I would wager that every single module in our codebase has something that looks like this:

  cache.add_message("Hey, your {zoop} is {boop}".format(zoop=zoop, boop=boop))

That that call to "format" just kind of sucks -- it repeats what's already pretty obvious by looking at the string. It's specially crappy when you have strings that need lots of variables. We've taken to doing this recently, which I'm usually okay with:

  cache.add_message("Hey, your {} is {}".format(zoop, boop))

The downside of that is you have to make sure that the order of the arguments matches exactly with the order of the empty brackets. It's kinda error prone... But generally not that big of a deal.

We could also do this...

  cache.add_message("Hey, your %s is %s" % zoop, boop))

Or this...

  cache.add_message("{1} alert! Hey, your {0} is {1}".format(zoop, boop))

So yeah, I would argue that string formatting in Python is ALREADY in a kinda nasty place. There's ALREADY a bunch of ways to do it, and it all just kinda sucks.

IMO, Ruby-style string formatting is probably the nicest I've seen. If it were in Python, it absolutely would be THE way to do string formatting, I bet.

  cache.add_message("Hey, your {zoop} is {boop}")

So much nicer.

dragonwriter2• 11 years ago

Actually, I kind of like the existing Python ways better in many respects from the Ruby one, since it lets you separate format strings from the parameters passed to them, and reuse format strings more easily (which is especially useful if you want to move string literals out of code and into resources, because then you can include format strings in that as well.)

BetaMechazawa• 11 years ago

I guess I'm missing the point. Why not use "Welcome %s to %s" % (who, place)

smcl• 11 years ago

From the README.md:

    I really enjoyed Ruby String interpolation, and "".format(...) or "" % (...) seems very verbose to me. I'm lazy by nature ;)

hashmymustache• 11 years ago

It's easier to see the point with more variables being interpolated where it becomes cumbersome to keep track of the order and you end up recounting everything multiple times when something's out of place.

PythonicAlpha• 11 years ago

This special syntax should be discouraged for any strings that shall be translated in the future. Different languages have different syntax, so the order can change, with this kind of syntax, you will be in big trouble very soon!

nostrademons• 11 years ago

For anything translated, you'd use the (admittedly gross, fixed in Python3) "Welcome %(who)s to %(place)s" % { 'who': who, 'place': place }. The Python3 version is "Welcome {who} to {place}".format(place=place, who=who), or if you want to be un-idiomatic and unsafe, "Welcome {who} to {place}".format(locals())

ForHackernews• 11 years ago

You can already almost do this in Python, with the native string format operation:

>> name = "Foo Bar"
>> age = 25
>> "Hi, my name is {name} and I'm {age} years old.".format(splatlocals())

"Hi, my name is Foo Bar and I'm 25 years old."

Arguably, this an abuse of `locals()`, but it gets you very nearly the same kind of use-variables-in-strings-with-curly-braces functionality.

Edit: HN markdown doesn't seem to let you escape the star italics operator. To be clear, you have to double-star splat locals().

maxerickson5• 11 years ago

Repeating my similar comment, str.format_map takes a dictionary directly (available since 3.2).

berdario2• 11 years ago

If the only thing you want is to use the {identifier} syntax, without having to repeat `identifier` more than once, the simplest alternative is probably:

    >>> package = "foo"
    >>> whatever = "bar"
    >>> "Enjoy {package}".format(**locals())
    'Enjoy foo'

(disclaimer: I wouldn't actually use locals() like this in real code)

Also, if this thing is truly mangling bytecode, it's not portable between different python versions

But i'm not quite sure about that: I only skimmed the codebase, and that works seems to be done by interpy_untokenize, which boils down to some string mangling

Also, having expressions (or worse, statements... like it would be in ruby since there's no difference there) evaluated when evaluating a string is quite bad (this is not Haskell, and thus we cannot have guarantees that side effects won't happen)

Nice hack, btw

rraval2• 11 years ago

Note that `locals()` actually does funky things around enclosing scopes.

Consider the following Python2/3 code:

    def outer():
        x = 1
        def inner():
            print(x)
            print("x = {x}".format(**locals()))
        inner()

    outer()

This actually prints the right thing when run:

    1
    x = 1

However, if you remove the `print(x)` line, both Python 2 and 3 don't hoist the enclosing `x` into `locals()` (since it can't see a single usage of that variable), resulting in a `KeyError`:

    Traceback (most recent call last):
      File "blah.py", line 7, in <module>
        outer()
      File "blah.py", line 5, in outer
        inner()
      File "blah.py", line 4, in inner
        print("x = {x}".format(**locals()))
    KeyError: 'x'

Python 3 has the `nonlocal` keyword that you can use to indicate that you're using a variable from an enclosing scope so that it's properly introduced into `locals()` but Python 2 doesn't have this facility.

wylee• 11 years ago

That doesn't seem funky to me, given that `x` in your example is in fact not local to `inner`. It may seem surprising, but it's consistent with how Python handles locals in general.

maxerickson5• 11 years ago

str.format_map takes a mapping directly.

berdario2• 11 years ago

Thanks! I'd edit my post if I could

csl• 11 years ago

    Also, if this thing is truly mangling bytecode, it's not portable between
    different python versions [...] seems to be done by interpy_untokenize,
    which boils down to some string mangling

It uses the Python file encoding property ("# coding: foobar") to rewrite the source code, not the bytecode, and they refer to pyxl as an inspiration.

For a good explanation, see https://github.com/dropbox/pyxl

volent• 11 years ago

Looks fun but it doesn't seem to support unicode in python2.

b6fan• 11 years ago

The `codecs.register` approach is interesting. I hope Ruby has something similar. Unfortunately I could not find such thing in Ruby.

est4• 11 years ago

Yup that's the point of my submission, too bad HN edited my title so it looks like a boring trick. It allows you to transform your code without significant speed loss.

Think of possibilities like

# coding: JIT

# coding: inline-C

etc.

maxerickson5• 11 years ago

The typical way to do that is to reuse the interpreter infrastructure. This hack could be done with the ast:

https://docs.python.org/3.5/library/ast.html#ast.NodeTransfo...

BerislavLopac• 11 years ago

There are also Python template strings: https://docs.python.org/2.7/library/string.html#template-str...

ekimekim• 11 years ago

I wrote a version of this some time ago:

https://github.com/ekimekim/pylibs/blob/master/libs/interpol...

It's not quite as natural as your one:

    def foo(x):
        print interpolate("Hello, {x}")

Though I do actually prefer having the explicit formatting call there so I know when the interpolation is being performed. Side effects and all that. In a perfect world, this is the syntax I'd prefer:

    def foo(x):
        print "Hello, {x}".format()

ie. a format() without args defaults to "all variables accessible in the current scope". I wouldn't actually want it to support arbitrary python the way ruby does, I find the .format() syntax flexible enough.

(Also, my current implementation is for locals only. It wouldn't be hard to extend to globals, but would suffer the "nonlocals won't be captured" problem described in other comments here no matter what)

(EDIT: Also, it relies on sys._getframe, which is CPython specific)

philh3• 11 years ago

String interpolation is one of the things that I've never understood about python. I'm not a massive fan of Ruby's particular syntax for it[1], but not having any syntax at all feels like such a massive oversight. And then I realize that it's probably a deliberate omission and that just seems really really weird to me.

So I'm happy to see this, even if I'm probably too conservative to use it in my day job. And I didn't know about the coding: thing, and it looks like this method could also be used on my other python-wtf, which makes me even happier.

(My other python-wtf is that there really ought to be nicer syntax for a['b']. For a while I thought that a::b would be nice, but then I remembered that that could be a slice, so it can't be parsed reliably. a$b is probably my next choice. Or even require that kind of slice to be written with a space or something, like "a: :b".)

[1] Requiring braces even for a simple variable name seems like a poor decision. There's a little-known language called Haxe which IIRC gets it right: you can embed variables with just "hello $foo", or expressions with "your score is ${kills-deaths}". I get that Ruby allows unusual characters in variable names, and it's not obvious whether "is this yours, #name?" means #{name?} or #{name}?. But I'd rather have that potential for confusion than force the braces even when there's no ambiguity.

aetherson2• 11 years ago

Note that Ruby allows non-braced interpolation with instance variables. So:

  irb(main):006:0> @foo = 'bar'
  => "bar"
  irb(main):007:0> "This is the value of foo: #@foo"
  => "This is the value of foo: bar"

toupeira• 11 years ago

There's also a very well-known language called PHP that does the same ;-)

philh3• 11 years ago

Are you sure? https://php.net/manual/en/language.types.string.php#language... suggests that it can only interpolate a few specific types of expression, but can't do e.g. arithmetic or or simple function calls. (It seems you can interpolate a variable named by a function call, but you can't interpolate a function call itself.)

mcbetz3• 11 years ago

Can anyone explain the creators' reasoning beyond the existing string interpolation in Python? I never thought it was cumbersome, on the contrary I liked the verbosity of "My {name} is, I am {years} old".format(name=name, years=years) and I think there are good reasons (beyond dragonwriter's reusability argument).

aetherson2• 11 years ago

So, like, just to point out, your example would be, with name Adam and age 10:

  "My Adam is, I am 10 old."

Correcting your example:

  "My name is {name}, I am {years} years old".format(name=name, years=years)

So to throw that out, that one line includes the word "name" four freaking times, and years four freaking times. You say you like the verbosity of it. Why? Would you like this format yet more?

  "My name is {name=name}, I am {years=years} years old".format(name=name, years=years)

If not why not? It's yet more verbose.

I think that the reason that other people like the non-verbose format of:

  "My name is {name}, I am {years} years old" # assuming the presence of local variables "name" and "years"

Is that, well, it's pretty obvious what's going on here, and repeating name and years a bunch more times do not, it seems, make it any more clear what's going on.

A reasonable argument might be that:

  "My name is {name}, I am {years} years old".format()

Is more clear about what's going on. But repeating the variable names is not particularly elucidating.

mcbetz3• 11 years ago

Well, I did not want to defend the way .format() works now. That's why I asked for the reasoning. Your reasonable argument sounds reasonable to me, I would certainly use it. But then my initial question is even more important: Why is this not the one Python way. What did the initial designers think?

maxerickson5• 11 years ago

The new style .format is talked about here:

https://www.python.org/dev/peps/pep-3101/

I guess finding discussion of the % formatting would be harder. I suspect that the discussion would have been about the tradeoff between the implicit variable insertion and simpler positional examples:

  "My name is %s, I am %s years old" % (name, years)

(where there is definitely at least a tendency to avoid implicit behavior in the design of python; of course positional formatting like that is implicit, but it is quite a bit less implicit than automatically pulling variables out of the current scope)

edit: the % formatting was probably informed by sprintf.

mcbetz3• 11 years ago

Thanks for the explanation and link!

languagehacker• 11 years ago

Something about this seems patently unpythonic. Also, the way the code is included reminds me of monkey-patching, which is a Ruby behavior I like to leave at the door when I'm coding in Python.

Ruby-like string interpolation in Python

Related Stories