Why Using .* in Regular Expressions Is Almost Never What You Actually Want

http://blog.mariusschulz.com/2014/06/03/why-using-in-regular-expressions-is-almost-never-what-you-actually-want

by mariusschulz • 12 years ago

184 66 12 years ago

blog.mariusschulz.com

perlgeek2• 12 years ago

Basically the same advice, from the year 2000: http://www.perlmonks.org/?node_id=24640

There is one legitimate use case of .* though: advancing to the last match of something. If you want to find the last digit in a string, /.*(\d)/ will readily find it for you.

Mithaldu• 12 years ago

Be aware though that \d will also find arabic or roman numerals. :)

mhaymo• 12 years ago

Roman numerals? I have never encountered a regex implementation where '\d' matches 'X'. It's certainly not the case for Javascript: http://regexpal.com/?flags=g&regex=\d%2B&input=10%0Ax%0AX%0A...

UweSchmidt• 12 years ago

\d finds arabic numerals? As in 0-9? Scandalous :)

http://en.wikipedia.org/wiki/Arabic_numerals

eCa• 12 years ago

A somewhat clearer way (imho) is /(\d)\D*$/ since it anchors to the end of the string.

Rynant2• 12 years ago

This is not the same as finding the last match though. The parent's example will match '2' in '1 of 2 steps.'

claar• 12 years ago

According to the debugger at http://regex101.com/:

  /.*(\d)/

- Searches right-to-left, backtracking until it finds the match.

  /(\d)\D*$/

- Searches left-to-right, going forward a step, backtrack, forward, backtrack, until it finds a match.

If you're looking for a match toward the end of a string, the .* version will be faster.

perlgeek2• 12 years ago

Yes, but that's harder to do for more complicated regexes, because you need to negate a regex (here \d => \D) for this trick.

If you have a complicated regex $r, you can only negate it with (?:(?!$r).), and in that case, .$r is much easier to read :-)

sanderjd• 12 years ago

Also, if you want to capture everything after some prefix. For instance, in a log: /(WARN|ERR) (.*)/

prohor• 12 years ago

A bit of a problem with lazy quantifiers is that they are not so widely supported out of the perl world. Therefore I often need to find some tricks to get similar behavior (eg. "[^,]*," - if coma is separator)

BugBrother6• 12 years ago

Huh, PCRE is universal (almost, not in e.g. Emacs :-( ) and supports (almost) everything?

Edit: Cough, after rechecking... PCRE is not as universal as I thought. It seems I've been lucky. :-) http://en.wikipedia.org/wiki/Comparison_of_regular_expressio...

Edit 2: "Atomic groups" on that wikipedia link is when you can write a full grammar in a large regexp, right? Answer myself: No, it is the name for stopping backtracking. I've seen it as named "possessive" (perldoc perlre).

collyw2• 12 years ago

MySQL doesn't have them, it has some POSIX version, which is way less intuitive. Plenty of people use MySQL.

SiVal• 12 years ago

I thought it was becoming more universal, but now I'm not so sure.

grep on the Mac used to use PCRE regexes if you used the -P option (`grep -P ....`), but beginning with OS X 10.8 the -P option was removed, so an important place that used to offer PCRE (default grep on a default Mac) actually removed support for it. They didn't replace it with something better; they just took it away. Maybe a Unicode issue?

Not only is PCRE not universal, overall support might even be waning.

hnriot• 12 years ago

Except they are! Java, python, JavaScript etc all support lazy quantifies. In fact I can't think of a single language that doesn't.

caio1982• 12 years ago

Sed doesn't AFAIK.

mkehrt• 12 years ago

Except bash, awk, sed and vim. So, everywhere I use regular expressions.

gesman• 12 years ago

.*? Is the solution!

tjgq• 12 years ago

Also read Russ Cox's writeup on implementing regular expressions [0]. Backtracking can be done efficiently; it's just that most regular expression engines have suboptimal implementations for it.

[0] http://swtch.com/~rsc/regexp/regexp1.html

BugBrother6• 12 years ago

I've cursed over Python's backtracking, at least a few years back. (Why can't they just use PCRE? :-( Any advantage at all?)

ori_b2• 12 years ago

PCRE has the same problems with backtracking.

maxerickson2• 12 years ago

Python beat PCRE to Unicode support by several years.

(So there at least was an advantage)

guynamedloren• 12 years ago

I've run into this problem so many times. Everywhere I think I want .* , I actually want .*? (non-greedy matching). Make a mental note of this. It'll save you lots of headaches.

moron4hire• 12 years ago

I think the advent of automatic regex match highlighting in text editors is changing the regex use-case for a lot of people. It certainly did for me. I no longer see regexs as just "something you use in code to test input". I now use them as general purpose text editing tools. In a way, it's like templated text output, with input specified in the same buffer.

I know this has been done forever, but usually only by extreme greybeards in Vi or Emacs world. The auto-highlighting now makes it possible for everyone to do it.

So that said, with the ability to restrict regexes to just a selection of text, it's more about regex golf--the fewest characters, the most productive--than it is about semantic correctness. If it works for my input, that's all that matters, because the regex is getting discarded thereafter.

Pxtl2• 12 years ago

Yeah, I do all my data-imports from flat files using regex - easy to export from spreadsheet programs as flat files, then regex them into a bunch of insert/update statements.

collyw2• 12 years ago

Yes, I do plenty of similar things.

Format a load of data using regexes first, then use it hard coded as string to do quick one off script to update the database. It beats trying to parse Excel directly, as you never know what data type a cell will return.

kstenerud2• 12 years ago

Actually, it only needs to be \[([^,]+),([^\]]+)\] because you're only going up to a comma in the first capture group and a square bracket in the second.

poolunion• 12 years ago

That would also match all of "[a] more [b,c]" though.

kstenerud2• 12 years ago

And the other regex would match all of "[a more [b,c]". Your regex must be designed around your expected input.

Pxtl2• 12 years ago

I've gotten into the habit of using the "not" operation instead of .* a lot. If I'm looking for bracketed text, I use not-bracket to match the contents.

I tend to avoid the non-greedy operator just because it often fails in terrible half-assed regex implementations (eg. visual studio 2010)

bane• 12 years ago

I wish the not operator allowed for sub-expressions instead of just character classes. It'll probably make it slower, but it would remove lots of unreadable convolutions people have to go through.

ori_b2• 12 years ago

There are some regex implementations that allow it, but it's a very confusing feature. Remember that '' is not 'a'.

Arbitrary expressions can have arbitrary length, so excluding an expression simply will match it, fail the match, and backtrack to the next option.

blueblob• 12 years ago

Me too. I find that if you work in a bunch of different languages it seems more portable (and one less thing to remember). It also seems easier to debug.

mschuster91• 12 years ago

For the "greedy" behaviour, PHP has the "U" flag... dunno about other implementations though.

chernevik• 12 years ago

To be picky, it's always what I want, but with a lot of other stuff I don't.

blt• 12 years ago

Very interesting. I've had the greedy .* "overmatch", like probably almost everyone who's used regular expressions. Had no idea they are a performance drain even when giving the right answer though.

I like posts about details of software craftsmanship like this.

larubbio• 12 years ago

Is the early example in the document correct?

Using an input string of abc123 he claims [a-z]+\d+ will match the entire string (which I agree with). He then says that [a-z]+?\d+? will only match abc1. Wouldn't it fail since the non-greedy match on [a-z] would just match 'a' causing the non-greedy match on \d to fail trying to match 'b'?

simcop2387• 12 years ago

no it'll still match but because both are non-greedy it could match on just c1 instead of abc123.

prawks• 12 years ago

It could match on c1, but I believe since most (all?) regex parsers parse left-to-right, it will match the a, look for another a-z character or a digit, find b, repeat, find c, then find 1 which completes the pattern.

gatehouse• 12 years ago

http://regex101.com/r/aR5xM2

I used this tester posted elsewhere in the thread, it seems like since the lazy components expand "as needed" to achieve a match, it will succeed on "abc1".

EDIT: I wrapped it in a group for clarity.

NanoWar• 12 years ago

The regex fiddle is really useful: http://regex101.com/r/qQ2dE4

icambron2• 12 years ago

About eight years ago I finally read Friedl's Mastering Regular Expressions [1]. I know, right? A 500-page book about regular expressions, a tool I already knew (or thought I did). But it's actually a great book-- easy to read and full of genuinely good information on the how and why of regex, and it totally changed my understanding of them. If absolutely anything in this article surprised you, I highly recommend you read the book.

[1] http://regex.info/book.html

bthornbury2• 12 years ago

I once used .* in a crawler. Came back the next day to find much rogue html amongst the content of my site. I find something like [^

bthornbury2• 12 years ago

I once used .* in a crawler. Came back the next day to find much rogue html amongst the content of my site. I find something like [^{{delimiting character}}]* to be better

spb• 12 years ago

This is why I like Lua's '-' for non-greedy matching in its pattern facilities.

Why Using .* in Regular Expressions Is Almost Never What You Actually Want

Related Stories