You Should Learn Regex

https://blog.patricktriest.com/you-should-learn-regex/

by quotable_cow • 9 years ago

90 47 9 years ago

blog.patricktriest.com

tmdvs• 9 years ago

One website I found extremely useful when initially learning regex was http://regexr.com. It's got a great pattern builder, and hovering over each component explains what its actually doing.

lowmagnet2• 9 years ago

From validating email addresses

Guys, no. Send an email. https://davidcel.is/posts/stop-validating-email-addresses-wi...

kuschku• 9 years ago

That works if you have content that you know is an email address.

If you have free-form content, and need to identify email addresses in it, then you can’t just email every subset of that text.

lowmagnet2• 9 years ago

That works if you have content that you know is an email address.

That's what validation is. You're talking about finding email addresses, which I agree that regex is relatively OK at doing, but there are still lots of edge cases in the spec that are irregular.

mhd• 9 years ago

Depends on the use case. For forms, a preliminary sanity check doesn't hurt, e.g. when the user enters their username and password, instead of email & password (accepting either would be best, but you can't always do that).

tyingq• 9 years ago

Mailcheck is great for this. People constantly make typos when entering emails. This utility suggests john.doe@gmail.com if they type in john.doe@gnail.com. But, it's their choice...it just suggests corrections.

My logs show that it helps a lot. Bounced emails from orders dropped by more than half.

https://github.com/mailcheck/mailcheck/blob/master/README.md

Invisig0th2• 9 years ago

The article you link to addresses just the one use case where you are registering a new user. Email addresses in business applications are just as often NOT the address of your user. Sales contacts, managers, points of contact, customer support addresses, etc. -- none of these should ever be validated by sending an e-mail. So you still need to validate the hard way in plenty of scenarios.

kqr3• 9 years ago

...but unless you actually send an email it is still just as unvalidated as it was before. Regexes aren't a tool to determine that a string refers to a mailbox capable of receiving mail.

If you use regexes (or any other method of that does not send emails) all you're saying is that you don't actually care whether or not the string points to a recipient (much less the correct recipient).

And don't get me wrong. It is absolutely okay to not care whether or not the string refers to a correct recipient. Most places that make me write my email have no business caring about it. But please also then make the field optional.

davidkuhta• 9 years ago

In fairness, the author does make note of that:

Note - In a real-world application, validating an email address using a regular expression is not enough for many situations, such as when a user signs up. Once you have confirmed that the input text is an email address, you should always follow through with the standard practice of sending a confirmation/activation email.

cies• 9 years ago

A very extensive article, and it shows how powerful RXes are.

It makes a point, in "8.3 - For Problems That Don't Require Regex" that RXes should not be overused when a simpler solution exists.

But there are also problems that are better solved without RXes, but with a "parser" instead. The article mentions parsers, fails to mention that self-made parsers also provide solutions in the same problem space as RXes. Some communities (like the Haskell community) prefer parsers as they allow to be inspected, tested and typed. Parsers provide a more robust and hackable solution, that is possibly faster than the equivalent RX.

Here a short tutorial for a popular parser library in Haskell, to get an idea of what it's like:

http://akashagrawal.me/beginners-guide-to-megaparsec

le-mark• 9 years ago

Kleene Algebra, generalized regular expressions, for the adventurous [1].

[1] https://en.wikipedia.org/wiki/Kleene_algebra

souriguha• 9 years ago

OP, your link to http://regex101.com/ is broken. It's my go-to web-app too.

quotable_cow3• 9 years ago

Thanks! It's fixed now.

Sean1708• 9 years ago

Also the link in this paragraph:

The source code for the examples in this tutorial can be found at the Github repository here - https://github.com/triestpa/You-Should-Learn-Rege

has a `href` of `""` which is making it point to the blog post itself.

submeta2• 9 years ago

When I used to code in Perl (more than ten years ago) I tended to solve many many things using regexes. I tried to reduce many programming challenges into transformations that I could do via regexes. Why? Because regular expressions are first class citizens in Perl. And Perl made me learn and use them. (Reminds me of a an APL programmer that saw vectors everywhere.)

When I replaced Perl with Python in my toolbox I abandoned regexes altogether because I didn't know too many Python programmers using regexes.

Now a decade later I am realizing how often I do need regexes in my day to day tasks (using them in bash commands, in my editor, and even in Python sometimes).

marktangotango• 9 years ago

Some elucidation on this point:

https://blog.codinghorror.com/regular-expressions-now-you-ha...

submeta2• 9 years ago

Great read. Thanks for sharing.

brightball2• 9 years ago

I found the same thing. When I was writing Perl half the point was using regex. I inherited a site about 5 years ago that was a huge Perl site built entirely on text files and regex. The developer had basically built a database out of text files. He had directories for indexes, associations, etc.

It wasn't pretty to look at but it was still to this day the fastest site I've ever worked on.

tzahola• 9 years ago

How did he achieve transactionality?

philipov2• 9 years ago

Regex patterns might not be first class citizens in python, but they're easy enough to use, and I think they're ultimately more powerful than in Perl. I don't remember being able to use a callback to calculate my replace string as a function of the match results when doing substitution in Perl. It's super easy to do tokenization with python regexes. We lose first class patterns, but gain first class functions and classes :)

kqr3• 9 years ago

Not to mention that writing a composable first-class API to contruct regexes should be fairly elementary in Python.

bmn__• 9 years ago

I don't remember being able to use a callback to calculate my replace string as a function of the match results when doing substitution in Perl.

That's easy with the `e` (eval) flag.

    say '13 37 123 42' =~ s
        { ( \d+ )           # get all the numbers }
        { my $total += $+;  # replace with running total }egrx;
    __END__
    output is 13 50 173 215

> I think they're ultimately more powerful than in Perl

No. Perl regex are more powerful than Python because

1. the programmer can pick from a large selection of delimiters

2. the full-featured engine is built-in and ready to use, but in Python you need to install the third-party "regex" module because the built-in one named "re" is hopelessly dyd¹

3. you may enable verbose (readable) character classes

4. the regex subsystem is pluggable with a common interface and you can within a lexical scope switch at run-time with more restricted implementations which have less features but better performance in some cases: https://metacpan.org/search?q=re%3A%3Aengine%3A%3A

5. you can embed arbitrary code with (?{...}) and (??{...})

We lose first class patterns, but gain first class functions and classes :)

Perl had first-class functions since 1993.

You love having first-class classes in Python, but do not realise the downside. Python's one true meta object system (with its ageing design from the last millennium) is flawed: horizontal composition of methods with the same name does not emit a warning. Since the system is baked into the language, it cannot be changed. Contrast with Perl, where the language only provides some primitives on which to build a meta object system (incl. first-class classes). This enables bugs to be fixed, and the competition among several implementations on CPAN breeds excellence.

¹ https://i.imgur.com/66z4KDA.jpg

habeebtc• 9 years ago

"Now you have two problems" . ;)

Tepix• 9 years ago

Regular expressions are super powerful and a great tool to have at your disposal. The best resource on the topic is the book "Mastering regular expressions" published by O'Reilly. Understanding the inner workings of a regular expression engine will improve your regular expressions. If you use regular expressions a lot, mastering them will be worth it.

yenwel• 9 years ago

Yea learn regex, so you know when not to use it. For parsing non regular languages like XML...

seven800• 9 years ago

I created a little web app to generate strings which match a given regex: http://regexicon.com

It's useful to ensure you're not matching things that you don't mean to, which is a problem I often see when code reviewing regex.

asicsp3• 9 years ago

haven't read/processed the article fully, some nitpicks:

* `\b([01]?[0-9]|2[0-3]):([0-5]\d)\b` you are using both [0-9] and \d, is that intentional?

* `cat test.txt | grep -E "^[0-9]+$"` UUoC and double quotes subjected to shell interpretation, should be `grep -E '^[0-9]+$' test.txt` or `grep -xE '[0-9]+' test.txt`

* `(?i)` won't work with `grep -E` (or at least not for me on GNU grep). features of BRE/ERE and PCRE like regex are very different - see https://unix.stackexchange.com/questions/119905/why-does-my-...

* avoid parsing ls - https://unix.stackexchange.com/questions/128985/why-not-pars... , use glob/find

* `.<star>?` again, this regex feature is not available with BRE/ERE, your example just happens to work. you can check it with `echo 'abc foo 123 bar 123' | perl -pe 's/foo.<star>?123//'` and `echo 'abc foo 123 bar 123' | sed -E 's/foo.<star>?123//'` (using <star> to avoid formatting issues)

quotable_cow3• 9 years ago

Thanks, that's all very useful feedback.

asicsp3• 9 years ago

for the ls parsing example, you can do

shopt -s nocaseglob

ls ~/Downloads/*.{png,jpg,jpeg,gif,webp}

----

for command line tools (grep/sed/awk/sort/etc), you can refer my ongoing project (https://github.com/learnbyexample/Command-line-text-processi...) as resource :)

spoxaka• 9 years ago

I just have to ask. Doesn't the above comment kind of defeats the article premise?

papey• 9 years ago

Why the fuck on earth, people cat and pip stuff to grep ? grep cat take files from args !

yorwba• 9 years ago

I cat files into programs that can take their inputs both from stdin and a given file, although I know people call that "useless". I do it because it makes my pipelines more uniform, by keeping the direction of data flow from left to right. It also makes it easier to insert additional processing steps, or changing the input from a file to the output of a program. Maybe I'm a fool for writing Bash like Forth, but I prefer it that way.

asicsp3• 9 years ago

you can still do `< test.txt grep ...`

see https://en.wikipedia.org/wiki/Cat_(Unix)#UUOC_.28Useless_Use...

Beyond other benefits, the input redirection forms allow command to perform random access on the file, whereas the cat examples do not. This is because the redirection form opens the file as the stdin file descriptor which command can fully access, while the cat form simply provides the data as a stream of bytes.

commands like `sort` are optimized to handle large input file

and how would you do `grep -l 'foo' *.txt` if you use `cat`?

or `awk 'NR==FNR{a[$1]; next} $1 in a' file1 file2`

or `grep -Fxf file1 file2` and so on....

oldoverholt• 9 years ago

The OpenBSD re_format man page is a quick read and super clear and helpful: https://man.openbsd.org/re_format

j_m_b• 9 years ago

...but..but.. then you'll have two problems!! right guys?

fredrb• 9 years ago

Section 0:

`[0-9] - Matches any digit between 0 and -`

Mistype the `9` for a `-`. :-)

quotable_cow3• 9 years ago

Thanks, good catch!

jlg23• 9 years ago

For those who prefer a more in-depth introduction than a blog post: Jeffrey Friedl's "Mastering Regular Expression"[1] covers the theory as well while still being very pleasant to read (I only know the first edition, apparently there are more recent ones).

[1] http://shop.oreilly.com/product/9781565922570.do

Endy• 9 years ago

I'm not a programmer, but I use Notepad++ and mp3splt, because I have uses for them. This might actually help me use the search functions more effectively.

ndr• 9 years ago

Some years ago I went through Regular Expressions Cookbook, By Steven Levithan, Jan Goyvaerts and can't recommend it enough.

henesy• 9 years ago

The Sam language and editor may be of interest:

http://sam.cat-v.org

http://doc.cat-v.org/plan_9/4th_edition/papers/sam

not_kurt_godel• 9 years ago

This sounds like an article I would have written in college. In my experience, if you're using regex often enough to remember how to write patterns that are sufficiently complex to actually be useful beyond short 'pre-processing' snippets, you're overusing regex. (Or alternatively, your memory is significantly better and/or less divided than mine.)

You Should Learn Regex

Related Stories