Ask HN: what could go wrong?

https://news.ycombinator.com/item?id=8831733

by dhbradshaw • 12 years ago

3 5 12 years ago

news.ycombinator.com

I'm building a website that lets people aggregate the numbers they care about into one spot.

Right now, the group that it is most popular with is authors, who use it to get alerts when they get a new review on Amazon.

They have suggested that I make it possible to track their author rank on Amazon. I've been playing with that and I have found that regex is a nice way to go for that particular job. (I've been using xpaths and selectors up to this point.) So soon I'll probably add that as a specialized function to my website.

Because regexes are so useful (not for parsing but for finding known patterns), I'm tempted to make it possible to create automatic scrapers using regexes. But it seems the kind of thing you want to research a bit first.

mtmail• 12 years ago

Does your target audience understand regular expressions? I like the approach import.io took: you go to one or more pages with their browser, select the fields you're interested in and they build the extraction (xpath, css selectors) for you. An engineer can take that configuration and instruct the scraper to call a URL and get JSON back. Even with their special browser, help pages, videos I had trouble explaining it to a non-technical person.

"Normal" regular expressions are probably fine. Only with back-tracing or look-forward it might be possible to create complexity so a regex takes too long. Wrapping it into a block with fixed timeout should work.

dhbradshaw• 12 years ago

Good question about the regular expressions. The Indie author community is made up of a bit of everyone. They learn aggressively and they communicate well. It seems to me that if I make something available it will soon be adapted to a use and shared.

import.io looks awesome. I was also looking at ParseHub, which seems very impressive.

Thanks for the idea of a fixed timeout.

romanhn• 12 years ago

Do take a look at Google's RE2 library (https://code.google.com/p/re2/) - it is designed to handle user-entered regexes safely.

rtcoms• 12 years ago

There is a ruby gem for friendly regular expression builder

https://github.com/andrewberls/regularity

joshschreuder• 12 years ago

Check out KimonoLabs too which seems to do a similar thing to import.io. I think it lets you do regex also but can't remember.