Show HN: Web Scraping Language (WSL)

by scrape_it • 7 years ago

11 10 7 years ago

This is something that I wanted—a rich language for scraping that can help me avoid writing a lot of boilerplate. However, I am not too sure about the P2P/decentralized nature. For my projects, that would be a little too complicated. Would I be able to use just the WSL part?

scrape_it5• 7 years ago

yeah I've actually had a sudden change of heart - implementing a p2p/decentralized theme is quite a bit more work than I anticipated, mainly syncing problems. originally planned on releasing 3 weeks ago....this p2p is taking up almost all of my time and I'm not sure if ppl even give a shit about it....so much for my ideological mania

I'm going to put that p2p stuff on hold and focus on getting a REPL terminal app that you can start running WSL on this week.

russdpale• 7 years ago

decentralized is very important, don't give up on it! Its fine if people don't care about it. People don't care about relational algebra but SQL is used everywhere in their life and they have no idea or care.

The idea of distributed cris-crossed data sets all over the internet, I think, is a powerful one in a day where data is wielded as power and is so asymmetrical in nature.

Fudgel• 7 years ago

What does it use to scrape? Is it just an xml/html parser, or does it run a headless browser?

scrape_it5• 7 years ago

It uses a sort of a DSL that I came up with, and removes a lot of boilerplate.

Headless yes.

You write WSL (Web Scraping Language) to scrape 2 pages of HN:

    GOTO news.ycombinator.com | EXTRACT {.morelink, 2} {news: .storylink}

(you may certainly remove that ' , 2 ' business based on the moral system you subscribe to)

catchmeifyoucan• 7 years ago

Download links are broken

scrape_it5• 7 years ago

I'm working on it as I speak.

slig• 7 years ago

Hi, I see that you used our project's [1] layout as a template for your project, but forgot to change the image and the link on it. Just pointing it out. Good luck on your project.

[1] https://github.com/javascript-obfuscator/javascript-obfuscat...

scrape_it5• 7 years ago

oh yeah forgot about that....its a great template, everything all one one html...many thanks. I will swap out the drawing.

Show HN: Web Scraping Language (WSL)

Related Stories