Perl script to dump the urls submitted by an HN user

https://gist.github.com/939629

by adulau • 15 years ago

26 14 15 years ago

gist.github.com

spudlyo• 15 years ago

What makes this program so small is the Scrappy library from CPAN, which gets most of its muscle from Web::Scraper and WWW::Mechanize.

I think you'll find if you try this program yourself, you're going to be spending a lot of time downloading and installing CPAN dependencies, few of which can likely be satisfied by your OS's package manager.

mapgrep• 15 years ago

The `cpan` command line utility automatically downloads and installs dependencies when you use it to install a module. It's the rough equivalent (and precursor to) ruby's `gem`.

telemachos• 15 years ago

I very strongly recommend cpanminus (aka, cpanm)[1] over cpan now. It's zero configuration, lighter on memory use and much more straightforward for most uses.

[1] https://github.com/miyagawa/cpanminus

fragmede• 15 years ago

Far from perfect (regexps on html???), but; it suffices to get latest 22 submissions:

  wget -q -O - http://news.ycombinator.com/submitted?id=adulau | grep -o 'title"><a href="[^"]*" rel="nofollow">[^<]*<' | sed -e 's/title"><a href="\([^"]*\)" rel="nofollow">\([^<]*\)</\2 - \1/'

mike-cardwell• 15 years ago

Without the crazy dependencies:

  #!/usr/bin/perl
  use strict;
  use warnings;
  use LWP::Simple;

  my %urls = ();
  process( "http://news.ycombinator.com/submitted?id=$ARGV[0]" );
  print join("\n",sort keys %urls)."\n";
  sub process {
     my $html = get( $_[0] );
     $html =~ s#<td class="title"><a href="(http[^"]+)#$urls{$1}++#eg;
     process("http://news.ycombinator.com$1") if $html =~ m#<a href="(/x\?fnid=[^"]+)" rel="nofollow">More</a>#;
  }

melling• 15 years ago

Anyone have a Perl script to get all my saved stories, with URL, title, and date?

pyre4• 15 years ago

It's a little rough, as I whipped it up in response to this article (it's something that's been on my todo list), but here it is:

https://github.com/bsandrow/hn-profile

Gambit89• 15 years ago

I wrote a Python script a while back to save your saved stories to a local SQLite database, but iirc, it doesn't save the date (dates are show "days ago" in our profiles, so it's relative and not an exact time, and I wanted an exact time).

https://github.com/achea/linkhive

rjbond3rd• 15 years ago

Nice script would be even better with these two (virtually mandatory) lines near the top:

use strict;

use warnings;

pyre4• 15 years ago

Or:

  use common::sense;

http://search.cpan.org/~mlehmann/common-sense-3.4/

Perl script to dump the urls submitted by an HN user

Related Stories