Show HN: Sleeping Spider – Web page change monitor

https://sleepingspider.com/

by linklet • 12 years ago

31 38 12 years ago

sleepingspider.com

akx• 12 years ago

Typo in the footer: "Sleeping Sider"

sleepingspider13• 12 years ago

Thank you. Fixed.

etw2• 12 years ago

What's your twitter id. You're trending on PH.

jmomo• 12 years ago

Creepy sleepy drinky spider cider.

I feel a children's book coming out of this.

seivui• 12 years ago

Which timezone does "8 o'clock in the morning" consider?

sleepingspider13• 12 years ago

It will try to detect your time zone with JavaScript. If that doesn't work, it falls back to Los Angeles and the user may contact the support to update this setting.

callesgg• 12 years ago

What is regarded as a change?

sleepingspider13• 12 years ago

Text content changes. For example the site corrected a typo or published a new post.

Change <h1> to <h2> will not trigger an alert.

lucasjans• 12 years ago

Adding watch results in a white screen of nothing, no new watches added. I'm trying to monitor the Nexus 6 page :D

https://play.google.com/store/devices/details/Nexus_6_32GB_C...

For my use case, anything more than an hour delay wouldn't be practical. Ideally it would be up to the minute, but I'd imagine that would cause other issues for a service like this. In the past I just used a Chrome Extension. But living in Vietnam I have to make sure my VPN is up in order to see the North American content.

sleepingspider13• 12 years ago

The probe received a 403 forbidden error from google. I realized that google store forbid crawls. So a simple http request won't work.

I was not aware of this situation before. I will see what I can do.

_asciiker_• 12 years ago

simple, clean and useful. I'm trying it out. Good job!

TheDong3• 12 years ago

Does this site respect robots.txt?

Does it render javascript content (so e.g. <div ng-view> angular sites work)?

Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes?

Will I get a notification if the page briefly 500 errors when you hit it, or does it have to get a 200 before it reports a change?

Will I get a notification if a page disappears entirely?

Does it scrape only once per day or more often and alert only once a day?

What user-agent does it scrape from on the off-chance someone wants to block it?

Does it diff binary data.. for example, if I ask it to watch an image url will it hash the image and notify me when it gets updated?

Does it auto-detect pages with rss feeds or pubsub feeds and subscribe to them?

Does it understand twitter so if I watch a user, it only alerts me of tweets, not e.g. a change in the number of followers?

Does it have special code for any website whatsoever?

Is the webpage parsing/diffing library / algorithm open source somewhere?

Is there a monitization plan, and if not how can I trust it won't vanish?

Is there a limit on the number of sites I can watch?

I like the simplistic design, but it would be nice if there were more information.

mattmanser• 12 years ago

Go write your own then.

I find this a bizarre response to a simple and easy to understand tool.

TheDong3• 12 years ago

If it's easy to understand then I guess you can answer my questions?

I don't think "go write your own" is a legitimate response to my questions (if that's ever a legitimate response).

gingerlime• 12 years ago

I thought that at least part of Show HN was to gather feedback and look into how things are implemented, which features are supported / unsupported etc.

Therefore, I personally find this list of questions very relevant. Some of those questions I was asking myself when looking at the page. The fact that this information wasn't clear from the website itself is also important. At least some of the users of such tool would most likely want to know what to expect.

Perhaps it could have been asked in a more positive way (not entirely sure how, but maybe explaining why those things matter), but the questions themselves are important and I'd be curious to know the answers.

aw3c22• 12 years ago

Your post is a bizarre response to a simple list of questions. And the "go write your own then" is kind of offensive.

lurcio• 12 years ago

I find it to be the kind of response to be welcomed (from a product developer/manager viewpoint)

sleepingspider13• 12 years ago

Thank you guys! I've been working on issues and just got the time to add a FAQs page on the site.

See https://sleepingspider.com/faqs for most answers.

It's an early product launch. I have no monetization plan. Too early to plan that.

In the current phase I want to figure out does people like it and how do people use it, how to make it better and serves more people.

TheDong3• 12 years ago

I appreciate the answers.

I mean no offense, but I assume you're not a native English speaker based on that FAQ. I've taken the liberty of rewording them to sound more natural: http://pastebin.com/Kxti3KP0

Whether or not you take any of my suggestions or not, I at least recommend throwing it through a spellchecker.

aw3c22• 12 years ago

If you prefer privacy and control over your own tools, there are the FLOSS http://specto.sourceforge.net/ or the free(as in no cost)ware http://www.cmcode.co.uk/webmon/ for Windows

PavlovsCat• 12 years ago

There is also the commercial, but very extensive Website-Watcher: http://aignes.com/

nousougi• 12 years ago

I too used webmon until recently, but got tired of monitoring the monitoring tool on my desktop. Now I use ChangeDetect - http://www.changedetect.com/ Despite what is said on the homepage, it is NOT free. Good luck...

netman21• 12 years ago

In the most amazing coincidence the first two pages I submitted had the same digest: 90c0568! One was the front page of a news site, the other was the management page of a company I follow. The mind boggles. Or, they have a bug.

edit: as did the 3rd. And 0 bytes returned so probably being blocked. On a black list already?

nacs• 12 years ago

Getting the same digest (90c0568) with no response code and a 0 byte response size so looks like something is broken (the crawler maybe).

sleepingspider13• 12 years ago

Problem fixed. All failed probes retried. Thanks!

watson• 12 years ago

Is it only for HTML, or will it alert on changes to JavaScript files for instance?

sleepingspider13• 12 years ago

It's only for HTML. But if you have a need for JavaScript files, just let me know, it's technically possible, I just don't understand the use case yet.

zapt02• 12 years ago

Nice branding!

ideaoverload• 12 years ago

Looks very similiar to https://followthatpage.com . Worked for me pretty well for non JS pages for last 3 years.

girishso• 12 years ago

I just built "Page Poller" for the Rails Rumble 2014, same concept! http://pagepoller.r14.railsrumble.com/

sleepingspider13• 12 years ago

Just added HTTP authentication support.

sleepingspider13• 12 years ago

Does it diff binary data.. for example, if I ask it to watch an image url will it hash the image and notify me when it gets updated?

Can you elaborate in what case you might want to do this?

sleepingspider13• 12 years ago

Does it understand twitter so if I watch a user, it only alerts me of tweets, not e.g. a change in the number of followers?

What do you think if I provide a numeric filter option to watches, which filters out numbers on the page before comparing if turned on.

sleepingspider13• 12 years ago

Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes?

I don't have a good solution for this problem yet. Let the user specify a css selector or xpath?

Show HN: Sleeping Spider – Web page change monitor

Related Stories