ChangeDetection, monitor any website change

https://github.com/dgtlmoon/changedetection.io

by serhack_ • 3 years ago

253 56 3 years ago

hdivider4• 3 years ago

Good stuff. Niche idea for ya: consider government contractors. Gov sites tend to be atrocious. Manual change tracking can mean you find a $1M RFP weeks earlier, which increases chance of winning. (Consider mda.mil. There's no email signup. What they consider a 'newsletter' is in fact a list of PDFs hosted on their site.)

And that's federal gov contracting. State/local is even more of a Wild West. So much so that many, maany companies just never have the bandwidth to even find RFPs and navigate those ancient sites.

solardev2• 3 years ago

Funny how much better government might get if we just let their employees post on Craigslist

MoOmer3• 3 years ago

While that may be true if contractors/competitors for contracts were generally unaware of the contract until posting, that’s usually not the reality. A lot of government contracting advice from experienced contractors is that if you’re discovering it via a public source (i.e. the website) you’re too late.

hdivider4• 3 years ago

This I am afraid is not true either, though. I know many dual-use companies who discovered an RFP (not just on SAM), late -- i.e. with a proposal deathmarch -- and got through.

chatmasta• 3 years ago

How can you be "too late?" Isn't the point of RFP postings that they're the first step in a process of soliciting proposals (bids) for the requested project? They're certainly not supposed to be granted on any kind of first-come, first-served basis... combatting that sort of corrupt insider dealing is the whole reason there are legal requirements to post them in the first place. So while I don't doubt the premise of your cynical truism that "if you're discovering it via a public source you're too late," I hope that this is a separate issue from the RFP process itself, because any project where that's the case must be a project where the RFP was meaningless in the first place. And that behavior is also explicitly illegal, even if it's unfortunately prevalent and infrequently prosecuted.

serhack_4• 3 years ago

I'm OP. I'm not the developer of the project, but I think it deserved a post on HN because it's pretty an useful tool to track website changes. Setup can be done in a few minutes and the lead developer is always looking for solid feedbacks.

TekMol• 3 years ago

It cannot be a Show HN then:

https://news.ycombinator.com/showhn.html

Please change the title.

serhack_4• 3 years ago

Uops, sorry

kierenj• 3 years ago

First time I've noticed a GH repo having so much SEO in the About section! Not sure how I feel about it..

serhack_4• 3 years ago

I actually found it via the about section so I guess it helped haha

realPubkey• 3 years ago

They should add an url into the description text. This will render as backlink any there are many pages out there which replicate github-topic pages which will then give your site many links for free.

metadat• 3 years ago

IIRC, GitHub adds the nofollow attribute to all user-supplied links links.

Lalabadie• 3 years ago

Right? First time I run into a README file on Github and I think "Oh, this text is for search engines, not readers".

mnstngr• 3 years ago

It's actually refreshing to see end-user documentation and answering the basic questions of “what can this do” and “why should I care” in the readme. Most projects jump straight into how to build it locally, with not even a screenshot of what they’ve created.

champagnepapi• 3 years ago

How does this handle A/B tests that sites are running?

masukomi• 3 years ago

wouldn't that depend entirely on which group its request got put in, which would be something it would have no control over?

Aulig• 3 years ago

I've been self-hosting this for a year or so and it's pretty neat.

It happens quite often that I need to wait for something which doesn't have a builtin alert.

Currently I'm for example using it to get an alert when a flutter package has been updated on pub.dev

wkat42423• 3 years ago

Interesting! I've been making some scripts privately for specific tasks, like an Nvidia video card I wanted to buy. However many sites are really hostile to scraping these days. I'll give it a try,.

weinzierl2• 3 years ago

camelcamelcamel is a really old service, but it's still around and still works well for product price changes.

wkat42423• 3 years ago

Yes but they collaborate with Amazon. For example during the pandemic they turned it all off for a few months because Amazon wanted it.

They're still around because Amazon lets them be around.

Mo3• 3 years ago

Kudos, this is a very feature rich solution. I like it. 8.99 seems a little bit much, but I guess it's probably the sweet spot on the curve.

serhack_4• 3 years ago

You can install it for free on your machine. I think it's a decent price, considering that they offer you to check a website even with a ratio of seconds.

johnnyworker• 3 years ago

For Windows, there is the OG https://www.aignes.com/

abbaselmas• 3 years ago

my favorite one is visualping.io

_flux• 3 years ago

I've used https://urlwatch.readthedocs.io but this definitely seems much easier to use—though possibly not quite as powerful regarding page filtering. But at least ChangeDetection supports jq which is already quite a nice feature in that department.

shortrounddev2• 3 years ago

A few years ago when I was a Junior engineer, I worked for a company which would provide us with free tickets to any sports team in our city. I'm not a sports fan, but my stepdad loves hockey. When the tickets went up, they got reserved pretty fast.

I noticed that the site we used to reserve the tickets had a predictable slug like: `{company}-hockey-tickets-2016-season`, and `{company}-hockey-tickets-2015-season`. The sites also didn't change between each refresh.

So, when it was nearly time for the 2017 season to start, I wrote a script which hashed a GET request of the site at `{company}-hockey-tickets-2017-season`. If the hash changed, it would send me a text message using twilio, and I would know to immediately get on a machine and reserve the tickets we wanted. After a couple false positives, the page eventually went up before it was announced and I reserved the tickets I wanted.

Unfortunately, the office manager who did these things told me I couldn't jump the gun and un-reserved my tickets. After the official announcement went up, the spots I had originally reserved quickly went to someone else

HardlyCurious• 3 years ago

Yeah, that office manager probably knew who would be getting the tickets in advance.

alach11• 3 years ago

Working hard: writing a script to detect when the website goes live.

Working smart: bring the office manager a pastry, ask them about their day, and drop hints about how excited you are for the tickets this year.

xenocratus• 3 years ago

And that's when I'd make that script and the knowledge behind it available publicly and advertise it on social media :)

solardev2• 3 years ago

It seems like bad company policy to let a bunch of techies fight over a limited resource via web requests. It's like network inspector gladiators. Maybe that was the point? Lol

mhb• 3 years ago

Why is the hash of the request better than comparing strings?

rented_mule• 3 years ago

I assume they meant they were hashing the response rather than hashing the request. Then you don't have to keep the entire response around for comparison, only its hash.

pc86• 3 years ago

Content on the page with slug `{company}-hockey-tickets-2016-season` changed, not the slug itself.

accidbuddy2• 3 years ago

Anyone has been utilized this tool to monitoring Amazon Prices (with login)?

stewx• 3 years ago

Use CamelCamelCamel

accidbuddy2• 3 years ago

As far as I know, doesn't work to Amazon.br (Brazil). Thanks by the reply.

cloudyporpoise• 3 years ago

Curious how this handles sites behind CloudFlare using bot detection.

_lvbh• 3 years ago

I see selenium and playwright in the dependencies. I would’ve just gone with tls-client

_boffin_• 3 years ago

It’s not too difficult to get past cloudflare using puppeteer. Being normal is the key.

mcstempel• 3 years ago

Yeah, +1. Even vanilla puppeteer is pretty successful against Cloudflare

sccxy• 3 years ago

I have used it for some time now. It has been very useful for me.

Some use cases for me:

* price changes

* calendar changes for sport events

* document changes for local gov

* new firmware releases

* terms of conditions changes

ephbit• 3 years ago

Here's an addon for Firefox that also checks websites in regular intervals to detect changes:

https://github.com/WaldiPL/webpageScanner

Not affiliated with it and I don't know if it still works, haven't used it lately. When I used it, it worked to my satisfaction.

mchenier• 3 years ago

https://visualping.io does a nice similar job and is free for moderate personal use.

davidinosauro• 3 years ago

On the theme of alternatives, I've been using https://www.followthatpage.com/ on and off for years. In particular for my partner's job hunt, watching some local trade association billboards or single school's `/jobs.html` pages.

Also has a reasonable free plan for personal use (20 pages daily + 20 pages weekly + 1 page hourly).

Definitely much simpler in terms of diffing (only text), but it has this 2000s vibe.

ChangeDetection, monitor any website change

Related Stories