The Craigwatch Story

by stickfigure • 12 years ago

146 48 12 years ago

kazinator2• 12 years ago

In the past, I have sometimes used Craigslist's RSS feature to watch for the appearance of items. That is to say, the ability of CL to export a keyword search as an RSS feed that you can save in your feed reader and watch for updates.

In what ways did Craigwatch improve over RSS? If it still operated today, why would I use that instead of scraping RSS items directly from Craigslist to my feed reader?

Did Craigwatch use RSS feeds, or was it scraping material from the rendered HTML?

pkulak• 12 years ago

I do that too, but it can be pretty late. At least hours after the item is posted. Which isn't bad for some things, but sometimes you _really_ want to know. :D

I've never actually heard of Craigswatch, but I think it's something most engineers have done at least a couple of times. I've got a script I use to watch the FredMiranda forums any time I need some new camera gear, for example.

dpritchett• 12 years ago

Looks like it was using XPath to parse the HTML: https://github.com/beaulm/craigwatch/blob/master/post.php

I imagine the 'selling' point was users not even having to know what RSS even means, let alone how to script it.

ErikRogneby• 12 years ago

Thanks for the link. Time to try out Google app engine's PHP preview.

wiseleo3• 12 years ago

Useful free stuff is typically gone in minutes. Only junk remains posted.

Its RSS feeds have a refresh period of 1 hour, which is simply too long to have the opportunity to grab hot items.

As Craigslist has very shallow category listings in its "free" category, where you can't separate furniture from clothes as an example, the site forces users to rely on manually refreshing pages, which change very often.

I have better user experience using Craigslist from my iPad. Craigslist appears to selectively make the service available to app makers but not to websites.

In the past 3 days of shopping for a specific car within about 15 Craigslist local communities, I managed to get automatically blocked by the service several times. What I am looking for is very specific and I don't mind traveling a bit to get what I want.

I ended up having to write code with sleep timers to reduce my number of web queries once I narrowed what I wanted.

An essential feature in the iPad app is that I can cross off listings that are no longer interesting to me and highlight favorites. There is no listing cross-off feature for the website.

I guess my next step is to add that feature. laugh

kazinator2• 12 years ago

Wiseleo, is that really true about the refresh rate? RSS readers have a configurable rate. Often there is a one hour default in order to be reasonably nice to the server. Basically, a Craigslist RSS item is just a URL with the search parameters embedded. Maybe I'm wrong, but I suspect that whenever you fetch this URL, the server executes the search and produces the results as RSS XML items, so the fetch rate is controlled by you (your reader). Or are you saying there is some additional throttling on the server side, so that RSS-based searches do not see up-to-the-second updates that are visible through the Web interface? So that no matter how often you refresh the RSS feed, you don't see new items that are already visible via HTML?

marincounty• 12 years ago

I used to give away good free stuff, but it's just not woth the hassle(going to the wrong people, people not showing up, the spam, over agressive Flaggers--you people need to get a life, etc.)

While I'm here. What's with the begging to lower an already low price on a service, or item. I used to price an item realistically, then knock off 40 percent. People still want to Haggle over the price. I now just double the price I think it's worth.

There's a huge need for another Craigslist. The site has gotten too Ghetto.

dpweb• 12 years ago

I would think CL using a proxy that silently blocks client ips that interact with the service suspiciously may be cheaper than sending out lawyer letters and more immediately effective. Just prefer technical solutions to legal threatening I guess..

pyoung• 12 years ago

For fun and as a learning experience, I built a simple CL scrapping web site. I set it up on a small aws instance about 6 months ago and after playing around with and making modifications/updates to it over the course of a month or two, I largely forgot about it. After seeing this thread, it dawned on me that I had not received an e-mail update on any of my search alerts for a while. I ssh'ed into the server and got a 403 with 'wget craigslist.com', so my guess is that they are doing some sort of blocking. Looking at the logs, the block probably started about a month ago.

Wingman4l7• 12 years ago

Modifying the user agent to mimic a normal browser might solve the problem.

stickfigure2• 12 years ago

I used Craigwatch. It was a useful tool - when a search pops, I get an email. I used to use Google Reader for this but when Reader died so did my RSS reading habit, so now email is the only part of my daily workflow that I poll frequently.

The real mystery to me is why Craigslist shut the service down, because (protocols aside) it wasn't fundamentally different from any other way of consuming craigslist. Craigslist's terms of service is actually pretty confusing:

USE. You agree not to use or provide software (except for general purpose web browsers and email clients, or software expressly licensed by us) or services that interact or interoperate with CL, e.g. for downloading, uploading, posting, flagging, emailing, search, or mobile use. Robots, spiders, scripts, scrapers, crawlers, etc. are prohibited, as are misleading, unsolicited, unlawful, and/or spam postings/email. You agree not to collect users’ personal and/or contact information (“PI”).

...which if you think about it, bans the use of RSS readers too. There's a short blurb on the RSS page:

craigslist RSS feeds are for your personal use only, and are not available for commercial use without first obtaining a license from craigslist. Please consult our <Terms of Use> for more information on using craigslist RSS feeds.

The only relevant part of the Terms of Use is the section mentioned above, which doesn't really shed any light on the situation.

Honestly this is really confusing. Once again, Craigslist disappoints.

jstanek2• 12 years ago

Craigslist has a really bad history of abusing its users. For instance, a few years back they changed the TOS so that CL becomes the copyright holder of all ads placed on the site.

pessimizer• 12 years ago

So trying to fend off padmapper (whether one agrees with that or not) has suddenly become "a really bad history of abusing its users"? I guess scrapers are users.

shakethemonkey• 12 years ago

You can safely disregard that part of the TOS. Because law:

(a) A transfer of copyright ownership, other than by operation of law, is not valid unless an instrument of conveyance, or a note or memorandum of the transfer, is in writing and signed by the owner of the rights conveyed or such owner's duly authorized agent.[1]

[1] http://www.copyright.gov/title17/92chap2.html

yonran3• 12 years ago

By my reading, prior to Dec 2013, the old terms[1] permitted using personal RSS readers but forbade aggregators. However, the new terms are much more restrictive; they prohibit any use of the craigslist website outside of a web browser. Thus, I think that it is now against the terms to use an RSS reader to read the craigslist RSS feeds! (By the way, it is also against the craigslist terms to write a crawler such as google.com or archive.org even if it respects robots.txt!)

I happened to review these terms because I too was sent a letter from Brian Hennessy representing craigslist last Friday regarding a little Chrome extension I wrote a couple years ago[2].

[1] https://web.archive.org/web/20131128233421/http://www.craigs...

[2] https://github.com/yonran/craigslist-shortcuts

stickfigure2• 12 years ago

How can Craigslist accuse you of violating their terms of use? The act of providing a chrome extension does not use their service. Maybe if this was some sort of copyright-circumvention software covered by the DMCA I could understand it. But this sounds like an obvious legal bluff, no?

IANAL, which is why I am asking these questions.

mkaziz• 12 years ago

I'm curious as to whether you got a response to your refusal to "comply." Did he get back to you after that?

gcb02• 12 years ago

another service i used a lot was padmapper(?) (CL killed and then stolen the feature) and then another site that allowed me to search on all craigslist sites around the US. i used it to check the price of whatever i was buying locally. or to find things so rare around i am that i would be willing to risk having my money stolen and nothing delivered... i don't even remember the name of the site, but nothing have come even close to filing the gap. one would think ebay is the logical solution, but ebay is filled with crap that it is useless.

8ig8• 12 years ago

A while back, I hacked together notifications using RSS, but lately, I just use IFTTT...

https://ifttt.com/craigslist

pwenzel• 12 years ago

I use IFTTT's craigslist channel all the time for long-term polling of craigslist. Not sure how instant it is, but notifications are pretty reliable.

SilasX• 12 years ago

Do they get CL's permission to run that? Because it looks like it violates the ToS as written.

atomical• 12 years ago

Did craigwatch.com agree to the terms on the site?

jstanek2• 12 years ago

There's probably a provision in the TOS that says that you implicitly agree to the terms by virtue of using the service. I don't know for sure, just speculating.

CanSpice• 12 years ago

So will IFTTT get a cease and desist sometime in the near future, or have they come to an agreement with Craigslist to provide their channel?

melvinng3• 12 years ago

Stand up to the MAN!!!

melvinng3• 12 years ago

Everyone is too law abiding.. But really what laws is he breaking here?? Trademark? Copyright?

Spoke to my legal counsel and I don't see either..

unreal37• 12 years ago

Terms of service are a contractual agreement between a service and its users. There's no laws being broken, but it's breach of contract.

Your legal counsel would know that I suspect.

xsmasher• 12 years ago

http://en.wikipedia.org/wiki/Trespass_to_chattels#Beyond_spa...

kalleboo• 12 years ago

CFAA?

dbla• 12 years ago

A little bit off topic but interesting nonetheless.

I've known Beau for many years now and his writing style has improved so much. He is a testament to the "practice makes perfect" (or better at least) mantra. It's really interesting to see the transition in his travel blog from when he started writing to now. If you're interested you can check out. Compare the first post to the most recent. http://dangertravels.com/

tabrischen• 12 years ago

Thanks for sharing, his travel stories are inspiring

intellegacy• 12 years ago

it's a great story.

BTW, can anyone give me perspective on what it would take to recreate this in Python? As a learning exercise

Is it possible to do? how long would it take for someone relatively new, etc.

bnejad• 12 years ago

With all the libraries available for parsing and scraping I suspect this could be recreated very quickly.

houston-mouse• 12 years ago

One thing I like about this reflection is the highlighting of the weirdness of the connection between a service's creators and its users. I'm on both sides of the fence, and I find myself wrestling with lots of different feelings towards my users (mostly positive, but not all). I guess this dynamic has existed since the printing press, but it feels a bit more intimate now since you can see the emails and read the tweets.

jarofgreen• 12 years ago

This is a very good point, and something that was also touched on in http://www.saurik.com/id/20 recently.

mkolodny• 12 years ago

I'd be willing to be that Craigslist shut you down because they're building their own "watch" feature. The same thing happened with Padmapper - they sent them a cease and desist letter, and then came out with their own map feature (http://newyork.craigslist.org/aap/#map).

laxk• 12 years ago

Does it mean that any browser extension which is modifying CL web page(s) violating their terms of use?

jijji2• 12 years ago

I ran a system that was a back end for craigslist for many years, and I never stopped even after getting a cease and desist from them. What really got me was after they started charging people to post ads... that really killed my business model. Generally its not a great idea to build a business model around the availability of a third party... I think the whole concept of craigslist sending a 'cease and desist' over what is generally considered public_html is a laughable joke in my opinion, and good luck trying to restrict public html from crawlers and robots.

asaamaraa• 12 years ago

mmm

The Craigwatch Story

Related Stories