unsettledtck

We've actually developed a way that you can convert Portia projects into Scrapy spiders: https://blog.scrapinghub.com/2016/06/29/introducing-portia2c...

and since this is all open source, here's a link to GitHub: https://github.com/scrapinghub/portia2code

There is actually a thriving industry in data collection and these range from companies that provide tools for you to do your own data collection and those who will collect the data for you (or provide pre-made datasets). Full disclosure, I work for one such: Scrapinghub where we provide tools, platform, and data services.

We did indeed model our Portia on the real spiders. We'd like to think our version is a wee bit cuter...

Out of curiosity, where are the boundaries of your gray area when it comes to scraping?

Thank you. Glad you enjoy Scrapy, we're pretty fond of it ourselves!

I just ran across https://www.opensecrets.org/ and found it quite useful and comprehensive in tracking contributions to candidates.

I live in the US and am privileged with the level of transparency that exists, but it's still not necessarily enough. Similar issues are present with the clunky nature of government websites and databases and so I think we're in agreement that it's not even close to the potential of what it could be.

Thanks for sharing all the links and information!

Not sure if you're interested, but we (Scrapinghub) do offer a Kimono to Portia migration https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-...

Otherwise, I'd recommend you check out Portia (open source). We're in the middle of releasing the beta 2.0 version.

So for Scrapy and Portia, they are both free as in beer, specifically because we believe in the power of open source. Scrapy actually precedes Scrapinghub and was certainly not developed as a marketing tool.

Charges come with large scale crawls (above certain limits on our platform), additional products like Crawlera (our smart downloader that routes requests from a crawl through a pool of IP addresses to avoid bans), datasets, and for us to handle complex crawls for companies outsourcing to us.

Our model is that there is something for everyone whether you are looking to dip your toes into web scraping (free), use it occasionally (usually journalists) or dependent on web crawling for your business.

Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code.

Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The publicized perception of web scraping is fairly negative, but doesn't take into account the benefits of data used in machine-learning or democratized data extraction (as in the case of this article or for building public service apps like transportation notifications), or the simple realities of competitive pricing and monitoring the activities of resellers.

Researchers, academics, data scientists, marketers, the list goes on for those who use web scraping daily.

Glad you enjoyed the article! I'm hoping that more examples of ethical data extraction will start to turn the tide of public perception.

Full disclosure, I work for Scrapinghub and the web UI you speak of is Portia - our open source visual web scraper. It's for those who range from non-technical to technical but want a quick way to scrape data. I think it's extremely important to develop tools to democratize the acquisition of data regardless of technical background and skill. Glad you find the article and tool interesting!

By overcharging, I mean that we compared prices to a baseline day in January (the 13th, to be exact) and then showed those prices against what was on offer from other retailers. Consumers do have access to this information and their buying decisions are their own. And that's true, this study reinforces the free market system we have. There is something to be said for the shadiness of certain retailers elevating their prices on days that are being touted as "sale" days.

There is a distinction made by those sold and fulfilled by Amazon.com vs sellers on Amazon. Both have inflated prices.

Part of their shadiness is also elevating prices on days that are dedicated to being major "sales" so as to jump on that bandwagon.

haha, I would say touche except for the fact that we never advertised a sale and then elevated our prices.

I think the unethical aspect of arbitrary pricing that goes beyond daily fluctuations really came out in this project. I actually was not aware of the situation with JC Penny, but really fascinating glimpse into their consumer mindset. Probably comparable to Sam's Club, since even I expected a ton of lower prices from them based on membership. I like your cultural analysis!

Very much agreed. And this was very much the case on specific days. Prices do fluctuate, but there were definitely products that were continuously inflated over the course of multiple days.

The fascinating thing about this study is that in certain cases, prices went beyond traditional fluctuations to the point of being inflated for multiple days in a row irrespective of time of day and demand.

Thanks for the support! I agree especially that it's not just random price fluctuation, specifically with the Apple TV example listed in the article. Target dropped the price and then systematically kept it at an inflated rate before dropping it partially again. This was not a case of hourly fluctuations according to demand.

I stick by the title. No, it's not a crime, but there's definitely a disingenuous aspect to overcharging for a product.

I like your extremely accurate title pitch. And no, we're not entitled to low or consistent prices, but the fact that there is inflation on "sale" days is noteworthy because of the air of false advertising. You're absolutely correct though, shoppers should buy when they feel comfortable because buyer's remorse is meaningless when you're happy with the purchase.

This is absolutely true and this study supports your perspective. Guilty is maybe too strong a word, but there is a hint of false advertising. The point of this study was to see how factual the "sales" were that are advertised on Black Friday, Cyber Monday, and Green Monday. We're at a point where people expect to spend their money that those specific days to capitalize on supposedly great deals. I found it interesting to tangibly demonstrate that not only are many of the deals negligible, but that many of the prices were inflated.

Well, actually we didn't believe them, which is why we conducted the experiment. And point of fact, there were actual stores that did offer the lowest price, but you needed to do your research and frankly, get lucky.

If I'm not mistaken (which I might be), there are only specific and regulated periods of the year where sales are actually approved. So it might be an interesting follow up project to track EU retailers. My qualm with the practices of certain US retailers that we tracked was that they offered the inflated prices on the same days as the sales, so that it wasn't even a matter of raising prices in the weeks leading up.

At this point, Black Friday and Cyber Monday have become ritualized cultural events. There is a cult around the supposed "rock bottom prices" on these days that allegedly remove the "research" bit and so we wanted to see if people were actually rewarded by capitalizing on the sales of these particular days.

Agreed, and so I will remain an Amazon shopper in spite of the fact that they (both the company itself and third party sellers) were by far and away the worst when it came to price inflation. Convenience is a part of the cost.

Well, the Toys category had non-electronic items and it was the category that was the most examples of inflated pricing (not such a shocker). However, all of the stores offered discounts on electronics at one point. You just had to be lucky in going to the right store on the right day.

Intriguing! That makes complete and utter sense, which is part of the reason why we included Sam's Club in this scraping project. Wasn't actually aware of the higher per-unit costs.

I agree and it was edifying to find the actual data to back this up. Although, you can see that there were genuinely lowered prices, so there are deals to be had specifically on Black Friday.

This is true and I definitely love the convenience of Amazon (who was the most guilty of higher prices, by the way). Frankly, it was Sam's Club that really surprised me since I always assumed that they offered everything at a lower price as a perk built in for membership.

Fair enough and consumers are rewarded for carefully researched shopping. However, many of the days we compared were specifically billed as major "sale" days and so I, at least, found the price inflation disconcerting.