HN user

malcolmhere

127 karma
Posts3
Comments17
View on HN

I've written similar code for investment banks, to extract financial reporting data from PDFs. It's shocking to think how much of the financial world runs on this kind of tin-cans-on-a-piece-of-string solution.

This is a recurring problem in the animal feed industry. Easiest explanation for the phenomena: the small particles fall through the gaps gaps between the large particles.

This is such garbage. If you clickthrough on this guy's "research", you'll find that he objects to Google's "Go Vote Today" reminder is unfair politically because:

- It increases voter turnout across the board (which favors Democrats marginally) - Google users tend to be younger (and hence more left-leaning)

Google has many questionable practices, but simply reminding people when it is an election day is not one of them

I worked on a product like this for a long while, so I can appreciate how hard the problem is you are trying to solve. Some things I realized along the way:

* Price capture needs to happen in a headless browser (e.g. PhantomJS), rather than just capturing the HTML with a GET. Too many sites use JavaScript to make raw HTML analysis feasible.

* You can get > 50% of the pricing information with fairly simple matching on the class/id value in the HTML tag. But you need a headless browser to make sure the tag is visible. And since most product pages contain multiple prices, you need some heuristic to determine the relevant price. Oh, and watch of out for "reduced from" prices too (e.g. "Old Price: $50, New Price: $35".

* It doesn't hurt to be able to override the general heuristic on a domain-by-domain basis, saved me a lot of headaches.

* You need to be honest with yourself about how reliable the price capture algorithm is, and built up a regression database of known good pages, so when you change the algorithm, nothing else breaks. Also, you need to keep ahead of site redesigns!

* Product URLs tend to look messy, but tend not to change very often, if at all. I was worried about retailers e.g. changing product identifiers, but changing URLs hurts their SEO, so they don't do it. You will find "zombie" products, though - things which appear to be still on sale, but aren't linked anywhere on the site. Deciding when a product is sold out is tricky.

* The best user experience presents the items the user is watching as a "shopping basket". (I took design cues from Pinterest.) For a really slick experience, you should pick out the product name and image (Facebook meta-data helps here) and include them in you "pinned" products.

* Cutting-and-pasting URLs is a hassle. Consider writing a browser extension or a bookmarklet - users don't like to have the browsing flow interrupted by having to click across tabs. Having the price capture done inline on the page really impresses people.

Best of luck with this! I'm yet to see someone solve this problem well, and I eventually moved on other things after losing a lot of my hair. :-)

- We don't have the breadth of Safelight's material (early days I guess), but the areas we cover, we do a much better job. It was our frustration with this kind of training material that inspired us to make Hacksplaining in the first place:

https://www.youtube.com/watch?v=jkQgVO993W8

Our exercises are interactive, rather than passive, and focus on specific ways to fix code, rather abstract concepts. Compare with what we have on SQL injection:

https://www.hacksplaining.com/exercises/sql-injection

We started with the question "what are the essential things we would want our development team to know?" and then figured out the most compelling way to teach about them.

- We are talking to a couple of firms that we could partner with to help establish credibility. It's a bit of Catch-22 selling this kind of training material - people buy your product on the basis of who your existing customers are, to some extent. Finding an established player to work with would really give us a leg up.

- Most companies reluctantly pay for security training, precisely because so much of it is onsite and expensive. Making security training mandatory for developers is a good policy for a CTO of a large company (particularly if they have been hacked recently), but it's generally impractical to send to send everyone out for a 5-day course. We hope engaging, online material can fill that niche.

Well spotted - I kind of mangled that explanation. The risk being mitigated is if somebody gets a dump of your old emails. Short-lived reset tokens don't help if they have full access to you email account.

Give it a try, it's free! We don't cover mobile specifically, but if you are building APIs, much of the advise is useful. And thinks like password management are useful for every developer to know. :-)

Author here! I put this together because I was bit frustrated with the quality of teaching resources for my development team. (And I find the OWASP wiki a bit of a mess.)

Not sure about the business model yet, though it's peaked some interest here and on /r/programming, so I figure there's an appetite for good training material.