Show HN: Cocktail search written in Python using scrapy and sphinx

http://cocktails.p24dev.de/

by wallunit • 14 years ago

29 21 14 years ago

cocktails.p24dev.de

chris_p• 14 years ago

I really like this! I recently started making simple cocktails, and I am certainly going to use this.

Things I'd like to see: * I prefer milliliters over oz (I didn't even know what it was before I googled it, since I live in Europe), please make this an option * Some form of autocomplete, preferably with a dropdown list (but make sure it doesn't obstruct the next textbox). There's a jquery plugin that does dropdown autocomplete well. * Pressing enter should make the next textbox active (seems more intuitive than tab in this case) * It would be nice if the ingredients in the ingredient list below a cocktail were clickable. Clicking them should add them in the list of selected ingredients and update the cocktails visible.

imjared• 14 years ago

On things to fix:

While I appreciate the use of history, I promise you I don't need to hold my position on the page for every single letter I type in the search box: http://screencast.com/t/1qFwxlz5

I ended up with a history entry for "search results for 'c'", "search results for 'co'", "search results for 'cog'", etc.

Maybe trim this down to only firing after 10s on inactivity in a text input or use onblur? I'm not too sure what the answer is but it (page history) definitely loses any usefulness when architected in this manner.

wallunit9• 14 years ago

The primary reason why I am using the history API, is that you can bookmark and send links to a specific search. However I agree that when you are actually using the history, it becomes annoying to have any search that was executed while you were typing in your history.

However when updating the history only every 10s, you might have to wait until you can copy the link to your search from the address bar. And if you aren't typing fast enough there still can be incomplete searches in your history.

Using onblur would be better. But what would you think about, updating the history as soon, as the cursor is moved? I think I like that idea pretty much.

wallunit9• 14 years ago

Thank you very much for the feedback. However the problem is, that the ingredients as imported from the websites I crawl aren't in any structured format. I didn't found a reliable way to extract the amount and the actual name of each ingredient. And without that, auto-completion and converting amounts is impossible to do. However if somebody is smarter than me, suggestions (and pull requests) are welcome. :)

tharshan096• 14 years ago

Very nice. I particularly like the crawler implementations. I have used scrapy before in a project. I could not guess from the code, but is the all the data going into the database? Could you give some overview details about how things in the backend work. Thanks

wallunit9• 14 years ago

Actually the entire backend is on github. But let me explain it for you.

There is as you have already discovered, a crawler implemented with scrapy. However I don't use the scrapy server and pipelines. Instead I have a script that lets scrapy generate JSON files with the crawled recipes and builds the sphinx index from the crawled data. There is no RDBMS. Basically sphinx is my database. :)

Well and than there is the website. Its server-side is implemented with werkzeug and its UI with jQuery.

tharshan096• 14 years ago

Oh i actually make scrapy do that to. Do you use the scrapy crawl with -o and -t json options? I did not know spinx can index json files etc. Any reason why sphinx was chosen rather than an alternative? Thanks for the info.

mapgrep• 14 years ago

Nice, but there should be

1. Some stemming/normalizing such that "créme" and "creme" (as in, say, creme de menthe) are not two distinct result sets. (To your credit, whisky/whiskey appear normalized)

2. A list of sources available apart from mouse-scrubbing the results. Any reason Esquire (considered borderline definitive by certain cocktail snobs) isn't in there?

Wishlist:

-A way to make a search for "brandy" return anything containing "cognac" or "calvados," but not vice versa. Ditto for whisky vs scotch/rye/bourbon.

-Ability to exclude certain sources from all searches

wallunit9• 14 years ago

Actually the english stemmer from libstemmer/snowball is already used. Additionally there is a wordforms list, that at the moment maps only whisky to whiskey. ;) Feel free to do a pull request, if you know other words that must be normalized. I have also just added an advanced charset_table, mapping amongst others "é" to "e". So when you search now for "créme", you get the same recipes as for "creme".

At the moment I only crawl Wikipedia, liquor.com and seriouseats.com for cocktail recipes. But thanks for mentioning Esquire. I didn't knew that website yet. Maybe I will crawl them as well in the future.

One direction mapping of words is unfortunately not possible with sphinx. However I could expand "brandy" to "brandy OR cognac OR calvados", just before I run the query. I will consider that approach.

jtchang• 14 years ago

Good stuff.

Back button behavior is pretty annoying :)

pscheufele• 14 years ago

This is great. I would really love to see this tool with a curated list of recipes. The recipes from scraping are hit or miss. Please message me if you want to chat about it, maybe I can help.

I would also have a couple suggestions for improvement. (1) include the recipe for how to make the cocktail, as opposed to simply the ingredients. This can make a big difference in the result. (2) include an AND function for the ingredients search (eg. a search for pisco AND lime, I didn't see that this exists currently).

wallunit9• 14 years ago

I'm not sure if either approach would improve the results.

Searching the introductions in additions to the list of ingredients, will rank recipes higher, when the ingredients you were searching for, occur more often in the instructions. But a recipe isn't necessarily more relevant, because of its instructions are more verbose.

I'm not sure if an AND function would add much benefit. Yes, at the moment only one of the given ingredients must be part of the cocktail, in order that it appears in the search result. But the search results are also sorted by relevance, so that cocktails that contain more of the searched ingredients are ranked higher, than cocktails that contain fewer of the searched ingredients.

I'm pretty sure that there is no way to PM other users, here at Hacker News and you don't have an email address in your profile. However if you want have a further discussion on that, feel free to write me an email. You'll find my email address in my Hacker News and Github profile.

Show HN: Cocktail search written in Python using scrapy and sphinx

Related Stories