deepu_256

Do you mind sharing the details of that hack ??

wonderful.

Can you also shed light on thrift protocol ? i have struggled with the blocking nature of thrift(client).

Does eventlet convert blocking thrift client written in python to non-blocking ?

A quick question ---

can i say use pymongo in Brubeck and you are saying the blocking nature of pymongo won't affect the async nature of Brubeck ??

I guess what he means is that for drivers like mysql which are blocking only there is little support in tornado to not freeze the tornado instance during the request response cycle.

So they are ok with making their tweets and posts on tumblr being exposed to public but they are not ok with their parents being on facebook ??

This is from my experience of trying to build a news feed (we are about to launch one for our site which gets about a million unique visitors a month).

The approach you suggested is the first approach that i tried. And frankly it worked great when i am testing the feature on my dev macbook with just a couple of 1000's of users. But when i started testing it on our server loading entire legacy data redis ended up taking more than 6GB of RAM for abt half a million users.

i told myself to just plan for 2 times our current traffic and then later on think of a better solution(best thing wud be to plan for 5-10 times your current traffic) and told my CEO that we might have to bump up RAM on the redis machine to 32 or 64 GB in the future(the easiest way but not an elegant way to solve the problem is throw more resources). Also mind that you need atleast 2 such machines to provide failover. i just hoped salvatore(a great guy BTW) would release a cluster solution for redis and will save me.

While testing the feed we came to know we needed to add more such actions from the user and every time you add a new type of action for every user you are looking at increasing the memory of Redis process by substancial amount. you will run out of memory faster than you have planned. i thought VM is the solution to all of this as user feeds typically use only latest data and the entire feed doesn't need to be in the memory. But quite frankly VM has it's own set of problems -- http://groups.google.com/group/redis-db/browse_thread/thread....

Not telling that your solution isn't good. Just wanted to warn you about some potential problems from experience of trying out your solution. Finally i am using cassandra for news feed. Working great as of now. At least much better than my experience with implementing the same with redis. The major problem with cassandra as of now is implementing pagination and counting. distributed counters are coming in 0.8 . pagination is still a major problem though. But for news feed you might not need pagination. providing a load more button as facebook and twitter does should be good enough.

Hope it helps.

This looks really useful.

In the coming weeks i will try and use it in one of the apps i am building on tornado.

Kudos to Greplin.

Just one question. I see Tornado related repos in Greplin Github account. Are you using it in one of your Tornado apps ? In that case any gotchas regarding blocking ?

How ??

For the entire site or only some parts ?

just curious.

Hi nechmads, Are you willing to review the business plan/provide letters of recommendation if you really like the application ??

saw that Letters of Recommendation can be from StartUp Chile Alumni. Hence asking.

Well, may not exactly big but Quora is using Tornado in production.

Wow. Hearing about this guy for the first time. Inspiring ....

i played with gizzard for one of our services with redis as backend. twitter engineers on #twinfra IRC channel helped me a lot on understanding gizzard and flockdb src code. Not yet using in production though.

You have any questions ?

They make code concise.List Comprehensions, magic methods - Yes . implementing __len__ on your class and using len() everywhere(with list, string and your new class) is better than coding a method called get_count() , total_count(), etc etc.

They help make code more declarative rather than filling it with implementation logic. Anyone can just read the code and understand it instead of getting bogged down with the implementation. Especially list comprehensions. [len(i) for i in k] is much better than reading a for loop implementation looping through k and append len of each object in the loop to a result array

Write tests, run code-coverage and code-smells. While good advice, that's just saying what to do. Not how to do. You can as well remove the word Python from the title and the advice holds true in general.

What i would like to see :- what in python we can use that will help us write quality, concise code. Things like using list comprehensions, using magic methods, higher order functions(decorators...) and may be something else that i didn't knew.

Cannibalization is not the norm. There are many successful platforms which didn't go the route of cannibalization. Ex - windows.

The OS 9 - OS 10 example is more of an exception than the norm. In the case of OS 9 everyone(even Apple people) knew it sucked big time. They tried hard for several years to fix it's problems or try to develop a new one. There was no way for them to continue with OS 9.

RIM isn't exactly is in such a bad situation. Agreed that they need to woo developers much much more to their platform but that doesn't warrant cannibalization. They should continue innovating but should be more evolutionary.

The bad thing is that for a long time RIM sat on it's laurels without much innovation. Good thing is that they still haven't lost the market share. They already moved from their strong place of qwerty keyboards to touch screens. Much more innovation is needed. Not just copying others but having a vision of the future and then aggressively executing on that vision. They did it once before which made Blackberry such a cracking brand. Now they should do it again.

I guess he is pointing out that - Although Andriod is gaining market share, it isn't exactly eating into potential RIM market share. Other way to put it is that both RIM and Android target different set of customers.

P.S :- I am not making the above assertion. Just writing what i understood from reading that article.

It is not about owing anything to the people of India. It is about owing himself the pleasure and satisfaction of making his homeland - the place where he was born and the place where he grew up - a better place.

i wudn't mind. more people attending the meet, the merrier it is.

Get in touch with ankeshk. he is organizing the meet.

From the looks on Hackerstreet.in , Bangalore has more HN users than mumbai. i guess someone has to go ahead and take the initiative.

Ankesh even sent us mumbai HN users an FB message. Nice :)

The part about Cassandra and replication is wrong. One can configure replication factor to be 3 and still make writes with consistency level one - which means the other 2 writes will happen async and won't block the client.

The part about write ahead logger is also useless and cassandra also does that. it's called commit log in cassandra.

It gave me the impression that the author really doesn't understand cassandra.

As for the decision of facebook to not use cassandra - i guess it has more to do with maintaining data hotspots or Map reduce like computation on the data than replication and scaling factors. Even though cassandra has key cache and map reduce from version 0.6 they are still not very mature.

Buy and iPad. use ipad strictly for consuming content - reading HN, facebook, twitter , Instapaper etc.

use your computer strictly to create stuff - code, write, design, photoshop etc etc. If you want you can install some software to block sites and apps on your computer.

This will reduce the distractions enormously. If you find yourself on ipad all the time you will feel guilty that you didnt create anything today and thus will be motivated to do something abt it.

amix, i am sure with the suggestions that Zed made and from your own experience of using that in production you would have made modifications to the code.

     is there any way you can post the code that you last used in production?

Yup. i learnt a lot by seeing Zed's code.

But it was very nice indeed that he takes time to review code voluntarily on blog comments.

I haven't tested a lot of queues but one solution that i played with recently and liked is zookeeper.

It is distributed, battle tested and has a small simple API. You can build some good distributed data structures(like queues for example) on top of it with minimal work.

For python examples - http://www.cloudera.com/blog/2009/05/building-a-distributed-...

and http://github.com/twitter/twitcher

@lethain - you beauty. Nice work. The thing is - it is not surprising that redis maps well with the python DS. It is that after reading this it occurred to me how powerful having Persistent DS(Not in the Functional DS sense but in data storage sense) can simplify web programming.