shenedu

Quite helpful posts. Try to keep track of them using a RSS reader[1]

[1] http://rssminer.net/a#read/281377?p=1&s=newest

pedestal is quite a large project, try to offer a complete solution for building web application in Clojure, from server side to client side, to tooling.

http-kit is a very small and focused library, It just do one thing: talk in HTTP. http-kit need to work together with other libraries like Ring[1] and Compojure[2] to do the server side.

Few good things about http-kit: 1. Very fast and scalable, almost as fast as what you can get from the hardware. 2. Focus, do one thing and do it well, thing about Linux/Unix's philosophy. Personally, I prefer this way of doing software.

[1] https://github.com/ring-clojure/ring [2] https://github.com/weavejester/compojure

Thanks for the kind words.

For the server, https is not planed, since It's better be done by Nginx or something alike [1].

For the client, https is planed. We have use case for it, too. Some time is needed to work on it. It will come out in one or two month, I guess.

[1] http://http-kit.org/migration.html#https

I am afraid http-kit is much feature less than SignalR. SignalR provides client side javascript[1], and hubs. http-kit provide non of these. http-kit just provide a way to get a Channel that can be used to push data to client, The Channel can be WebSocket or HTTP (long polling/streaming). IMO, http-kit is more like a unix command line tool, tiny and focus.

Socket.IO[2] is more like SignalR compare to http-kit.

[1] jquery.signalR-1.0.1.min.js [2] http://socket.io/

Yes, it's 2.0.0, that's a promise of the API. No API breaks after 2.0.0 get released (It's already released today).

To tell the truth, I, Peter[1], and a few others take more than a month to think and discuss the API, even though there are just few functions to export. We try our best to make it better. In the end, I think, We are quite happy with what we get now: the unified API.

If a very good idea found to do the API, but the API would break, then it's version 3. But version 2 will still be maintained, and any bugs will get fixed. That's the promise.

Is this address your concern?

[1] https://github.com/ptaoussanis

Good job! Thanks for doing this. Will use it in my next project.

You may find this useful, also in python:

http://code.activestate.com/recipes/437932-pyline-a-grep-lik...

Hi, I am the author of open source rss reader Rssminer, live demo: http://rssminer.net/demo

How about your guy build on top it, I can contribute the whole source code (anyway, it's open source)

The reasons I do so:

1. Rss Reader needs a lot of hardware resources for storing/fetch feeds. A big company have much better resource than I can offer. 2. I can provide some paid support for rssminer's code.

The reasons why it may be helpful for you guys:

1. Rssminer is fully working, you can save a lot of time, faster product 2. The code is very clean. https://github.com/shenfeng/rssminer 3. The code is very fast. In order for it to be fast, I even write an event driven http clent and server: https://github.com/http-kit/http-kit

If interested, drop me an email: shenedu@gmail.com

Hi, Rssminer is a personal weekend project, it's open source [1]

I create it because I want to learn how to do one page webapp with Backbone.js, but at the last, I drop backbone, life become much easier.

When writing it, I need an async HTTP Server and Client, so I write one myself, called http-kit[2], it seems that it's more popular than Rssminer.

Rssminer is not feature complete as other ones, It has:

1. Import subscription list from Google Reader 2. Feed reading 3. Keyboard support 4. Fast (the landing page is not very fast, though) 5. Run it yourself, the readme on github has detailed procedure about how to run it locally.

Bug report or pull request are welcome. Let's build a Rss reader we like.

[1] https://github.com/shenfeng/rssminer [2] http://http-kit.org

Of course rssminer[1], since I write it. It's open source, you can contribute too: https://github.com/shenfeng/rssminer

1. http://rssminer.net/demo

NewsBlur is pretty good. I've used it for a while. I also write my own feed reader: Rssminer. It's not as full-fledged as NewsBlur, but is a much lighter alternative. It's support login with google and import google reader's subscription list.

http://rssminer.net/demo

I hope HN finds it useful too. It's also fully open-source too: https://github.com/shenfeng

Hey, http-kit can be used to from java directly: https://github.com/http-kit/http-kit/blob/master/test/java/o...

Not recommended! The API is very lowlevel.

Maybe you can try to tweak the max allowed open file to a larger value. The default is about 1024, that how you just get 1000(I guess). Jetty is quite good at concurrency, you can double check it.

Why not try Clojure? Your web dev's productivity increase by few times instantly.

Yes. something like long polling do require server capable of maintaining many connections. http-kit provide `async-response` for this kind of usage: see http://http-kit.org for more detail.

I am the author of http-kit, also one of the authors of delicious China: http://meiweisq.com, trying to do some useful thing for our users (like realtime) with it. :)

Yes, JVM is strong. Clojure inherit it.

http-kit needs a few kilobytes of memory (buffer for parsing HTTP request, maintain state, etc) for a connection.

The thread model used by http-kit: a dedicated thread(server-loop), only doing events IO and parsing, when done, queued the request for a thread poll to take it, thread pool compute the response, queue it for the server-loop thread to write it back to client.

Since epoll and kqueue's readiness selection is O(1), idle connections does not hurt latency at all.

author here: Java part: 1. Java NIO's performance is amazing: event driven r/w bytes 2. maintain state machine when parsing HTTP from bytes buffer need many local variables, I get used to do it in a C style code.

Clojure: I like this language. It's brilliant. So I write a fast http server/client for it. Clojure is also write in JAVA, great interoperation

Linux's epoll or FreeBSD's Kqueue is ridiculous scalable. Both http-kit and netty take advantage of it. http-kit is all about HTTP, netty is a general framework.

Author here. The server is open source, on github: https://github.com/http-kit/http-kit Test test code is also on github: https://github.com/http-kit/scale-clojure-web-app

Suggestions very welcome!

yes, SQL is blocking, JVM also have threads. The http-kit is asynchronous(epoll on linux), the API for programmers is sync, which is more familiar.

Author here.

> Add an endpoint with a poorly optimized SQL query: 20 concurrent connections.

Agreed. http-kit use threadpool to compute the response, try to minimize this a little.

author here. Yes, Clojure can use more than 1.4G of heap, and can use threads. http-kit is multithreaded (one just for IO, others for computing response). A strength of Clojure?

author here.

> Notice that in his tests 97% of these connections don't do anything, just idle. He maxes out at 18764 req/sec.

Yes, just testing how many concurrent connection can be held. When the 600k are held, ab confirms that it can do about 31405.53 per seconds, the http body is 1024bytes.

> Notice that they are "keep-alived", coming from the same IP, so not truly separate connections

Not from the same ip, from many ips: 192.168.1.200~230

> Keep in mind that 600K concurrent connections cannot possibly do anything useful at the same time for many reasons (CPU, bandwidth, server I/O), so they are not truly concurren

They send a request every 5s~30s to server, and wait for response

Hey, http-kit's author here. willing to answer any questions.