A fast and static web server for serving web apps

http://shortfin.io/

by timothy89 • 13 years ago

77 130 13 years ago

shortfin.io

tunnuz• 13 years ago

You're from Lund! I've been studying in Lund!

By the way, can you show some statistics and performance metrics with respect to nginx, Apache and such?

To put it another way: why did you choose to write your own webserver instead of using, say, nginx?

davidcollantes• 13 years ago

I am curious too, on the performance metrics. He is still using Nginx though.

timothy8910• 13 years ago

Nginx is just acting as a reverse proxy because I host other sites on the same server. Benchmarks will come!

timothy8910• 13 years ago

Lund is awesome! ;)

I will perform some benchmarks and compare it to nginx (and maybe apache) in a while.

By the time (2010, 2011) I was working at a company that needed a tracking server for tracking clicks and views in email. So I started to mod lighttpd and thought that I could make it faster. So in true "challenge accepted" spirit I started to write my own HTTP server. And since then I've made it better and better. It's not meant to be an nginx killer. It will always be very basic, fast and lightweight. It's great for projects where you need a web server included.

As I wrote in another comment I have used the code base myself in various projects such as:

* Tracking clicks, view, etc. for AlphaMail * The send-API server for AlphaMail * A DMARC report server (https://github.com/amail/comfirm-dmarc-report-server) * A REST server with redis as storage

I've never used it as a static web server in production yet. Mostly because of the lack of a reverse proxy and other great features that nginx has and shortfin not.

sciurus• 13 years ago

There's a sample configuration at https://raw.github.com/timothyej/Shortfin/master/config/shor...

babuskov4• 13 years ago

I don't get it. You create a server to solve C10K problem, and then host its own website with nginx? Why?

Having HN crowd coming and testing your server would be a great test, no?

clone10182• 13 years ago

The actual site is down for me too, http://shortfin.io:88/

SirSkidmore• 13 years ago

The link works fine for me.

kajecounterhack• 13 years ago

I think in this case nginx is being used as a reverse proxy.

untothebreach• 13 years ago

From the footer:

It was created in 2011 by Timothy E. Johansson and is the base for the send-API and the tracking server for AlphaMail.
It also powers this website (though behind nginx). You can reach it directly on port 88: shortfin.io:88

babuskov4• 13 years ago

I read that. But why?

What's the point in using this if you're going to have nginx in front?

timothy8910• 13 years ago

I host more websites on the same server and as kajecounterhack said I'm using nginx as a reverse proxy. The server is fast and have no problems at all with the HM crowd ;)

babuskov4• 13 years ago

What are advantages over nginx? Memory footprint, higher through-output?

I mean, what's the main "selling point". Why should one use this instead of battle tested nginx?

Don't get me wrong, I'm not criticizing, I'm just curious.

acqq2• 13 years ago

But why do you need reverse proxy at all if your server is as good as Nginx? And if the speed is comparable and Nginx has more features why did you write your server then?

dictum• 13 years ago

wget http://shortfin.io/install.sh && sh install.sh

Isn't it recommended that any project doing one-line installs use HTTPS?

chc8• 13 years ago

In the same way that it's recommended for any other program, yes.

bru• 13 years ago

Side-note for the author: the download can be made more elegant that way

    $ curl http://shortfin.io/install.sh | sh

I definitely agree with the HTTPS. For example this is what meteor proposes:

    $ curl https://install.meteor.com | sh

pnathan• 13 years ago

asdfsdfgkjlsdfg!

Streaming your install script directly into computer execution without even giving it a once-over, md5 compare, etc is atrociously insecure.

sehrope2• 13 years ago

... and especially not if you're suggesting run it as root. You should never run anything as root unless you know what it is. Having it over HTTP vs HTTPS makes things really bad though.

jfb• 13 years ago

  Just run the following command as root to install the server.

  # wget http://shortfin.io/install.sh && sh install.sh

What? Are they serious? Is this becoming a thing? Please don't tell me that this is a thing.

chrissnell• 13 years ago

This has been a thing since the mid-90s when you could execute 'telnet sci.dixie.edu 1 | sh' to install an IRC client. It preys on the clueless and one of these days, some project that uses this install method is going to get owned and a bunch of machines will get malware.

Then again, most of us blindly run what we assume to be GNU autoconf scripts and never read Makefiles before we 'make install' as root. Nor do most of us read or audit source code, even for security-sensitive applications.

ripter• 13 years ago

a lot of times it's curl instead of wget.

chc8• 13 years ago

I don't see any rational basis for this. If you weren't vetting the code you downloaded before, you don't lose anything by blindly running the commands on one line instead of three. And if you do want to vet the code, you know what to do. Practically no installation instructions include a "vet the code" step, so you aren't losing anything here.

The lack of TLS is worrisome, but that's not so much "becoming a thing" as it is "a bad thing we're just finally starting to get rid of."

rsync2• 13 years ago

Although this isn't new, it should make you throw up in your mouth a little bit.

Malicious compromise is not the only way that an install script can become broken, or corrupted, on a remote site. The ability to checksum what you're running is a very useful sanity check and you should be doing it with all downloaded scripts/packages.

I suppose this is tilting at windmills, though.

eksith7• 13 years ago

"If you weren't vetting the code you downloaded before, you don't lose anything by blindly running the commands on one line instead of three."

Nginx is included in the OpenBSD base system (as of 5.2) and it does get vetted.

tlrobinson4• 13 years ago

Yes, it's a thing, and aside from the lack of HTTPS it's not much worse than your standard "./configure", "make", "make install" install flow of the last 20-whatever years of open source. If you don't trust the developer to serve you a safe install.sh script you shouldn't trust the rest of their code either. If you don't trust them you should audit the code no matter what the install process looks like.

That said, if you're going to do this, you should do the following:

1. HTTPS only. No excuses.

2. No URL shorteners, which make or may not use HTTPS, and introduce another point of failure.

3. Ideally serve directly from Github repo. Eliminate another point of failure. The user can be reasonably confident that the install script is the same one that's in the Github repo.

ebbv3• 13 years ago

It's a lot worse than the standard configure, make and make install.

For one, it's only requiring one simple script to get compromised and replaced to compromise your entire machine. At least when I download the tarball, unpack it, etc. I've got several steps before I even get to the make install that I would run as root. And I might not even do that depending on my install target. If I do, I have ample opportunity to verify the package in question and examine the script I'll be running via sudo before I do so.

With the instructions provided by this site, not only is there no opportunity to verify the script I've received is the one I wanted to (HTTPS alone is not enough for this, the site or domain may have been compromised) but they're instructing me to just run it immediately as root without any examination.

This is a terrible practice, and while yes it has been around for a long time in the past it was something only done and encouraged by idiots, it seems to be gaining popularity.

This is a bad idea, stop doing this people.

Provide an install tarball and provide a checksum of it that we can verify.

rglullis• 13 years ago

It's not "curl | install.sh" vs "configure/make", you know? What about using any package management system?

babuskov4• 13 years ago

It's not the same as configure,make,make install as you can always: configure --prefix=$HOME/whatever

and later run it on some unprivileged TCP port over 1024.

vacri• 13 years ago

The install.sh goes off to github and pulls the latest - it's not a method for keeping the same version across machines. It may be fine for your needs, but it'd still be nice to have 'stable' or suggested versions to use.

toupeira• 13 years ago

This has indeed become so established that chocolatey.org can offer the following command with a straight face and call it "Easy Install!":

  @powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin

ahknight• 13 years ago

okay ... why? what's special about this? features? benchmarks? anything?

grey-area• 13 years ago

Why not?

The author claims it's 'high-performance and open-source' - sounds good to me. It's not as if we have a hundred web servers all competing as we do static blogging systems, the more web servers the better in my opinion, just as with web browsers.

The zip file is 312KB as linked from here, so it's pretty compact, possibly suitable for embedding in apps, etc:

https://github.com/timothyej/Shortfin

tuananh• 13 years ago

author should add a benchmark, comparison. or maybe it's just a fun, personal project and not ready for production.

reidrac2• 13 years ago

I was going to say that there are *~ files in the repository, they should tidy a little bit because things like that don't look good.

Then I thought may be I was being a little bit to picky and unfair, so I checked the code of some random files and... well, it looks like it was part of a learning experience but it definitely needs work before being considered a viable option.

ds93• 13 years ago

What's a "static web server"? I haven't seen this term.

A webserver for only static pages, i.e. it won't talk to a runtime? But it's "for serving web apps", this means all the dynamic content is pulled from client side JS? Then you would still need programs on the server to answer AJAX requests.

I did not find an explanation on the page or by Googling.

sehrope2• 13 years ago

A server that only serves static files. It doesn't enable the use of dynamic content like CGI[1]. Compare this to something like Apache or nginx. Both can serve static files but also support a number of ways of either directly running scripts (ex: mod_perl or mod_php in Apache) or proxying to other servers that handle dynamic content.

[1] https://en.wikipedia.org/wiki/Common_Gateway_Interface

ds93• 13 years ago

OK, then, sorry if I'm being dense, but how can it be "for serving web apps"? App seems an odd term for a site consisting of only static files.

rsync2• 13 years ago

thttpd is probably the most well known "simple" webserver, and is very, very fast[1].

In fact, I think the op needs to explicitly mention thttpd on the page for shortfin and provide a performance comparison, since that's all anyone really cares about ... "is it faster than thttpd ?"

[1] http://acme.com/software/thttpd/benchmarks.html

erikig• 13 years ago

The citation should include a caveat that the benchmarks are from 1998 and that they were ran on a Pentium 100Mhz.

WhitneyLand2• 13 years ago

One big thing with this is to develop a web app consisting solely of static files.

The trick is that most of the dynamic stuff you get from REST services on OTHER servers.

For example, people maintain static blogs with Jekyll and can enable commenting with an HTML widget.

Or your app can use MongoLab for it's back end data which again is a separate server.

It turns out you can do almost anything just developing with your own static files.

This is one of the reasons hosting prototype web apps on DropBox is popular.

ksec• 13 years ago

Waiting For Benchmarks.

nine_k2• 13 years ago

I think the landing page would greatly benefit from two short paragraphs:

Why Shortfin when we have Nginx/Lighttpd?

Why Shortfin when we have thttpd?

timothy8910• 13 years ago

1. Shortfin is not a replacement for nginx. I use it myself to create other projects, e.g. fast API servers with one purpose only and a REST server that uses redis as storage.

2. See #1.

3. I don't try to sell anything, it's just an open-source project that I think maybe someone could benefit from. I've learned a lot while coding it.

nine_k2• 13 years ago

OK, Shortfin is just trying to be an efficient HTTP server, without a radical differentiating feature or implementation approach.

Well, let a hundred flowers blossom.

eksith7• 13 years ago

While I appreciate the time and effort that went into this, I'm more concerned with security than just speed alone. It's still relatively easy to throw hardware at the load problem (up to a point, naturally), but safety shouldn't take a back seat to performance.

The wget shell script installation does make me nervous and I'm glad the source is available separately. Blind installation of scripts was never "a thing" with me.

Of course, I'm still glad people are writing proper web servers (as opposed to simple 2-10 line ones). That creates an opportunity to explore the field with fresh ideas.

ChuckMcM4• 13 years ago

Close.

Can add this to the list like thttpd.

Something I keep thinking I'll build is a fast, high connection count, limited HTTP server, something that is essentially a wrapper around a program that works like 'regular' and emits HTML. It is a corner case in a custom corner, but the target it something which is essentially a 'transponder.'

In the 'Internet of Things' I want to build a wrapper/environment such that my program can be

  main(int argc, char *argv[]) {
    uint16_t sense;
    uint16_t chan;
   
    sense = adc_read(atoi(argv[1]));
    printf("Sensor %d reads : %d\n", chan, sense);
  }

And then link my 'wrap around web server on it' and then it can be accessed with wget 'http://ip:port/?1' have it do the right thing.

The keys are low memory footprint, lots of connections, easily wrapped around a 'regular' program.

tlrobinson4• 13 years ago

"easily wrapped around a 'regular' program"

Sounds like you want CGI :)

ChuckMcM4• 13 years ago

Yes, "static CGI" if such a thing was a thing :-). I've got a thttpd tree that I do this sort of hack in (these are great for monitoring large numbers of servers for example).

ams6110• 13 years ago

Sounds like you want AOLServer.

http://www.aolserver.com/docs/devel/c/c-overview.html

eksith7• 13 years ago

I had an idea like this when I thought about creating a small server on an Arduino that I can use to control/get status updates for a pellet stove for my cabin (TBA).

Luckily, there's already code for that, in case you're interested : http://arduino.cc/en/Tutorial/WebServer

kingatomic• 13 years ago

I've used Snorkel[1] for a few C/C++ projects in the past, you may want to check it out; it sounds like it could be what you're looking for (or quite close). Link it with your code, pass it a few config parameters and let it serve static files -- or you can overload GET/POST functions to provide dynamic functionality.

1: http://snorkelembedded.webs.com/

ChuckMcM4• 13 years ago

That looks interesting, if somewhat abandoned.

BlackDeath32• 13 years ago

Very cool. I've been doing the same thing lately, writing my own little web server that serves static files. It's been a fun learning experience!

timothy8910• 13 years ago

Nice, in what language? I've really learned a lot from creating shortin! Shortfin is written in C so it's a nice project to spend some time on when I get tired of writing javascript and html ;)

BlackDeath32• 13 years ago

I'm writing mine in C as well. I've never been too into web programming, but the lower-level stuff is quite interesting to me.

Keep it up!

joeblau• 13 years ago

I just ran two speed tests since there aren't any benchmarks provided

1. First Test is using http://www.webpagetest.org/

Results shortfin.io:88 - http://www.webpagetest.org/result/130806_71_13HK/

  First View: 1.945s
  Repeat View: 1.632s

Results shortfin.io - http://www.webpagetest.org/result/130806_QV_13F7/

  First View	2.099s	
  Repeat View	0.084s

2. Second Test is using Apache Bench

Results (Best results of 3 runs): ab -n 100 -c 100 http://shortfin.io:88/

  Time taken for tests:   12.125 seconds
  Requests per second:    8.25 [#/sec] (mean)
  Time per request:       12124.501 [ms] (mean)
  Time per request:       121.245 [ms] (mean, across all concurrent requests)
  Transfer rate:          38.46 [Kbytes/sec] received

  Connection Times (ms)
              min  mean[+/-sd] median   max
  Connect:      210 1114 1517.2    360    4041
  Processing:  5017 8407 2034.0   8715   11762
  Waiting:      169 1084 3054.8    188   11408
  Total:       9057 9521 1068.7   9058   12123

Results (Best results of 3 runs): ab -n 100 -c 100 http://shortfin.io/

  Time taken for tests:   5.790 seconds
  Requests per second:    17.27 [#/sec] (mean)
  Time per request:       5789.949 [ms] (mean)
  Time per request:       57.899 [ms] (mean, across all concurrent requests)
  Transfer rate:          82.97 [Kbytes/sec] received

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:      171  249  35.7    263     290
  Processing:   172 1798 2345.5    212    5524
  Waiting:      170 1055 1825.1    212    4949
  Total:        344 2047 2368.2    469    5789

rarestblog• 13 years ago

You need some better hardware

$ ab -n 100 -c 100 http://shortfin.io/

Requests per second: 1363.85 [#/sec] (mean)

rorrr2• 13 years ago

Benchmarking a fast remote static server is pointless.

You're basically benchmarking your internet connection.

riledhel• 13 years ago

I can't find anything related to the response headers the server can send to the client, and the OP site just serves responses with content-length, content-type and server. Anyone?

c-a• 13 years ago

The request parsing code seems to be of dubious quality https://github.com/timothyej/Shortfin/blob/master/src/reques...

From a cursory glance:

1. https://github.com/timothyej/Shortfin/blob/master/src/reques...

should be (data_len - i >= 4) since it accesses data[i+3]

2. https://github.com/timothyej/Shortfin/blob/master/src/reques...

shouldn't headers[header_count]->key also be null terminated?

3. https://github.com/timothyej/Shortfin/blob/master/src/reques...

header_count can become greater than 49 which causes a heap overflow at https://github.com/timothyej/Shortfin/blob/master/src/reques... and https://github.com/timothyej/Shortfin/blob/master/src/reques...

timothy8910• 13 years ago

I'll look them up, thanks for noticing!

andrewcooke• 13 years ago

the code in general is not that great, imho.

as well as the header count (which i came here to post), there's another suspicious hard-coded size limit in the number of servers (1000). although you could only crash the system in that case by configuring too many.

there's very little error handling. good c code returns error codes all over the damn place. this hardly has any.

i wouldn't use this.

el-mapache• 13 years ago

My favorite part is this, under the config section: # vim /etc/shortfin/shortfin.conf # /etc/init.d/shortfind restart

No explanation of what the configs are, just that "its important to configure it right."

Why not put this information on the site?

zzzcpan• 13 years ago

I find it very hard to trust any C code without tests. Is this a thing for C code? Why most of the new projects in C lack tests?

simon_vetter• 13 years ago

I would recommend adding ipv6 support. Should be fairly easy to do in socket.{c,h} and in your configuration parser.

halayli• 13 years ago

This code is far off from being production ready. Skimming through the code quickly, there's barely any error handling. HTTP parsing is not compliant either.

https://github.com/timothyej/Shortfin/blob/master/src/respon...

I am not sure what's the point of using it instead of nginx.

timothy8910• 13 years ago

A lot of you wanted benchmarks so here's one comparing Shortfin with Nginx. The tests was performed with a 56.1 kB PNG image with keep-alive turned off on my laptop. The best result out of 3 tests is shown below.

tl;dr:

Shortfin: 18 914 req/sec

Nginx: 15 603 req/sec

SHORTFIN

  sudo ab -n 100 -c 100 http://127.0.0.1:40/timothy-johansson.png

  Server Software:        shortfin/0.9.5
  Server Hostname:        127.0.0.1
  Server Port:            40

  Document Path:          /timothy-johansson.png
  Document Length:        56089 bytes

  Concurrency Level:      100
  Time taken for tests:   0.005 seconds
  Complete requests:      100
  Failed requests:        0
  Write errors:           0
  Total transferred:      5618000 bytes
  HTML transferred:       5608900 bytes
  Requests per second:    18914.32 [#/sec] (mean)
  Time per request:       5.287 [ms] (mean)
  Time per request:       0.053 [ms] (mean, across all concurrent requests)
  Transfer rate:          1037701.56 [Kbytes/sec] received

  Connection Times (ms)
              min  mean[+/-sd] median   max
  Connect:        1    2   0.0      1       2
  Processing:     2    2   0.1      2       2
  Waiting:        1    1   0.2      1       2
  Total:          4    4   0.1      4       4
  ERROR: The median and mean for the initial connection time are more than twice the standard
       deviation apart. These results are NOT reliable.

  Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      4
  99%      4
  100%     4 (longest request)

NGINX

  sudo ab -n 100 -c 100 http://127.0.0.1:41/timothy-johansson.png

  Server Software:        nginx/1.2.6
  Server Hostname:        127.0.0.1
  Server Port:            41

  Document Path:          /timothy-johansson.png
  Document Length:        56089 bytes

  Concurrency Level:      100
  Time taken for tests:   0.006 seconds
  Complete requests:      100
  Failed requests:        0
  Write errors:           0
  Total transferred:      5631000 bytes
  HTML transferred:       5608900 bytes
  Requests per second:    15603.06 [#/sec] (mean)
  Time per request:       6.409 [ms] (mean)
  Time per request:       0.064 [ms] (mean, across all concurrent requests)
  Transfer rate:          858015.83 [Kbytes/sec] received

  Connection Times (ms)
              min  mean[+/-sd] median   max
  Connect:        1    2   0.7      2       3
  Processing:     1    2   0.5      1       3
  Waiting:        0    1   0.7      1       3
  Total:          2    4   0.9      4       5
  WARNING: The median and mean for the processing time are not within a normal deviation
        These results are probably not that reliable.

  Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      5
  95%      5
  98%      5
  99%      5
  100%     5 (longest request)

corresation• 13 years ago

I don't know how valid a test of five milliseconds is to one of six milliseconds (much less one requesting a single known resource with no keep-alive). Though practically, even if those results held in a realistic test, is it still a viable alternative to nginx? You eliminate an enormous amount of flexibility (I admit -- I ♥ nginx) and proven trust for a margin-of-error theoretical speed advantage?

I understand the desire to get software out there, but webserver is a ridiculously hard nut to crack. nginx broke in through a new architectural paradigm.

timothy8910• 13 years ago

Nope, I'm not trying to replace nginx. I love nginx too ;) The "challenge" was to build a very lightweight server that was faster than nginx - which I did. Then I've used it as a base for a various of projects (see my other comments).

nickzoic• 13 years ago

Lots of people have asked "Why not Nginx? Why not thttpd?" ... but if all you want to host is static files, why not AWS S3 or similar CDN?

A fast and static web server for serving web apps

Related Stories