The design document is a good read and includes high-level details of how they're grabbing packets (AF_PACKET), the packet index format (leveldb), and defensive action they took (fuzz testing via AFL, setcap, setcomp):
https://github.com/google/stenographer/blob/master/DESIGN.md
I founded 4chan eleven and a half years ago at the age of 15, and after more than a decade of service, I've decided it's time for me to move on.
4chan has faced numerous challenges over the years, including how to continuously satisfy a community of millions, and ensure the site has the human, technical, and financial resources to continue operating. But the biggest hurdle it's had to overcome is myself. As 4chan's sole administrator, decision maker, and keeper of most of its institutional knowledge, I've come to represent an uncomfortably large single point of failure.
I've spent the past two years working behind the scenes to address these challenges, and to provide 4chan with the foundation it needs to survive me by bolstering its finances, strengthening its infrastructure, and expanding and empowering its team of volunteers. And for the most part, I've succeeded. The site isn't in danger of going under financially any time soon, and it's as fast and stable as ever thanks to continued development and recent server upgrades. Team 4chan is also at its largest, and while I've still been calling the shots, I've delegated many of my responsibilities to a handful of trusted volunteers, most of whom have served the site for years.
That foundation will now be put to the ultimate test, as today I'm retiring as 4chan's administrator. From a user's perspective, nothing should change. A few senior volunteers—including 4chan's lead developer, managing moderator, and server administrator—have stepped up to ensure a smooth transition over the coming weeks.
I'll need time away to decompress and reflect, but I look forward to one day returning to 4chan as its Admin Emeritus or just another Anonymous, and also writing more about my experience running 4chan on my personal blog. The journey has been marked by highs and lows, surprises and disappointments, but ultimately immense satisfaction. I'm humbled to have had the privilege of both founding and presiding over what is easily one of the greatest communities to ever grace the Web. It was truly an honor to serve as 4chan's founding administrator, and I look forward to seeing what the next decade holds for the site.
On to the next chapter,
–moot
mootnote: I plan to dedicate this Friday afternoon (ET) to host a livestreamed Q&A session, where I'll hold court with the community one last time. Be sure to subscribe to our YouTube page, and keep an eye out for a global message the day of. As always, I welcome and read all of my e-mails, and you may contact me personally at moot@4chan.org.
Eleven and a half years, in numbers:
42,176,061,890 Total pageviews 1,771,091,423 Total posts 1,071,189,182 Total visitors 620,125,147 Monthly pageviews 21,128,887 Daily pageviews 20,360,487 Monthly visitors 1,223,807 Daily visitors 2,838 Terabytes per month 105 Volunteers 63 Boards 1 Administrator
This looks great, thanks :)
Is there an online archive that I can view before signing up?
Thanks for the shoutout! Twitpic-Backup is pretty old but will definitely get your images. Props to pcgMongo for today's PR:
I have a mirror of this site. wooledge.org runs off of greycat's (#bash on freenode) home DSL connection:
Have you tried a recent version? This commit should have made things much better:
https://github.com/linode/longview/commit/dc48b6ddce04dc7155...
Do you remember any? I'd love to hear some!
With regards to the real-time analysis, you might find perl's Regexp::Debugger fun to use:
http://search.cpan.org/~dconway/Regexp-Debugger-0.001011/lib...
That's the documentation for using the module in your code. If you're interested in the standalone tool, you'll want rxrx:
http://search.cpan.org/~dconway/Regexp-Debugger-0.001011/bin...
Off-topic, but you searched AltaVista? Did you choose to use it specifically or is it your daily driver?
Rob Pike actually wrote something similar in his short, self-proclaimed polemic, "Systems Software Research is Irrelevant":
http://herpolhode.com/rob/utah2000.pdf
"Only one GUI has ever been seriously tried, and its best ideas date from the 1970s. Surely there are other possibilities. (Linux’s interface isn’t even as good as Windows!) There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts. The future is distributed computation, but the language community has done very little to address that possibility."
"The world has decided how it wants computers to be. The systems software research community influenced that decision somewhat, but very little, and now it is shut out of the discussion..."
And particularly relevant to the HN community at large: "Be courageous. Try different things; experiment. Try to give a cool demo."
I'm copying this from a previous comment of mine because I always end up writing about the great features of Taskwarrior:
Seconding taskwarrior for being a great todo handler. It has a minimalistic interface, but doesn't lack any features that you (might) find yourself wanting after a while of use. The IRC channel (#taskwarrior on Freenode) is active too. The command-line tools are perfect and let you make your system as minimal or as complex as you want. Over time, I've:
1) Set up a cron job that pumps my current list out to a text file behind an .htaccess'd directory. This way, I can see my list without needing SSH access.
2) I've also got a little Dashboard widget that pulls that text down, so I can swipe to the top-left hot corner and see them at a glance.
3) I'm using Alfred (http://www.alfredapp.com/) on my Mac, so I wrote a simple trigger called "task add" that connects to my box and adds it there. There's also a few posts out there for DropBox integration if you use that.
4) I didn't write this, but if you use oh-my-zsh, there's a plugin for Taskwarrior. I've learned about a couple options by pressing tab.
Seconding taskwarrior for being a great todo handler. It has a minimalistic interface, but doesn't lack any features that you (might) find yourself wanting after a while of use. The IRC channel (#taskwarrior on Freenode) is active too. The command-line tools are perfect and let you make your system as minimal or as complex as you want. Over time, I've:
1) Set up a cron job that pumps my current list out to a text file behind an .htaccess'd directory. This way, I can see my list without needing SSH access.
2) I've also got a little Dashboard widget that pulls that text down, so I can swipe to the top-left hot corner and see them at a glance.
3) I'm using Alfred (http://www.alfredapp.com/) on my Mac, so I wrote a simple trigger called "task add" that connects to my box and adds it there. There's also a few posts out there for DropBox integration if you use that.
4) I didn't write this, but if you use oh-my-zsh, there's a plugin for Taskwarrior. I've learned about a couple options by pressing tab.
Of course, there's always Hacker News Daily, an automatically generated list (RSS available) of the top HN stories:
http://www.daemonology.net/hn-daily/
I particularly like the one aspect of non-repeating entries,
"The 10 highest-rated articles on Hacker News on September 29, 2011 which have not appeared on any previous Hacker News Daily are:"
On an unrelated and related note, while I appreciate the effort that goes into a hand-picked newsletter, I also accept that I might end up getting sucked into a cool example of technology. The solution isn't a newsletter or even a condensed RSS feed -- the fix is self-control (which can come in different forms and is typically a mix of motivation, guilt, and software).
This warning is much clearer (and useful). I completely agree -- disk-heavy moves can be painful.
On that note, if you're using TextMate with MacFusion/SSHFS, you'll want to get the ReMate plugin. It disables refresh on regaining focus for TextMate: http://ciaranwal.sh/remate/. No more beachball when you bounce over to the browser for a second :)
What was the problem?
I use MacFuse on Snow Leopard daily at work and home and the worst I've ever seen was TextMate move slow when I woke from sleep (and the drive wasn't mounted)
Prepaying for a year saves you 10%, prepaying for 2 years saves 15. If you already have an account, just open a ticket and you'll have the discount in minutes.
The actual task information is stored in your home directory. I use it on a Linode and stay connected all day (cloud enough for me :) ) I've seen other people use it with Dropbox, which would fulfill the syncing requirement.
There's a guy on the forums that developed a RoR frontend for it too: https://github.com/theunraveler/taskwarrior-web
I use Task on my Linode too. I have the output of "task list" cronned to a text file under a virtual host. The file uses auth-based htaccess rules, and I have the unique user/pass saved in the URL bookmarked on my machine.
I usually have a terminal window open, but it's nice to be able to click the bookmark bar and have it right there. I also have a widget that pulls in text files, so I can see my tasks at a glance by moving the cursor to the top-left of my screen.
Taskwarrior is great. Like david_shaw said, if you're looking to finally start doing your to-do list, use Taskwarrior.