madhouse

Not really a public plea when it is behind a signup wall.

Flowers for Algernon by Daniel Keyes.

Read it first when I was 16, at the recommendation of my literature teacher in high school. For the past... - darn, this was long ago! - ...more than 20 years, I've read it at least once a year, and intend to do that for as long as I can read.

That book changed me, like nothing else since.

You can limit your search to certain repositories on GitHub, with advanced search: https://github.com/search/advanced

You will need to enter the repositories by hand, though. Mind you, writing a small tool that'd list the most popular/forked/starred/etc C repos, and do a search for you in those should be a few minute task.

You know what would be amazing? Combining this with logic programming.

(run 100 [box] (fresh [x y z] (randomo x) (randomo y) (randomo z) (locationo box x y z)))

(Overly simplified and terrible DSL for it, but you get the idea!)

Is there anything like blessed + blessed-contrib for either of those? It'd be fairly trivial to write the bulk of potential-happiness in Java, if there was a display library like blessed + blessed-contrib in the JS world.

Author here. I'm running this in "production" (kinda, on laptop, looking at data from my own servers), and works for me. There are some annoying shortcomings, though, which I will fix, eventually.

Among them is error reporting and handling: there's pretty much none. If the network dies, potential-happiness will not be too happy: it will cry and crash.

Bug reports, feature requests and whatnot are of course appreciated. I wrote this mostly for my own use, so it isn't exactly friendly just yet. (That, and my JavaScript is terrible.)

Why do you need an URL shortener to begin with? (See http://joshua.schachter.org/2009/04/on-url-shorteners for some great reasons against them)

I remember http://lamsonproject.org/ being a python SMTP server library thing. It may be possible to piece together something based on it. That covers the SMTP part.

Mind you, the git repo looks dead, with the last commit being over two years old.

you're saying that yours is unconditionally better

I don't think I'm saying that. The article presents two setups and a few related use cases, where I believe binary log storage is superior.

With the services you run, you might be able to dictate that the log formats are restrictive enough that writing a parser for each one isn't a problematic overhead.

I don't need to dictate all log formats. If I can't parse one, I'll just store it as-is, with some meta-data (timestamp, origin host, and so on). My processed logs do not need to be completely uniform. As long as they have a few common keys, I can work with them.

For some apps or groups of apps, I can create special parsers, but I don't necessarily need that from day one. If I'm ok with only new logs being parsed according to the new rules (and most often, I am), I can add new rules anytime.

Parsing up-front, assuming you know what data you can safely throw away, might appear to some as a premature optimisation.

> We have well documented tools and workflows, so anyone new to the system can catch up and start working with the logs within minutes. It sounds like this is something which could be usefully open-sourced, to show how it's done.

LogStash is a reasonable starting point. Our solution has a lot of common with it, at least on the idea level.

Pre-parsed binary logs in a locked-down environment might be as flexible as freeform text, but I'd need to see a running system to properly judge.

Only our storage is binary. That is all the article is talking about. Within that binary blob, there are many traces of freeform text, mostly in the MESSAGE keys of application logs which we care less about (and thus, parse no further than basic syslog parsing). You still have the flexibility of freeform text, even if you store it in a binary storage format.

I assume this comment is related to the journal. The article is not.

But, to reply: yes, many organisations have fully functional, well-debugged logging infrastructures. A lot of them also use binary log storage, and have been for over a decade, and are more than satisfied with the solution.

Both rsyslog and syslog-ng have been able to assist with setting such a thing up for about a decade now.

Where are the rsyslogd and syslog-ng competitors to systemd's journald? Where is the interoperability? Where is the smooth, useful upgrade mechanism?

The journal has a syslog forwarded, but both rsyslog and syslog-ng can read directly from the journal. Interoperability was there from day one. Smooth upgrade mechanism took a while to iron out, but it's there now, too.

I experimented with that, and heard others toying with the idea too. There are even products out there that do something similar.

Agreed. But if I keep my logs in a database, I may aswell use the database to query my logs, instead of grepping in them.

(And voila, you have binary log storage.)

Why would I need to be able to parse everything up front? Taking the syslog example, that has a commonly understood format. As a default case, I can just split the parts and have structured data (esp. with RFC5424, where structured data is part of the protocol to begin with).

Then, I can add further parsers for the MESSAGE part whenever I feel like it, or whenever there is need. I don't need that up front.

With a binary log storage system, nothing stops you from browsing all logs that happened around the time of the incident. Instead of locating the files, you just tell the engine to show you the logs from that time onwards (or from a little bit before).

As for our logs being too verbose: nope, read the article.

Also, it's not an one-size-fits-all solution: I have no problem with people using text. All the article wants to show, is that binary logs are not evil, bad, useless, etc, and that there are actually very good reasons to use them.

For example, storing logs in a database is one kind of binary log storage: most databases don't store the data as text.

Please don't confuse log storage with log transport. We can transform the stored format into any other, if so need be.

(For example, I use Kibana at home. Works great, though I have no text logs stored.)

The machines I'm administering will all log the same way, therefore, within that context, they are universal. We have well documented tools and workflows, so anyone new to the system can catch up and start working with the logs within minutes.

We don't unexpectedly find machines that don't conform to our policies. We control the machines, we know where and how to find the logs. If we found any where we had to grep, we'd be having a very bad day.

Our lowest common denominator is not text, because we control the environment, and we can raise the bar. Being able to do that is - I believe - important for any admin.

* Why would you need a proprietary tool? * What if they get broken? I don't want to look at them raw anyway. * Text logs are easy to corrupt as well. Oh, append only? Well, you can do that with binary storage too.

And again, there is no need for proprietary tools at all. Everything I want to do is achievable with free software - so much so, that I use only such software in all my systems.

As for compressing - yeah, no. Please try compressing 100Gb of data and tell me the performance cost is nonexistent.

As for LogStash & ES: Guess what: their storage is binary.

Also note that my article explicitly said that the Journal is unfit for my use cases.

I've worked with a number of implementations, both embedded and others (ranging from a PC under my desk, through dozen-node clusters to ~hundred nodes). For most cases, binary storage triumphed. Most often, we kept text based transport.

Again, transport and storage are different. While I prefer binary storage, most of my transports are text (at least in large part, some binary wrapping may be present here and there).

My small system is usually two computers. Having a router/proxy/firewall box at home is not all that uncommon, and some examples I gave apply there nicely.

You're missing the point. I'm not using a custom logging format. I'm using binary log storage, with emphasis on the storage. There is a database and a search engine behind it.

Logging format and log storage format are two very different things.

Also, I'm not shocked people prefer text files. I'm shocked why they're so much against binary log storage. There's an important distinction between the two: you can prefer text, if that fits your case better, without hating on binary storage.

Bringing a few extra tools for forensics should not be a problem. You bring grep, strings, less, and a bunch more to read text logs. Why not bring one more to read the binary dump too?

I'm sorry, but I don't find the "but I can view text on a machine from the last century" argument convincing. We're not in the past century, and when doing forensics, we usually do that on a reasonable machine, where all the tools we need are available. Otherwise its an exercise in futility.

Hy has macros, and they work fairly well. I'm using plenty of macros in Adderall, would not be possible without them. There are a few rough edges with Hy macros, mostly due to the underlying Python, but I at least, can live with that, and they don't hinder my work.

I have about a thousand lines of Hy in production, but sadly, that code is not open source. I am, however, working on porting a few Clojure libraries to Hy, those will appear on my github profile (github.com/algernon/) in the coming months.

I hate significant whitespace, and the syntax of Python (not enough parens; allows crazy oneline hacks with comprehension, yet still forces indentation otherwise; blindly following pep8 often results in horrible mess of a code, etc).

Hy gets rid of both the significant whitespace, and the syntax too. Yet, allows me to have bidirectional interop, so people using my modules are none the wiser, as long as I make a little effort to remain python compatible (macros used only internally, and not exposed in the API, and using valid python names for functions and other stuff).

I've been using Hy for a while (in production too), and while it has its rough edges here and there, it is a million times more pleasant to use than python. But I hate python with a passion, so there's that. Bidirectional interop is fantastic, too - at one place, they're using Hy without knowing, because they just import a python module, which happens to be Hy. I find that both funny and an awesome achievement at the same time. Every time I look at that deployment, I can't help but feel geeky pride.

A real fun language, one which matured a lot within the year. Keep up the good work!

A coding style is there for a reason, and Laravel even has a documentation that tells you in advance the coding style and gives you tips on how to submit a good pull request: https://github.com/laravel/framework/blob/master/CONTRIBUTIN...

This is how contributions work. Not just open source. At any sane company, you will have a common coding style, and your patches will not pass review, or be merged, if you do not follow. Same goes for open source.

I wanted to learn a functional language, because I was frustrated with C, and none of the OO languages I could use at the time (C++, Java or Python. These were the languages used at work, so I was restricted to these) filled me with confidence. So I went looking, found Haskell and the "Learn You a Haskell (for Great good)" book, skimmed through it, loved it, bought it, studied it for months. But I never did anything with Haskell, it's a great language, but I never found a task I could do with it, to experiment and deepen my knowledge.

At the same time, I've always been fond of Lisp (due to Emacs, Sawfish and partly GIMP-Fu I believe), and one day, reading HN, I came across Clojure. A functional Lisp on the JVM, with great interop, and strong support for concurrency. Hell yes! I toyed around with it too, but shelved it soon after, until I found a problem I could solve with it. And solve I did. The sheer beauty of the language still mesmerises me. It's insanely powerful, yet concise and easy to understand.

The thing that got me hooked into Clojure was that I could express myself a lot easier and a lot better with it, than in any other language I worked with before (and I worked with a lot of languages in the past). Everything else (core.logic, core.async, overtone, riemann, pedestal and ClojureScript) is just an added bonus.

Coding in C is, in my experience, a terrible, terrible thing. There are part of it that are interesting, for a while, but every time I have to touch C, I get frustrated, because I have to run long laps to get anything meaningful done in a sane way.

I can't imagine how android app coding would be worse, but arguably, I haven't tried.

Never said it would be leading. But more distributions switching to journal-only is a good thing, because that makes it that much easier for the rest to follow suit.

While I'm not a Fedora user, I wish them good luck, and hope the change goes through. It would be of great benefit for other distributions aswell, and perhaps we could get rid of traditional syslog for good at some point... (wishful thinking, I know).