jmoiron

I played from 1995~2001/2ish. Probably to this day the most rich online gaming experience I've had; the book version of a succession of inferior films.

97th to 104th between 2nd & 3rd is basically all public housing, and there's public housing along the east river that will also have closer, less crowded stations.

A lot of the tunnels for Phase 2 (extending the Q to 125th) were built or partially built in the 1970s, and this phase already has funding commitments.

What? He also predicted how and why it would fail. Sun was a big enough player then that it could have survived plenty of other ways. Apple of today looks nothing like the company in 2000, but Sun got caught out more or less exactly as described and never adapted.

Same author as GORM so its to be expected.

A few comments on some things I'm seeing in these.. comments.

- What about Cassandra/PostgreSQL/Redis?

One of the implications of "99% of data is never read" is that it's incredibly wasteful to keep it all in memory. You might be assuming that you are letting the data expire eventually, but I'm actually not; expiry is a secondary concern to storage.

Once you start to involve the disk (PostgreSQL and Cassandra), you start to get into locality issues that these databases weren't really designed for.

For a more concrete description, lets say I have 2000 machines. Our app is Python, so they run 16 dockerized processes each, with each container reporting 10 simple system metrics every 10 seconds. These metrics are locally aggregated (like statsd) into separate mean, max, min, median, and 90pctile series. I've not instrumented my app yet and that's already 160k writes/sec on average; if our containers all thunder at us it's 1.6m, and frankly this is the "simple" case as:

* we've made some concessions on the resolution

* we only have dense series

Anyone who has used graphite at scale knows that this is a really painful scenario, but these numbers are not particularly big; anywhere you could take an order of magnitude there are a few other places you can add one.

I'm also assuming we are materializing each of these points into their own timeseries, but that's more or less a necessity. It gets back to the locality issues; If we wanted to see the "top 10 system load per node", it's actually quite imperative that we aren't wasting IO cycles on series that _aren't_ system load; we need all we've got to handle the reads.

(As a side point, this is why people in the cloud are adopting so much Go so quickly; it's proven to be easy to write AND read, and also to reduce orders of magnitude in several of the dimensions above, eg. "we can use 1 process per box not 16 containers, and we can get by with 1000 machines not 2000." Having to write your own linked list or whatever doesn't register in the calculus.)

- 1s resolution isn't dense:

No, not always. It's hard to please everyone with these things. In my world, 1s is good (but not great), but 10s seems to more or less be accepted universally, and much sparser (5m, 30m, 6h) is not actually uncommon. At the other end of the spectrum, you can be instrumenting at huge cardinality but very sparsely (think per-user or per-session), perhaps only a few points per timeseries, and the whole of what I've described above kinda gets flipped on its head and a new reality emerges. For what I've described, I quite like the Prometheus approach, but for my very specific use case 1-file-per-metric only beats the filesize block overhead often enough for very long timeframes; to long.

- Why are all TSDB over-engineered?

I hope some of the above has made explicit some of the difficulties in collecting and actually making this data readable. I've only actually thusfar discussed the problem of "associate this {timestamp,value} pair with the correct series in storage"; there are also the following problems:

* you can't query 3 months of 1s resolution data in a reasonable amount of time, so you need to do rollups, but the aggregations we want aren't all associative so you have to do a bunch of them or else you lose accuracy in a huge way (eg, if you do an avg rollup across min data, you flatten out your mins.. which you don't want); this means adding ANOTHER few dimensions to your storage system (time interval, aggregator)

* eventually you have to expire some of this junk, or move it to cold storage; this is a process, and processes require testing, vigilance, monitoring, development, etc.

* you need an actual query system that takes something useful and readable by a human ("AVG system.cpu.usage WHERE env=production AND role=db-master") and determines what series' actually fall into those categories for the time interval you're querying. Anything holistic system that _doesn't_ do this is an evolutionary dead end; eventually, something like Prometheus or Influx will replace them.

These are minimum requirements once you "solve" storage, which is always a very tricky thing to have claimed. If you get here, you've reached what decent SaaS did 4 years ago and what very expensive proprietary systems handled 10 years ago.

- What about Prometheus/InfluxDB/Kdb+/Et al.

Kdb+ is very expensive, its open source documentation is difficult, and its source is unintelligible. It is basically from a different planet that I'm from. Even recently, when I encounter people from, say, the C# world and tell them I work with Python and Go, they ignore Go and say "Wow, there are like no jobs for Python", which I find utterly bewildering. Of course, I never encounter any jobs using C#, either. This is how little some of these spheres overlap sometimes. Someone from the finance world is going to have to come in and reproduce the genius of K and Q for us mortals in a language we understand.

As for Prometheus and InfluxDB, I follow these more closely and have a better understanding of how they operate. I think that they are both doing really valuable work in this space.

From a storage aspect, I think the Prometheus approach is closer to the one that I need for my particular challenges than the InfluxDB one is, and in fact it looks a bit like things we've already had (but also Catena, Parquet, et al..) For most people, storage actually isn't important so long as it's fast enough.

And this is kind of the point of my article. There's starting to be a bit of a convergence among a few Open Source TSDB, and I've taken I've tried to highlight some issues in those approaches and suggest that there's room for improvement. I have my own ideas about what these improvements might look like based on my work at Datadog, and once they're proven (or even disproven) they'll be published somehow.

I have actually heard of Kx and Kdb+ and Q and have looked into those products. You're right that I should have mentioned them, because they are important in that space, and in many ways they are miles ahead. The state of time series query languages is quite poor (outside of Influx's effort), so there is a lot to learn from Kdb+ as well.

I was however focusing on recent Open Source efforts and on the general approaches. Hence, I didn't really discuss in detail my own tsdb.

I also find virtually everything to do with K and Kdb to be be simultaneously impressive and utterly unfathomable:

http://code.kx.com/wsvn/code/kx/kdb%2B/s.k

This is a cheap jab to make, but it makes these systems pretty impenetrable from a source level.

I have a lot of experience with Django's ORM and I would say that the essential character of these heavier ORMs is missing from the afore mentioned libraries. They can be used in the way you describe, but that's not their typical usage.

AR/Django encourage you to describe your entire schema, _including_ the relationships between tables, as attributes on your model objects. Upon doing this, you get simple and reliable programmatic access to some basic access and storage patterns.

Using this knowledge of your data model, the ORM can now provide you with more advanced tools: it can automatically join across tables (`select_related`), lazily load dependent data[1], generate SQL schema for you, transparently cache queries[2], automatically provide HTML form validators, and even automate database schema migration for you. The more completely you model the system, the more it can do for you.

In these systems, the database is subservient to the model. This is a problem, because the database is reality and the model is a model.

Dropping to "opaque SQL strings" is discouraged, both because it's considered error prone ("You should leave SQL to the professionals! There are lots of eyes on this!") and because there is often no graceful way to integrate custom query code with your model layer; instead, you investigate how to do so within the confines of the ORM. For every case where you can eventually find what you need (`select_related("user__friends__email")`), there are a dozen where you can't.

People start writing things in application code that could be handled easily and more efficiently by the database: aggregations are a classic, since ORMs support is either missing or incredibly complex. As soon as the application becomes non-trivial[3], the problems magnify. They don't do this because they are stupid, or bad developers, they do this because it's what the tools encourage: they encourage simplistic CRUD access and mistrust/suspicion/fear of SQL.

SQLAlchemy is quite different from this, because its primary focus is to model databases, not to provide some kind of declarative object language with its own set of semantics that do not exist in SQL.

Because Go can't do things like meta classes or creating/modifying types at runtime, a lot of this simply isn't there, even though people want it.

Sqlx does like 2 things; it adds named query parameter support, and it marshals rows into structs. Squirrel is just a query builder. The whole philosophy that the database must somehow be modeled and that access to the database is done via that model is absent.

[1] This is especially durable tarmac with which to pave your road to hell.

[2] http://github.com/jmoiron/johnny-cache

[3] This can mean "lots of requests", or "complex schema", or "complex reporting requirements"... all sorts of things.

The belief that ORMs are evil is precisely the belief that this sort of code should be repeated everywhere database access is performed. If you have generalized routines for interacting with the database with more comfortable abstractions then string concatenation, you are using an ORM

This is incorrect.

The sqlx library, included in OP, has generalized routines for interacting with the database, but is not an ORM. The squirrel[1] library for Go lets you produce queries without "string concatenation", but is also very much not an ORM.

An ORM is a specific style of library that attempts to map object oriented data patterns to relational concepts. That's why it's called an Object-Relational Mapping. There are good reasons[2] why people find this approach problematic, which aren't down to cargo culting them as "evil" or believing that everyone should repeat data access code in all projects.

[1]http://github.com/lann/squirrel [2]http://en.wikipedia.org/wiki/Object-relational_impedance_mis...

When I wrote the same kind of article in Nov 2011 [1], I came to similar conculsions; ujson was blowing everyone away.

However, after swapping a fairly large and json-intensive production spider over to ujson, we noticed a large increase in memory use.

When I investigated, I discovered that simplejson reused allocated string objects, so when parsing/loading you basically got string compression for repeated string keys.

The effects were pretty large for our dataset, which was all API results from various popular websites and featured lots of lists of things with repeating keys; on a lot of large documents, the loaded mem object was sometimes 100M for ujson and 50M for simplejson. We ended up switching back because of this.

[1] http://jmoiron.net/blog/python-serialization/

Chopsticks are superior for lots of traditional chinese dishes.

What tulip should have been.

Being a long time gevent user, I found goroutines pretty familiar and even unspectacular (no pool.map?!). What really converted me to Go was all of the other ways I found writing Go programs to be more pleasant and less error prone than Python programs:

* consistent formatting for all code

* superior distribution story

* compiler magnitudes faster and more effective than pylint

* programs run faster and use far less memory

* far simpler semantics make it easy to read

* higher quality standard components (net/http, eg)

What I traded for this was a 10-20% drop in productivity, which I was fine with. I use Python for all sorts of quick & dirty tasks still, including some batch processing where there's a big discovery phase, but I write all my software in Go.

There is no type hierarchy and there are no subtypes. Interfaces do not describe type relationships. The conversion required between `[]int` and `[]interface{}` further demonstrates the significance of this distinction.

As for losing type safety when writing generic containers, this is true, and another reason why people don't do this.

Go does not have a type hierarchy, so there is no top type.

You can easily use interface{} to make generic data structures, but interfaces have a time/space/code-complexity cost which makes them not worth it most of the time.

The way you describe it ends up being more awkward, because you have to change code to reference the new path, both in your own code, and potentially in the library you've forked and other dependencies that use it. Vendoring by just maintaining a GOPATH for your dependencies that you control is the preferred way.

Python 2.7 was released after 3.1 and contains many features backported from the 3.x series in order to ease migration.

I spent years out in the woods with my own projects page that nobody could find and very few cared about, happily cutting trees down with nary a witness. I wanted control over backups, presentation, availability. I was terrified of relying on services that could disappear. I found self promotion distasteful, and I was happy to do the work just for its own sake, for the enjoyment of that process.

But then I wrote something people actually used. And they wanted it on Bitbucket, and then on Github, so they could contribute to it and track its progress.

Now that I've come in from my Zen training out in the wilderness, I feel that the old me who wanted "control" over all of my creations was immature and selfish, abstaining from participation in a community and helping no one but myself. I think that the social aspect of hosted coding sites, both for collaboration and exposure, is much more valuable than the control you get from running the whole show yourself.

Yeah, I think that bold looks better with sans serif typefaces than underline, so that was a design rather than an ergonomic decision; perhaps not a correct one. I think underlined blue text is probably the "most linky" looking text, given the history of the web.

I've independently come up with the same decisions on many of these points and, weirdly, a very similar blog design:

http://jmoiron.net

I decided even to dispatch with the header bar, putting navigation at the bottom of the main page; presumably, people will only care to see more of my writing if they've read the article. Not as good for people revisiting the page, but they can likely remember http://jmoiron.net/blog or bookmark it in order to get to the search.

My fonts are slightly smaller and slightly less contrast, and after looking at Gemmell's blog I wonder if I should change that decision. I wanted to have emphasized elements of the text easily distinguishable from the body while not distracting.

Also, I agree with Gemmell's "The basic tenets of hypertext should be left alone". One of these basic tenets, to me, are the semantic meanings of Blue, Purple, and Red text for normal, :visited, and :active hyperlinks. If you want to chose a link color, do not invert or violate these classical meanings.

Unfortunately if you focus on beauty you sometimes break pragmatism. The JavaScript example in particular is not merely a typographical convention, but a way to avoid common errors.

    var i=1
      , j=2
      , k=3

It can be difficult to spot the lack of a trailing comma, or the end of this declaration list having a comma instead of a semicolon (, vs ;), both of which will break the execution of your script.

So please, do not change your code to make it look better without understanding why it's like that in the first place.

The classic example is tchanging the following to allman/gnu style braces would break it in JavaScript:

    // works, returns {a:1,b:2}
    return {
        a: 1,
        b: 2
    }

    // semicolon inserted after return, returns undefined
    return
    {
        a: 1,
        b: 2
    }

Redshift is a real physical phenomena describing the way light wavelengths get "shifted" (stretched, to visualize) towards the red as they are seen coming from something moving away from the observer:

http://en.wikipedia.org/wiki/Redshift

http://en.wikipedia.org/wiki/Hubbles_law

From the twitter announcement (https://dev.twitter.com/blog/changes-coming-to-twitter-api):

"To ensure that Twitter users have a consistent experience wherever they see and interact with Tweets, in v1.1 of the Twitter API we will shift from providing Display Guidelines to Display Requirements, which we will also introduce for mobile applications. We will require all applications that display Tweets to adhere to these."

If Marco's reading is wrong, it's still the most obvious reading of the original text.

Also:

"Nearly eighteen months ago, we gave developers guidance that they should not build client apps that mimic or reproduce the mainstream Twitter consumer client experience. And to reiterate what I wrote in my last post, that guidance continues to apply today."

Apropos to the Anil Dash version of the piece, the original one is already heavily tarted up with euphemisms like "guidance" where "warning" would more clearly convey their intentions.

I understand the type of petty social engineering that Anil has attempted here, by spreading "Awesome" and "Excitement" around and accentuating the good instead of the bad, but I think it does developers a great disservice to assume that the biggest problem with bad news was the tone of its delivery and not the news itself.

I don't know how most other people code, but I rarely write out a module, even from scratch, in a way that doesn't require editing and rearranging. I start with a skeleton which reflects my best understanding of the problem and then try my best to fill it in with code. Sometimes the organization is not quite right, or sometimes you realize you need an extra parameter somewhere that is out of scope; but whatever it is, there's always some kind of stumbling block that requires reorganization.

The overall shape is usually close enough, but it always requires refinement. That refinement happens in a loop; you move stuff around, do a little nip tuck, maybe attempt to fix some bugs, and then run the module (or the tests). Optimizing this loop is important (it's one of the major benefits of scripting languages); once you figure out what change you should make, or what experiment you should run, the faster you make it happen, the faster you get the results, and the faster you can get back to the important mental aspect of figuring out how to fix or create whatever you're working on.

This is available via the pkg-info in all installed packages; iirc version is required for distutils, and there is pep386 for version number format, so it should be possible to determine version number as well as compare them for all well-behaved packages. There is even a package which will find and parse pkg-info for an installed package called pkginfo:

http://pypi.python.org/pypi/pkginfo

http://markdotto.com/bs2/docs/less.html

Odd, it says that lessc can take --compress, but using the most recent version of less in npm says that there's no such flag.

Maybe it's on the way, or the bootstrap authors know something we don't.

full disclosure: johnny-cache author

The top commenter in OP gave a great rundown of these projects and their evaluation of them at YCharts at a Django NYC meetup a month-ish ago; I'm sure his slides are available on the nyc django site somewhere.

All of these projects "automatically" manage cache for querysets, but they do it different ways, and can be susceptible to poor performance under different usage patterns.

From what I can tell, JC adds the lowest amt of overhead to cache misses and hits, and uses the simplest (it's mildly sophisticated, but still straightforward) management algorithm. It's the only one that works fine when using UPDATE queries that do not mention row ids, and (as a result) is the one that most greedily invalidates on writes.

The others are fine projects run by smart people, and depending on your site's situation, I'd recommend some of them over johnny-cache. It's a good idea to evaluate them all, as they certainly did at YCharts (his section on JC was very accurate), and as OP seems to have done.

Except that lots of people don't agree with you. People have already been using gnome-do (quicksilver) for fast keyboard app launching for a few years; it's faster, more focused, and less keystrokes than the Unity bar. You also can't use it without installing some extra stuff and getting unity to give up its monopoly on the mod (win) key.

As for the alt+tab functionality.. well, if you are used to OSX and its third rate window management (can't move or resize windows without aiming at a tiny bar or handle?) and broken virtual desktop implementation, then maybe this kind of impoverished behavior is a welcome change. Since I have OSX experience, global menu's aren't a big problem for me, except when they are hidden by default and show up on different screens depending on window location. An interesting compromise for multi-monitor users, but mystery-meat menu decision is mind boggling.

For people who have now had ~16 years of window-centric keyboard navigation, having no good muscle-memory way to switch among 3 windows in 2 different applications is a nightmare. Having alt-tab slide your entire desktop over is just an insult on top of that injury. We use exact same set of tools (vim, chrome, and a terminal), but you must only ever use one window each; I can't imagine how you could tolerate the new app-switcher's behavior otherwise.

Your point about the only change being the font is also interesting, since in Ubuntu 11.10 you cannot change your fonts (or any colors on your theme, or gtk2 themes which a majority of apps will still use) without installing some third party tool with a terrible UI that looks worse than anything has since gnome-2 was released. The terminal is the only program that provides a UI for changing it.

In general, I agree with the posts in this thread that say you can massage the XFCE desktop into a worse version of what gnome-2 was about 3 years ago. After trying very hard to get along in Unity and 11.10 (gvim freezes for 30 seconds on start-up if you have installed foreign language packs, a bug fedora fixed about 9 months ago but persists in ubuntu), I've given up am using XFCE w/ Compiz (which doesn't work that great but is usable) until I find a weekend where I can ditch it for something else.

The part that's sad is that I did not find Gnome3 to be much better than Unity; it still has the same "enhanced" (broken) window/app collection which I find completely unsuitable for doing real work without encountering incredible friction. So for me (and probably others), Gnome3 has joined KDE4 as some unusable tangent, essentially killing off systems I was very happy and productive with. Gnome2 is a temporary fix; it's a dead end. XFCE still feels very spartan; in the past 3 years Gnome2 has got better and better and XFCE has barely moved. Hopefully, someone (maybe Mint, or maybe the ElementaryOS guys) will raid the remnants of Gnome2 come up with a system that has a living development path.

I've been using FuzzyFileFinderTextmate for a long time. I'm not sure it's still in active development, but it's worked just fine for me; is there a reason for me to switch to Command-T?

I also stick to buffers rather than tabs (for terminal/gui consistency) and use buftabs, which I quite like.

I normally shy away from writing provocative information-light rants. The important parts (for me):

1. github is a social service, or perhaps best used as a social service

2. it has "the momentum", which is paramount for social services

3. git has "the momentum" wrt adoption of dvcs

There are assumptions implicit within these arguments that I've accepted, based largely on the summation of 'soft' and anecdotal evidence that has built up over the last couple years. People are right to challenge them, and right in their criticism that I take them at face value.

I've been watching this pretty closely; the multi-db in 1.2 alone makes it worth the upgrade. It's about a 20 line backend router to get round-robin RODB usage, which is very welcome for sites under high read loads.

A lot of systems have been redone to take custom XBackend classes (like the CacheBackend before it, EmailBackend and others are now available), and in my experience this is a good level of access for customizing parts of Django.

Lots of applications aimed at Django are already 1.2 compatible. They've really spent a long time (a few months after freeze) making sure this release is going to be solid, so I have high hopes.

I've found actionbutton to be a step below the old reviews, perhaps for similar reasons that Tim now finds his honeymoon in Japan souring. The reviews there tend to be a lot more about being entertaining than enlightening, and their stated goal of having the average score being 1/4 stars has been thrown to the wayside. I also don't really play games anymore.

There are a few potentially "important" things that I hope escapes IC and its graduates and becomes more mainstream. Reviews that explain the context of their creation, and reviews that examine the philosophy of the game design. Unfortunately these are hard, require work, and don't look any better on the glossy page. Some examples:

Rogers wrote a fantastic review[1] of Romancing Saga: Minstrel Song that starts off by chronicling the entire career of its designer (Kawazu) and reviewing some of his earlier games with depth and insight. There are clear patterns evident in his early games, which are then used to give context to and explain the current game under review.

The second is probably my favorite video game review, which is of Windwaker by Eric-Jon Rossel Waugh[2]. EJRW is a better and more insightful (but less colorful) writer than Rogers, but perhaps not as colorful. He made an old forum post I distinctly remember lamenting the laziness of linear game design, which is something of a cliche. When challenged for an alternative, he backed it up with a very interesting review of the use of "danger" rather than contrivances like keys or inventory items as a limit on exploration in Dragon Quest/The Legend of Zelda.

As for this particular posting of Tim's, I don't really have much more to comment. Like I said in my original comment, it's classic Tim Rogers, and it's hard to explain what "classic Tim Rogers" is without just pointing people to another 15-20 articles similar in construction and content. It feels odd to write about it in a familiar tone since I was largely a lurker at IC (as I am here).

[1] http://www.largeprimenumbers.com/article.php?sid=saga

[2] http://www.insertcredit.com/reviews/windwaker/windwaker1.htm...