cscotta

It is unfortunate that many young tech companies, in an effort to differentiate themselves, work to invent a new term and cloak it with the appearance of a movement or industry trend. It's further unfortunate when those to whom this term is being marketed recoil with laughter and amusement at its implications, forcing its purveyors to double down and attempt to re-assert control over the meaning of a term which may please analysts and folks with Twitter accounts who fashion themselves "thought leaders," but whose roots and meaning are firmly in marketing rather than industry and art.

So let's talk about the marketing term "No-Ops."

In AppFog's attempt to assert a meaning over the marketing term, we're told that it means "developers can code and let a service deploy, manage and scale their code." I have yet to meet a single company facing a sizable technical challenge whose performance and availability needs could be met by a strait-jacket PaaS with an autoscale button. I'm not saying that many smaller companies don't use such services successfully – that's very much true. But let's be clear: the dream of push-button autoscaling while letting "somebody else" handle deployment, monitoring, instrumentation, and anything that may go wrong in the middle of the night is a marketing dream. As engineers, we have a business need and emotional need to own our availability [1]. Placing the sum total of your operations into a PaaS providers hands, biting down hard on that marketing dream of NoOps, and throwing the pager out the window doesn't mean you have nothing to worry about. It just means that you don't care, and can do nothing about it.

But it's not enough to stop there. Instead, the author sees fit to posit his / her marketing term as the continuation of a history in the evolution of web operations, and proclaim that the term is one that "traditional operations" personnel revile. If "No-Ops" is a success of any sort, it's a marketing win, not a technical one – and certainly not an operational one. If you think that you can place every single egg in your company's basket in the hands of PaaS providers and never worry again until you have to twiddle the auto-scale dial after which everything will be fine, you're only fooling yourself.

Who will migrate data on an oversubscribed Postgres shard to two more shards by night by partitions of account IDs? Who will enable dual-writes, run a migration, then cut over reads as we move into Riak? Who will notice a spike in await on the Kafka RAID, recognize the week-over-week trends pointing to your team running out of iops, order, and rack a set of new boxes with SSDs before it's too late? Who's watching the switches and keeping track of which racks have GigE and which have 10GigE uplinks to the next rack to avoid oversubscribing the network?

It's rare in our industry to see a promise so removed from reality. Indeed, if NoOps is a movement at all, it's powered only by the dream of not having to do one's job which is to ensure that a company is able to deliver on their business value. Who among us isn't tempted by such a promise? [2]

[1] http://www.whoownsmyavailability.com/

[2] http://www.youtube.com/watch?v=fM9o8MxLX3Q

>> "It doesn't really fail - none of the issues described were "failures" really."

These absolutely were failures.

The author listed several instances in which the database became unavailable, the vendor-supplied client drivers refused to communicate with it, or both. Some of these scenarios included the primary database daemon crashing, secondaries failing to return from a "repairing" to an "online" state after a failure (and unable to serve operations in the cluster), and configuration servers failing to propagate shard config to the rest of the cluster -- which required taking down the entire database cluster to repair.

Each of the issues described above would result in extended application downtime (or at best highly degraded availability), the full attention of an operations team, and potential lost revenue. The data loss concern is also unnerving. In a rapidly-moving distributed system, it can be difficult to pin down and identify the root cause of data loss. However, many techniques such as implementing counters at the application level and periodically sanity-checking them against the database can at minimum indicate that data is missing or corrupted. The issues described do not appear to be related to a journal or lack thereof.

Further, the fact that the database's throughput is limited to utilizing a single core of a 16-way box due to a global write lock demonstrates that even when ample IO throughput is available, writes will be stuck contending for the global lock, while all reads are blocked. Being forced to run multiple instances of the daemon behind a sharding service on the same box to achieve any reasonable level of concurrency is embarrassing.

On the "1GB / small dataset" point, keep in mind that Mongo does not permit compactions and read/write operations to occur concurrently. As documents are inserted, updated, and deleted, what may be 1GB of data will grow without bound in size, past 10GB, 16GB, 32GB, and so on until it is compacted in a write-heavy scenario. Unfortunately, compaction also requires that nodes be taken out of service. Even with small datasets, the fact that they will continue to grow without bound in write/update/delete-heavy scenarios until the node is taken out of service to be compacted further compromises the availability of the system.

What's unfortunate is that many of these issues aren't simply "bugs" that can be fixed with a JIRA ticket, a patch, and a couple rounds of code review -- instead, they reach to the core of the engine itself. Even with small datasets, there are very good reasons to pause and carefully consider whether or not your application and operations team can tolerate these tradeoffs.

If you're curious about Ordasity's load balancing strategies, check out Section 4: "Distribution / Coordination Strategy" in the docs: https://github.com/boundary/ordasity

The code is also fairly straightforward to follow if you'd like to take a glance in the repo as well.

Hey Mitchell,

Thanks for asking -- in short, the use cases are a bit different. Finagle is a framework for building asynchronous RPC systems (e.g., services or APIs with transports over HTTP or Thrift), and Ordasity is a library for cluster membership, load balancing, and distribution.

Twitter's put together a nice toolkit atop Netty for building reliable services and describing communication between them. These services are generally stateless and might be balanced by something like HAProxy (when using an HTTP transport), or via a round-robin approach in the case of Thrift clients. A closer comparator for Finagle would be something like Scalang, our library for building hybrid Erlang / Scala distributed systems (though it's also excellent for pure-Scala systems as well): https://github.com/boundary/scalang

Ordasity is designed for describing stateful clusters in which individual nodes are responsible for claiming longer-lived work units -- think of it in terms of "a processing shard" of the system's total load. The library's primary goals are to help you:

- Describe the cluster in simple terms such that when each node comes online, it joins the cluster and is aware of all other nodes (and vice versa)

- Distribute work across the cluster by directing each node to claim an even "count" of work units or "even distribution" of the total load imposed by those work units.

- Automatically rebalance work units across nodes in the cluster as the amount / intensity of work or cluster topology changes.

- And gracefully manage maintenance or downtime scenarios by draining work units to another node.

In the end, Ordasity and Finagle are both tools for building distributed systems on the JVM. However, the types of systems they're designed to describe are a bit different. Hope that helps; let me know if you'd like me to clarify something.

Hi Roland,

Ordasity could certainly be used in building distributed JRuby applications. However, I'm not sure that pairing it with Rails would be the best use case.

Ordasity is designed for building stateful distributed services -- e.g., systems which can be described in terms of individual nodes serving a partition or shard of a cluster’s total load. Rails applications tend to follow a stateless model in which any application server could serve any request.

Ordasity is more appropriate for building services in which a specific node will be serving all [requests / queries / events] for a work unit in the cluster. In our case, that's our netflow aggregation systems (which pull data together from a network of edge nodes into a single Kafka stream), and our event stream processing system (which we shard by client). In this example, we have two Ordasity clusters -- one comprised of Kafka nodes, and a second containing our event stream processing tier. Ordasity handles work distribution across both tiers, ensures that both clusters are aware of each other and able to communicate, and helps us guarantee that everything is wired up properly (i.e., each node on each tier is communicating with the proper node on another tier) when data is passed between them.

In short -- yes, you could build a distributed service in JRuby (or Clojure, Mirah, Scala, Java, etc.) with Ordasity. Just wanted to offer that note to clarify use cases.

I'd like to offer a counterpoint to the author's suggestion that "you'd be an idiot not to buy [and dispose of] a new laptop every year." I'm a software engineer and put a lot on my computer's shoulders, running an entire cluster's worth of software as part of my development environment.

I value longevity and durability in products I buy. It's nice to pick a machine and stick with it. It's a long-term companion. It's about slowing "disposable computing's" cycle of production and obsolescence. It feels good to prove that, with a few upgrades over its lifetime, a well-engineered product can be useful -- even as a primary computer -- for years to come.

My Spring 2008 MacBook Pro (http://cl.ly/2H1l2X1Q2w181Z2P1Y3P) will be three years old this Saturday, and I couldn't be happier with it. My previous machine was a 15" PowerBook G4 purchased in 2004. Both have been fantastic primary computers. An occasional upgrade and maintenance can make all the difference in extending the useful lifespan of a machine.

A few months in, I maxed out the memory to 4GB, which is still sufficient despite running our entire stack and an IDE or VM. Last summer, I replaced the 7200RPM drive with a 160GB X25-M. A few months ago, I added a second 48GB SSD drive via the ExpressCard slot to regain a bit of the storage sacrificed by choosing a faster drive. Over the three-year lifetime of the machine (so far), these upgrades cost about $625.

During that time, the manufacturer has also done a great job standing behind the laptop, replacing the keyboard/top case, one battery, and one power adapter. I'll take it in for one last servicing before the warranty runs out (to fix an unreliable Caps Lock key and clean the DVD-RW drive I never use), and may purchase one more battery at some point. Aside from this, it's in perfect condition and plenty fast enough for Java/Scala/Python/Ruby/Android development and testing.

This computer's followed me from the week I graduated college as an aspiring freelancer through three years of building a career in software engineering. It's got some life in it yet.

A few of these options are good in principle, but are not necessarily informed by the reality of operational experience with the more-common failure modes of AWS at a medium to larger scale (~50 instances +).

The author recommends using EBS volumes to provide for backups and snapshots. However, Amazon's EBS system is one of the more failure-prone components of the AWS infrastructure, and lies at the heart of this morning's outage [1]. Any steps you can take to reduce your dependence upon a service that is both critical to operation and failure-prone will limit the surface of your vulnerability to such outages. While the snapshotting ability of EBS is nice, waking up to a buzzing pager to find that half of the EBS volumes in your cluster have dropped out, hosing each of the striped RAID arrays you've set up to achieve reasonable IO throughput, is not. Instead, consider using the ephemeral drives of your EC2 instances, switching to a non-snapshot-based backup strategy, and replicating data to other instances and AZ's to improve resilience.

The author also recommends Elastic Load Balancers to distribute load across services in multiple availability zones. Load balancing across availability zones is excellent advice in principle, but still succumbs to the problem above in the instance of EBS unavailability: ELB instances are also backed by Amazon's EBS infrastructure. ELB's can be excellent day-to-day and provide some great monitoring and introspection. However, having a quick chef script to spin up an Nginx or HAProxy balancer and flipping DNS could save your bacon in the event of an outage that also affected ELBs, like today.

With each service provider incident, you learn more about your availability, dependencies, and assumptions, along with what must improve. Proportional investment following each incident should reduce the impact of subsequent provider issues. Naming and shaming providers in angry Twitter posts will not solve your problem, and it most certainly won't solve your users' problem. Owning your availability by taking concrete steps following each outage to analyze what went down and why, mitigating your exposure to these factors, and measuring your progress during the next incident will. It is exciting to see these investments pay off.

Some of these:

– Painfully thorough monitoring of every subsystem of every component of your infrastructure. When you get paged, it's good to know exactly what's having issues rather than checking each manually in blind suspicion.

– Threshold-based alerting.

– Keeping failover for all systems as automated, quick, and transparent as is reasonably possible.

– Spreading your systems across multiple availability zones and regions, with the ideal goal of being able to lose an entire AZ/region without a complete production outage.

– Team operational reviews and incident analysis that expose the root cause of an issue, but also spider out across your system's dependencies to preemptively identify other components which are vulnerable to the same sort of problem.

---

[1] See the response from AWS in the first reply here: https://forums.aws.amazon.com/thread.jspa?messageID=239106&#...

Absolutely - thanks, Matt. This is a very well-written, thorough article, and it provides a clear, thoughtful survey of the landscape for folks who are interested in parallelizing their programs and learning more about the advantages and tradeoffs implicit in different approaches. This piece has a lot of potential to catalyze thought within the community about how best to move forward.

The explosion of interest in the subject intrigues me. The work that's been going on in terms of making mainstream Ruby libraries threadsafe is great to see, along with fibers and evented approaches. MacRuby's exposure of Grand Central is especially unique. But interest and upvotes must also be followed by action. There will be a lot of false starts. We'll probably see a handful of approaches which bloom and fade from popularity. While the "fragmentation" issue is there, I tend to think of it in terms of spirited experimentation. In any case, people really care now, and it's that drive that forces a community forward. That's good to see.

Keep it up!

Pushing a language, its libraries, and extensions which have traditionally presumed linear execution into a threaded/parallel world is a minefield. I'd like to offer a few thoughts on some factors which might have contributed to this complexity, to reframe the question of what might constitute the "best" concurrency strategy in terms of tradeoffs, and conclude with a call for us to take up the task of reasoning about our programs in a parallel context.

We've seen an explosion of interest in non-threaded, single-process approaches to concurrency in the Ruby and Python communities in the past couple years. Much of the difficulty here lies in frameworks, libraries, and development paradigms which were not designed to be threadsafe (or to teach threadsafe programming) from the start. Developers and library authors have for years been able to operate under the assumption that Ruby code in a single process is executed serially and not in parallel. In environments which permit parallel, concurrent execution inside the boundaries of a single process, authors must return to the foundations of what they've created and scrutinize every bit, reconceptualizing their programs in terms of shared state and mutation.

Individual developers relying on these libraries must also scrutinize each and every gem they require to ensure that the code is threadsafe as well. This is not an easy task, as reasoning about state is inherently difficult, especially when the original program may not have been designed with concurrency in mind, and even moreso when one was not the original author. Further, we haven't seen a significant chasm between things which "are" or "are not" threadsafe emerge in the Ruby community, and it is not common to certify one's code as "threadsafe" on release. That said, it's commendable that the Rails team has worked diligently to piece through each component of the framework and certify it clean.

I do not mean to suggest that multithreaded code written in Ruby is uncommon, unsafe, or a bad idea in general. Many applications running in environments without a GIL/GVL such as JRuby utilize this functionality effectively today. What I'm suggesting is that process-based concurrency (Mongrel/Passenger/Unicorn, et al) is often favored by developers because it eliminates a large swath of potential pitfalls (or rather, trades them for increased usage of system resources).

In light of this, we've seen developers in the Ruby and Python communities experimenting with and popularizing alternate concurrency models, not the least of which include cooperatively-scheduled fibers/co-routines and evented programming. These approaches avoid a specific subset of the challenges in concurrent multithreaded execution, while enabling limited concurrency in one process. Matt is quick to point out that these models don't achieve true concurrent parallelism, but do offer significant benefit over standard serial execution. At the same time, they impose a different sort of complexity upon the programmer -- either requiring her to reason about a request or operation in terms of events and callbacks, or to use a library to cooperatively schedule multiple IO-bound tasks within a single VM (which may hide some complexity, but introduce uncertainty by clouding what is actually happening behind the scenes).

I wish Ruby and Python the best here. I have a significant investment in both. But as long as developers must ask of libraries "Is it threadsafe?" with fear and negative presumption, traditional multithreaded concurrency might not be ideal given the investment required to achieve it safely and correctly.

More than anything, I hope for continued complex reasoning and thought on this question. We simply must move beyond reductionist statements such as "threads are hard!" to progress. I'd suggest that threaded programming in and of itself is not hard per se -- reasoning about shared state is, and the developer bears a responsibility to either eliminate, or minimize and encapsulate it properly in her code. Evented and coroutine-oriented programming brings its own challenges. Actor models are interesting and may be appropriate for some contexts. STM's pretty cool too, but it would not be proper to hang one's hopes upon something which is not likely to cure all ills.

There is no panacea for concurrency. There will always be challenges and tradeoffs involved in developing performant, efficient applications within the constraints of both hardware and programmer resources. We would do well to push ourselves toward a greater understanding of the challenges in developing such programs, as well as the tradeoffs involved with each approach. I think this article is a step in the right direction.

"It's maddening to see an article that you think will let you make progress in a problem you're working on, but unable to access it because of the $15 fee..."

Enough progress to justify spending $15 (less than the price of dinner downtown)? I agree, many find the revenue models / cost structures of professional organizations outmoded. However, I find this rather harsh aversion to paying a few bucks for peer-reviewed, scientific content curious.

If it's of any help to you, many public universities, schools, and local libraries provide full access to ACM content, often both on-site and off. The most common method of providing access to this is via EBSCO's "Business Source Premier" database. I frequently read ACM articles from my desk via the web; a quick title search in EBSCO will pull up the title in about 20 seconds, and I can download a PDF of about any article published since 1965 to send to coworkers.

That said, if price is an issue, please check with your local library. Odds are good that your tax dollars are already paying the cost for you to read these articles from the comfort of your home or office. This isn't just true for the ACM -- even in the age of paywalls, your library's probably been quietly working to provide digital access to all of this for the past decade.

Mike's a great guy, and I appreciate his post but would respectfully disagree with the proposal to strip the standard library down to barebones and break out the removed classes into gems (along with the post's title to which I object, but will not take issue).

Despite its lack of use in web projects, DRb is one of Ruby's most interesting modules. Geoffrey Grosenbach's recounting of _why's 2005 FOSCON presentation is among the highlights - see here for an example: http://www.smashingmagazine.com/2010/05/15/why-a-tale-of-a-p...

The English RDoc is eminently fixable as well, whether by a core contributor or not. In fact, the first commenter on the post highlights a quick guide to contributing to the language.

More importantly, the process for formalizing the Ruby language into an international specification is progressing nicely, first with Japan's ISC committee, and later with ISO. While the draft specification is restricted to the language itself and does not include several components of what we know as the standard library, it would be prudent for the language itself to continue to stabilize. You can find the 12/2009 draft spec here: http://ruby-std.netlab.jp/draft_spec/draft_ruby_spec-2009120...

Finally, many groups of developers have made tremendous progress on non-MRI/YARV implementations, such as JRuby, Rubinius, IronRuby, and MagLev, with others such as SAP's BlueRuby continuing to show promise (passing over 75% of RubySpec as of last year). Sweeping changes to the language and standard library impedes the maturation of these implementations, placing additional burden upon their developers and sponsors who are already doing tremendous work to advance the state of the language across multiple platforms.

We would also do well to remember that the community of English-speaking web developers using Ruby represents a small subset of the Ruby community as a whole. Though some of us might not find daily use for all of the modules in the standard library, it needn't be assumed that they should be removed and/or broken out.

Many languages experience similar growing pains. The Python standard library contains a few modules which seem out of place to most web developers, the JDK is a bit of a grabbag, and the .NET class library has a handful of oddities. Nonetheless, it's important to remember that many people use these components and they work quite well. What some may decry as cruft and stagnation, others might regard as a sign of stability and maturity.

What I appreciate most though is the civility of this discussion. Between Mike's first post, Eric's reply, and this follow-up, the conversation's professionalism and camaraderie is impressive. It's great to see programmers coming together to discuss improvements to the language and its direction in a calm, refreshing environment. I haven't written much Ruby in the past year, so it wouldn't be appropriate of me to offer an extended post as a reply, but I'm glad to see the discussion happening.

This a great, succinct guide - glad to see it.

One very important step five to add: check with your state's Secretary of State office to see if there are any additional steps that must be taken. Many states require foreign-registered LLCs and corporations to submit an "application for authority to transact" or otherwise register with them.

Depending on where you're located, a local business registration fee or license may be required as well.

It's also important to apply for an EIN number, which is generally required before opening a bank account in the business' name, and for tax purposes: https://sa2.www4.irs.gov/modiein/individual/index.jsp

There may be other steps that I'm missing, but as with any guide to navigating these waters, always ask for help in your particular situation, and don't be afraid to (indeed, please do!) consult a lawyer.

I've read through this post a few times and am having some difficulty understanding both the original problem and the solution that Greg is suggesting.

From what I can gather, it seems that Netflix's product development team is frustrated to find that there is no integrated support for DRM'd media streams provided by the OS. From here, he jumps to the conclusion that the best approach is working with individual handset manufacturers to patch in support for this at the OS or kernel level. This is difficult to understand.

It's an operating system. You build things on top of it. One does not go to every PC manufacturer to add a "feature" to one.

Spectres of "fragmentation" aside, let's remember that these phones are real computers running a real Linux-based OS that run real software, written in real programming languages like C, Java, and Scala. They also have support for encrypted transports like SSL and high-quality video codecs like H.264. Heck, Adobe's even done it - secure content streaming on Android is possible, and works fine (speaking strictly of the transport and decoding layer - leave it to the bloggeurs to hash over what happens when you paint content on a screen).

These facts suggest that implementing secure DRM'd video streams and a player for them is not only possible, but much easier than working directly with two of the most backward-facing industries in technology now (film studios and mobile phone providers) to add this functionality.

Hey Hugo,

I've somehow become a bit of a devil's advocate for Java lately. Thanks for pinging me - happy to offer a few thoughts.

I've not actually tried using Jersey/JAX, but it looks like a very clean setup for quickly writing low-latency JSON APIs. I'm impressed by the Getting Started example (here: https://jersey.dev.java.net/use/getting-started.html) and find the use of decorators to specify path, methods, and content types creative.

As for the rest, keep in mind that you don't have to use anything that you don't want to. I've done a fair amount of Java development and never actually used Hibernate, Maven, or JBoss. For the most part, the complexity that these tools can introduce, while great for many projects, kind of scares the crap out of me. Like you, I'd always go for the simplest workable solution available.

Jetty is a very nice, simple alternative to Tomcat/JBoss, though I find Tomcat to be rather nice myself. One thing I've come to love about Java deployment is that you're literally deploying just one file (a .jar or .war) rather than hundreds.

If you're looking for a nice, clean ORM, check out ActiveObjects by @djspiewak. Its design is based on the active record pattern, and it's very easy to get up and running with about any database as a backend. I used this on a large project ending a few months ago and had a great experience with it. Daniel's also available via Twitter and IRC and tends to be happy to offer an answer to a quick, informed question. Here's the post he introduced it with (http://www.codecommit.com/blog/java/an-easier-java-orm), and here's the project page. (https://activeobjects.dev.java.net/). I'd also strongly recommend checking out these docs he put together: https://activeobjects.dev.java.net/ActiveObjects.pdf

I really love the "static files + AJAX" approach that you've suggested (it's actually the same one mentioned in the project I referred to above). If you were to build something with Jersey and ActiveObjects, perhaps with a small caching layer in between (either an in-app cache if you'll only have one app server, or memcached), you will have produced a screaming API for your application.

The Play framework (mentioned below) is also very good, and has emerging support for Scala, which I and many others find to be much cleaner and more expressive than the equivalent Java. If you're feeling adventurous, I whipped together a quick little framework for building zippy JSON APIs and simple frontends via a lightweight wrapper around the servlet API. It's called Miso. Here's the info on that (and do ping me if you feel like playing with this - I may have a more recent version): [http://blog.paradoxica.net/post/401365886/hello-miso]. See here for docs, and especially page 6, which shows what a sample controller / API endpoint looks like: http://u.phoreo.com/gx.pdf

Of course, this also depends on whether you actually need that, and how much experience (or desire to learn) you have. If you're comfortable with Python, something like Flask or Piston might be more appropriate. At the same time, I would suggest that those who say that '"x language" or "y framework" makes developers infinitely more productive' aren't offering terribly productive advice, as I find myself about equally productive in Java, Scala, Python, and Ruby, and often opt for Scala or Java unless I'm hammering out something quick and imprecise. That is to say, it's definitely a matter of preference. So, choose what you think will make you most productive, or, choose to learn something new which might be a bit difficult at first, but will also teach you a lot about statically-typed languages and could potentially meet the needs of your application with far fewer resources and lower latency on the server-side for years to come.

Regardless of which path you choose, feel free to hit me up with any questions along the way.

You've received several votes for your comment, and I respect that.

But I also respect those who prefer a language which offers type safety. One which does not lay traps of unexpected coercion and unfathomably unusual truth tables. I respect a language which can guarantee and enforce immutability at compile time. I respect a language which has had a sensible packaging system from the outset. One which served as the foundation for many as an introduction to object-oriented programming. A language whose primary implementation is based upon a powerful, performant virtual machine, and one which has had excellent support for concurrent execution of programs and components thereof for nearly a decade. I respect a language whose implementations are not wildly divergent. I am fortunate enough to work with a language whose underlying implementation is flexible enough to host dozens of both static and dynamic languages - including JavaScript itself.

What I do not respect are blanket unqualified absolutist statements and fundamentalisms. I understand that you may have found yourself frustrated at points during which you've written Java in the past. I would like to hear about these, and understand them better. But I'd also encourage you to avoid such totalizing statements, as for a variety of reasons, many may have very good reasons for enjoying things you do not.

I've been watching this come together for awhile now - Aerostat's a pretty cool little service for managing a cluster.

Rather than relying on a custom internal DNS implementation, Aerostat automatically names servers as they are added to a cluster based on the service they provide, then replicates these names across all nodes in the cluster. This process ensures that all nodes are aware of a new service in under a minute. It also solves the "swiss cheese" problem of instances named cass-00, cass-01, cass-02 and so on by filling in gaps as nodes are added and removed.

For those in the Portland area who are interested, Gavin will probably be giving a talk on this at PDXDevops sometime soon.

I'd consider asking yourself what you expect and/or hope to get out of it. A two-hour commute for an uninspiring job does not sound like something terribly enjoyable or fulfilling.

Perhaps others here could suggest different opportunities in the Vancouver area?

Some really interesting points on the connections between starting a company and a life together, especially regarding "learning to learn." Suffer through the hard times, adapt, and celebrate the good when it comes.

Unfortunately, in the majority of America and the rest of the world, it is illegal for many entrepreneurs to marry. If "marriage teaches you what real love is all about," one would hope that it's not the only way.

> 90% of Java's libraries are very, erm, "enterprise."

Are they? Apache's Commons collection of libraries such as ArrayUtils, StringUtils, IO, and the like, along with JSON-Simple/jvYAML, the servlet API, any database driver, Log4J, and just about every other library I use on a daily basis has a clean, focused, well-designed, and impeccably well-documented API.

I'm sure there are no shortage of "enterprise-y" libraries and APIs, but could you offer a few examples of common, popular libraries used in a large portion of J2SE (not EE) projects whose APIs are poor on a level beyond that which may offend your aesthetic sensibilities? Some that come to mind for me might be Swing (I don't write GUI apps), but am curious to hear what you've encountered.

I'm not a CPA or lawyer, but am almost absolutely positive that these taxes are applied based on your residential address. Attempting to cheat (get around, avoid, evade, what have you) taxes is almost universally a Bad Idea (speaking as someone who's been audited - though came out owing nothing).

Regardless of whether they seem onerously high, communities pass payroll taxes to generate essential revenue, and people vote on / for them. In that sense, these taxes can be understood as something a community has proposed and agreed to pay. Attempting to circumvent this agreement (the law) by setting up a sham address that's nothing more than a PO box isn't just shady - it's unethical and deprives the city of revenue its residents have agreed to pay and deemed essential for municipal operations.

If I'm mistaken, feel free to correct me, but legal or not, I would still not attempt this.

About a year ago, I spent a weekend building a letterpress. It's turned into a fun hobby.

Printmaking is much more difficult than I'd have expected. One of the biggest challenges is maintaining a clean, inkless field. It's so easy to get ink on your hands, at which point it's nearly impossible to pick up a sheet of paper without smudging it before you're within three feet of the press itself. Registration (proper alignment) is a challenge as well, but is manageable. Working slowly and deliberately is difficult but certainly rewarding (though the reward here is spending an hour with lacquer thinner cleaning off the type, but I digress).

Even so, it feels great to take a Saturday evening away from the Internet, sit down with a set of lead 72pt Franklin Gothic, a brayer and ink, and print a few runs of something that strikes you as amusing at the moment.

Here are a couple photos from the last experiment:

Printed: http://www.flickr.com/photos/cscotta/5018870586/lightbox/

Letterpress: http://www.flickr.com/photos/cscotta/3825133446/

(See the "luck" tag for a couple in-process shots)

Right on, thanks Ask! I'm checking out some of this right now and might not be able to get through it all today, but will give it a try. Just shot you a couple messages outside of HN - love to talk when you have a chance!

Hi there,

The task benchmarked was from a component of our messaging stack at work that I'd ported. My intention was to offer an example of the end-to-end performance of a real-world task that someone might be writing, rather than a sample task that's not much more than a no-op. I'd suggest that this particular one is real-world as it's straight out of one of our applications. I've no intention of trying to be dishonest here - just offering the measurements of my (and our) internal evaluation of the tool for our needs.

While I can't provide the source of the task, I can offer the quick-and-dirty source of the quick little "PyInvoker" I'd whipped up (sorry - didn't realize there was something called PyInvoke at the time). It just takes a message, unpacks the JSON, and uses getattr to call the appropriate task. Nothing fancy like e-mail error notifications, retries, and the like: https://gist.github.com/18a30689832569d67861

Anyhow, always and absolutely take any claims regarding the performance of a tool with a massive grain of salt, and try them for yourself to see if they suit your needs. Octobot's not designed to replace background processing in most applications as tools like Celery and DelayedJob (both of which I use myself) are great, a bit easier to write for, and a bit simpler to get up and running, depending on the application and language being used.

There's no intention to have slighted anyone or any other project here. I'm stoked that a lot of tools exist in this space. I just hadn't seen one on the JVM that offered this level of simplicity and parallelism with a bit of restraint when it comes to feature creep. But if you have an application that demands high throughput / low latency execution of tasks, this might be worth evaluating.

Unfortunately, the iOS platform does not support the Objective C garbage collector available in Mac OS. Instead, memory is controlled using manual allocation and autorelease pools. So, while the TOS would permit this, a technical limit remains for the time being.

To the good folks at 10Gen, Antirez, the Cassandra project, LinkedIn, Google, Amazon, and everyone working to advance the state of datastores available for increasingly specific applications, your work is undeniably important. The team I work with has been evaluating a handful of emerging datastores for some applications and are making plans to migrate a few types of data in some of our systems from one to another. Thank God that we have choice beyond a standard RDBMS and BDB.

To the armies of bloggers parroting slogan after slogan, ricing benchmarks so far removed from real-world applications as to place themselves somewhere on the spectrum between meaningless and malicious in the name of pageviews, well, that's not helping anyone.

It's great to see people get excited about new technologies. But if tech/startup culture is one that embraces and celebrates the fail, then we damn well better also talk about the different areas where many different datastores are not appropriate choices, and where some of them downright break, risking extensive downtime or corruption, require unreasonable amounts memory for indexes, or multiple machines to provide for durability, or return inconsistent "postmodern" results, or what have you -- without it being understood as a personal attack on any one individual, company, or sector.

The productivity of this video is debatable, sure. But perhaps we can appreciate that it parodies the "magic scaling sauce" image that the "tech press" (face it, we're it) has given surprisingly young but maturing technologies. There's no magic sauce, no drop-in answer to "web scale," and certainly nothing fraught with difficulty. It's data. If you have a ton of it and it's valuable, it's worth mountains of expensive programmer time and effort to ensure its integrity, accessibility, and utility based on the storage and query requirements of an application.

It really is a shame that so much of the Android marketing has taken its cue from Verizon's techno-masculine Droid branding.

Google was dead on with the Nexus One's "Web Meets Phone" ad. If you haven't seen it, it's worth watching - right at the center of Google's brief expressive / cute / minimalist advertising phase. Think back to the "Parisian Love" Super Bowl ad and the Motion Theory "Google Chrome" short. They're delightful.

These shorts present Google as a very different company - one that's personal, bright, and helpful. Much of this work is coming from (or commissioned by) a division called Creative Lab that's largely independent from the majority of Google's culture and management structure. I had the pleasure of collaborating with them on a couple projects and was amazed by their creativity, insight, and lighthearted approach.

I would love to see more work in this style from Google. They've demonstrated they're perfectly capable of creating very good ads that deliver a consistently solid impression, but have ceded much of the opportunity to brand Android to carriers who have screwed it up to the nines (though we shouldn't forget that the platform is actually called "Android"), with the lone exception of HTC's highly personal commercials.

I hope they can turn it around. If and when they do, I further hope that it will be a part of a more comprehensive strategy to rebrand Android as a whole. It's a solid platform with a lamentably robotic image.

---

[1] Nexus One ad: http://www.youtube.com/watch?v=I6COwgigJ-g

[2] Parisian Love: http://www.youtube.com/watch?v=nnsSUqgkDwU

[3] Motion Theory's "Chrome": http://vimeo.com/5721933

Apologies for not publishing the EC2 Small number in the original post. It was 120,000 for the pure Java NIO implementation, but I'd like to explain a bit more on that one.

We hit 120k clients at the time the process was killed by the kernel, but had about 500mb free memory left when the OOM kill occurred. Here's why (and please correct me if I'm mistaken - I'm not a kernel expert, but have done a lot of reading in this area over the past couple months):

Small EC2 instances are restricted to running a 32-bit kernel. The 32-bit Linux kernel (2.6 series) allocates memory into three zones, with the first and smallest slot reserved for DMA (~16mb), the next portion reserved for kernel functions ("low memory", ~896mb), and the rest allocated as "high memory" for userspace. There are very limited options to configure this allocation, aside from recompiling and maintaining a custom kernel for our purposes, which we do not want to do. The Hugemem kernel allocates low memory differently, but is no longer being actively recommended (that I can see), and only makes sense for servers with > 4GB anyway.

Because TCP sockets are allocated in low memory, and its size is relatively fixed, 120k sockets will exhaust low memory despite about 500mb of high memory being free and unallocated. At this point, the kernel has no memory left allocated to itself to do work, so the OOM killer steps in and shoots the process.

The 64-bit kernel makes no distinction between "low" and "high" memory, and does not suffer this problem. We switched to an EC2 Large instance in order to avoid this limitation and properly test the service. In the end, if Amazon were to offer a 64-bit Small instance, it would be ideal for our application and would push our cost per client even lower, though it's currently at a very acceptable spot.

While the metrics in this post focus on connections per node, our ultimate goal involves both maximizing the number of connections per instance, in addition to minimizing the number of instances required (essentially, we want to [safely] maximize density across the board). The sentence you quote refers to this cumulative goal rather than the particular comparison of implementations and instance types. I should've been more clear.

Anyhow, pardon this omission from the original post. At some point, I might write a bit more about the TCP/kernel-level issues we ran into if anyone's interested.

Notice Stallman's revised comment at the head of the article:

"Since this article was first published, Sun has relicensed most of its Java platform reference implementation under the GNU General Public License, and there is now a free development environment for Java. Thus, the Java language as such is no longer a trap."

This case is among few high-profile instances in which the GPL may not be free enough -- that is, Oracle may have a strong claim despite Sun's licensing of much of the JVM and JDK under the GPL, because the provisions of the license only extend to VMs which fully implement the Java spec. The license grant for the Java specification explicitly does not apply to VMs implementing a sub- or super-set.

See the patent grant released in Dec 2006 [1] – specifically the portion where it's stated that the implementor must "not modify, subset, superset or otherwise extend the Licensor Name Space," and that the VM must "[pass] the Technology Compatibility Kit." Under this interpretation, Google's App Engine platform might also be considered non-compliant, depending on the underlying VM.

While I personally choose to release code under BSD or MIT-style licenses to avoid the GPL's viral restrictions, it is interesting to consider cases where liberal licensing alone is not enough to ensure the liberty of a platform.

[1] http://java.sun.com/javase/6/jdk-6u21-doc-license.txt

The biggest issue I see with Oracle's lawsuit is not with Android itself, but the future of Java as an open platform.

The claims are pretty serious, and Oracle is going straight for the jugular. It'll be interesting to see how it shakes out, but I've got to hope that Google will come out of this in good shape for the sake of Java. It's unfortunate that Oracle is interpreting Google's implementation of Dalvik and a Java-based system as a direct infringement upon their patents.

Android aside, it raises some fairly serious questions around Java's future development as a platform vis a vis the uncertainty recently resolved between Microsoft and the Mono project. While Microsoft extended their "Community Promise" to Mono implementors and users, Oracle seems to be taking the opposite approach to companies developing alternate JVMs and Java-based devices. It'll be interesting to see Oracle's stance toward other alternate JVMs such as IBM's.

If you're curious, the original complaint is here: http://www.scribd.com/doc/35811761/Oracle-s-complaint-agains...

The patents upon which Oracle claims infringement are:

- Protection domains to provide security in a computer system (6,125,447)

- Controlling Access to a Resource (6,192,476)

- Method and apparatus for pre-processing and packaging class files (5,966,702)

- System and method for dynamic preloading of classes through memory space cloning of a master runtime system process (7,426,720)

- Interpreting functions utilizing a hybrid of virtual and native machine instructions (6,910,205)

- Method and system for performing static initialization (6,061,520)

Apologies, Thomas - wasn't attempting to hop on the FUD box, proclaim doom, or voice an opinion about the relative security of one platform to another.

I intended the comment to bring a few possibilities to the surface that I've not seen raised elsewhere surrounding an exploit which is otherwise being received by many in the community as a Good Thing.

Thanks for pointing out the error in the (original) write-up I'd linked. Yesterday I found myself telling a friend I thought it unfortunate to see others hopping blame upon Adobe for something they clearly had nothing to do with. If you're aware of a more detailed link to a writeup on the vulnerability in the font file processing, I expect people might be interested to see it.