divideby0

While this is clearly xenophobic and just plain awful, it seems like the most sensible workaround for this is for universities to offer very small (4 students or less per session) in-person classes specifically for their new international students. This could work essentially like office hours with faculty leading small group discussions to help acclimate international students to their new lives abroad. I can imagine anyone moving to the US in the year 2020 (the peak of our crazy) could use some emotional support.

I'm not quite certain I understand that last part. To the best of my understanding, Kafka's design is such that consumer ingestion is completely decoupled from production throughput. All messages from the producer are first copied to disk and then zero-copied from disk to network to the consumers at the byte array level. If a consumer falls behind, it has its own independent offset stored on the broker that keeps track of where in the byte array (or log) it left off. This, by design, allows Kafka to handle different profiles of consumers and even have consumers drop off entirely and later join to catch up. But perhaps I'm missing something about what you're saying.

That works for some use cases, but for others (Elasticsearch, Zookeeper, Kafka, etc) the service inside the container needs to bind to an interface associated with an IP that's also addressable by the host. Even in host networking mode, eth0 inside a DFM-powered container will bound something like 192.168.x.y but that 192.168.x.0 subnet is completely inaccessible from the host.

I believe it's called a Sankey diagram:

http://bl.ocks.org/d3noob/5028304

us-east-1 is where they typically deploy new features/hardware first (with the exception of efs which went to us-west first for some reason). it's also by far the largest region, with the most tenants and the heaviest traffic, so it's approaching the limit on what's physically possible to do in a public data center.

Sign-ins to AWS console also appear to be timing out:

https://www.evernote.com/l/ABkKLgp3RjRDe5uV4pMlyVg1uzkW41DG4...

that's completely untrue. there are many ways to do fault-tolerance in AWS. it's expensive, but it's possible. netflix even goes as far as simulating the failure of entire aws regions in their simian army testing suite:

http://techblog.netflix.com/2011/07/netflix-simian-army.html

That's why Netflix stays up when us-east or us-west are down.

In addition to some of the other replies, Consul also has a writeup on their website comparing itself with Zookeeper, etcd, etc:

https://consul.io/intro/vs/zookeeper.html

So one of the challenges in a containerized environment is that services start up on random ports. If I'm running a Postgres container with the default Docker networking mode, for example, the internal port of 5432 may be bound to the host port of 12345. This allows me to spin up multiple instances of Postgres on the same machine for greater service density.

However, in a distributed environment, services can spin up on different machines. The instance of Postgres my application needs could be on server1:12345 or server2:23456. But in a distributed system, you need a cross-cutting service that's available to all servers so that if my app is running on server1, it can find the right Postgres instance running on server2.

I'm not an expert on etcd, but my understanding is that the most common use case is to run etcd on each host machine. When services start up, their supervisor registers the service's hostname, port, etc with etcd's key-value store. This registration is then propagated to other etcd nodes in a consistent manner, using a consensus algorithm called Raft:

http://thesecretlivesofdata.com/raft/

Consensus actually turns out to be one of the harder problems in a distributed system design. If I have a network partition that prevents etcd instances from seeing each other, you don't want one instance reporting incorrect or stale data. Otherwise, my application could be writing to the wrong service, causing data loss.

Etcd does consensus extremely well, and in a way that scales to support hundreds of nodes. It's one of the two distributed systems I'm aware of that have (mostly) passed Jepsen testing:

https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul

There are also alternatives like Consul and Zookeeper, but in the case of Zookeeper you have to do a lot of the heavy lifting yourself to support service discovery. There are also some well-documented caveats:

https://tech.knewton.com/blog/2014/12/eureka-shouldnt-use-zo...

Consul also has a pretty fair writeup (IMHO) on the tradeoffs of each solution on their website:

https://consul.io/intro/vs/zookeeper.html

Charlie Nutter gave a great talk covering this and a lot of additional new JVM features here:

https://vimeo.com/114187541

Edit: The JNR stuff is covered around 42 minutes in.

This seems quite cool. For very large clusters, I'd also consider looking into Optaplanner, which exposes a variety of probabilistic metaheuristics for balancing. There's a "cloud balancing" example in the documentation which is fairly close in terms of the use case: https://docs.jboss.org/drools/release/6.0.0.Beta1/optaplanne...

There's a great scene from the West Wing about this:

https://www.youtube.com/watch?v=vVX-PrBRtTY

Spantree Technology Group, LLC | Senior Software Engineer and/or Devops Specialist | Chicago, IL; Grand Rapids, MI | Onsite; Remote | Full-Time | Citizen | Elasticsearch, Groovy, Docker, Mesos/Marathon, Drools/Optaplanner, Angular/Backbone/React, D3.js, Clojure/ClojureScript, Grails, Spring Boot, Puppet, Spark, Kafka

Spantree Technology Group is looking for a new member to join our existing US-based team members in Grand Rapids or Chicago. We'd also be interested in a remote team member (preferably based in the Midwest or East Coast US) depending on expertise.

---

Hello, we’re Spantree, a rapidly growing boutique software engineering consultancy headquartered in West Loop of Chicago. We’re looking for a brilliant and personable engineer to join our team. Our clients range from small and scrappy startups to large banks and insurance companies. While we have a pretty broad technical focus, a key element to most of the stuff we do is building tools to help people make better decisions.

Our technical role on projects varies a bit depending on the customer, so we’re hiring organically for generalists that can be flexible up and down the stack. Though if you have deep expertise in some of the technologies listed above (esp Elasticsearch or Docker/Mesos), we can teach you the rest on an as-needed basis.

We offer competitive compensation, relatively flexible work schedules, health/vision/dental, 401K matching and the usual tech company perks (a copious amount of catered lunches in our Chicago offices, etc).

To find out more about who we are, please feel free to visit our website at http://www.spantree.net and reach out via jobs@spantree.net.

The freak fest thing is true:

http://gawker.com/man-unwittingly-rents-out-apartment-on-air...

Whether or not this is the same person or an imposter? Who knows. But it wouldn't be surprising.

If true, it's entirely possible that Airbnb is more concerned about setting a precedent for any and all auxiliary damages that go along with a bad stay. What happens when someone gets kicked out of their apartment because a guest accidentally tipped off management that they were an Airbnb renter?

You can do Docker with VMWare using Docker Machine:

docker-machine create -d vmwarefusion --vmwarefusion-boot2docker-url https://github.com/cloudnativeapps/boot2docker/releases/down... my-docker-machine --vmwarefusion-memory 4096 --vmwarefusion-disk-size 30000

Ge0ffrey, any plans to release the source code on this optimization? It'd be great to have a writeup on how to actually run the tsp tests for this. It's sort of buried in the test suite at the moment.

Optaplanner is an incredibly useful, often underrated framework. Keep up the good work!

This is a great way to handle offline search, but as the article mentions, indexing is still a pretty heavy operation to perform on the client-side. I'm curious to see if it'd make sense to invert the index server-side, and pass the pre-inverted index along with the documents via REST when an app loads. Then, you could still perform offline queries, but the client would only be responsible tokenizing/analyzing the search terms, comparing them against the index and doing the BM25 scoring piece.

Looks pretty awesome, esp the clean DSL for your page model, but it seems like most of the documentation might be missing. How sophisticated is the crawler portion? Does it support Nutch-style generators that crawl more frequently updated pages more frequently? Or is it more designed for focused, one-off crawls a la Scrapy?

Kibana 4 should have some of that:

http://www.elasticsearch.org/blog/kibana-4-beta-1-released/

It exposes quite a bit of the new aggregation functionality added to ES 1.x.

You forgot to mention its also home to Strangeloop, one of the best developer conferences I've ever attended: https://thestrangeloop.com/

Full disclosure: I'm hosting a workshop at Strangeloop this year.

Groovy has supported compile-time safety since version 2.0 with the optional @CompileStatic annotation. It's one of the few languages where you can mix static and dynamic typing, even within the same class. With this annotation enabled, you get pretty close to pure Java performance but with 1/5 the lines of code in some case.

For something like Android, running Groovy with CompileStatic enabled by default would make a lot of sense.

This makes me wish for a downvote button.

Chicago, IL - DevOps and Build Automation Engineer @ Spantree - http://www.spantree.net

===

Systems Management: Vagrant, Puppet, AWS

Build Automation/CI: Gradle, Grunt, Jenkins, Sonar, Maven, Cordova

Test Automation: JUnit/Spock, Selenium, Mocha.js

Web Tier: Java 7, Nginx, Tomcat, Jetty, JBoss, Grails, Redis, RabbitMQ, Node.js, Varnish

Data stores: MongoDB, Elasticsearch, Postgres, Redis, Couchbase, Neo4J

===

Spantree is a boutique software engineering firm based (mostly) in Chicago. We're looking for our very first DevOps engineer to join our team in the West Loop.

This candidate should enjoy learning new technologies and solving hard problems. Ideally, we're looking for someone that can whip or systems engineering stack into shape, taking over cross-cutting systems work that we each handle on our individual projects. Our existing team of 7 are mostly programmers. We've had to learn things tools Vagrant, Puppet and AWS to make our lives easier. But we're growing to the point where we could really use someone who lives and breathes this stuff.

We work quite a bit with OSS and we're not shy about jumping into other people's code to figure things out. We constantly hunt for smarter ways to do things, so we're often experimenting with emerging tech with little or no documentation. You should be motivated to submit pull requests and hunt down project committers on Twitter and IRC when you're stuck.

===

Who are you?

===

* A nice person

* An effective communicator face-to-face, on the phone, and over the web

* A pragmatic engineer that can quickly go from problem to solution to working software

* A lazy programmer who leverages test, build, and deployment automation wherever possible

* An confident teacher of what you know and a humble student of what you don't

* An altruist who wants to participate in open-source projects

===

What do we work on?

===

Our clients range from small and scrappy startups to large enterprise companies. We have 3-7 projects going at any point in time. We tend to focus on greenfield development, building web applications from the ground up. We also work on integrating the old with the new, bringing technologies like Elasticsearch, Grails and Drools to large companies. Helping people make better decisions is the central theme of most of our projects. This can take the form of a search engine to help people find interesting stuff to do on a Saturday night, a complex rules-oriented workflow management system for evaluating health insurance claims, helping people find the right doctor, or a matchmaking and scheduling system to help people book face-to-face meetings at conferences.

In the next year or two, we also plan to get into product development, taking our experiences solving the same sort of problems for multiple clients and rolling them into a generic framework which we plan to open source and support directly for our clients and the rest of the world.

===

We offer full health, vision and dental for our full-time employees. 401K. Free snacks and often free lunches as well. Big monitors and top-of-the-line equipment. We also have a shiny new office treadmill.

Drop us a line at jobs@spantree.net and tell us a bit about yourself. If you have a resume or CV, feel free to pass that along as well. Github and LinkedIn profiles are also helpful. We're looking forward to hearing from you!

Spantree - http://www.spantree.net/jobs - Chicago, IL

We're Spantree, a small, nimble boutique consultancy based mostly in Chicago. We're looking for another smart person to join our team in the West Loop of Chicago. This candidate should enjoy learning new things and solving hard problems.

We work almost exclusively with open source software and we're not shy about jumping into third-party code to figure things out. You should be motivated to submit pull requests and hunt down project committers on Twitter and IRC when you're stuck.

Some of our favorite day-to-day technologies include Groovy, Grails, Elasticsearch, Drools, MongoDB, Backbone.js/Marionette, CoffeeScript and Puppet.

Our clients range from small and scrappy startups to large hospital networks and software as a service providers. Historically, most of the stuff we've worked on has not been public-facing: it's either guarded behind the client's firewall, neatly tucked away as a library or jammed into the minds of the developers we've mentored. But we're starting to diversify from "behind-the-scenes" work to more early-stage product development.

Fundamentally, we like to work on tools that helping people make better decisions. This can take the form of a search engine to help people find interesting stuff to do on a Saturday night, a complex workflow management system for evaluating health insurance claims or a matchmaking and scheduling system to help people book face-to-face meetings at conferences.

We're looking for generalists who can quickly go from problem to working solution and communicate effectively with our peers and clients along the way. While we have a very strong technical stack already, you don't expect new hires to be an expert in everything right away. In general, though, we're looking for people with the following qualifications:

* One or more fun languages: Ruby, Python, Groovy, Clojure, PHP, CoffeeScript, etc.

* One or more boring languages: Java, C++, C#, etc.

* One or more web frameworks: Grails, Rails, Django, SpringMVC, Play, CakePHP, etc.

* One or more databases: SQL, NoSQL, NewSQL, we like it all.

* HTML, CSS, JavaScript and at least one framework

* Git, or a colorful excuse for not knowing it.

* The typical software development toolchain: text editors, IDEs, step debuggers, issue trackers, etc.

If you'd like to learn more, please email jobs@spantree.net to set up a Google Hangout.