zarvox

https://media.ccc.de/v/27c3-4087-en-console_hacking_2010

It's substantially less risky, less invasive, and easier to do in parallel with other ongoing feature development than a port to another language would be. It's much easier to get incremental value for the investment if you're keeping the same language and codebase than if you're trying to replace everything all at once.

Not yet (still incomplete) but I'd be happy to drop you a line if/when that changes - send me an email?

I've recently been building an IRC bouncer and webapp with Actix, and it's been really smooth sailing so far -- excellent documentation, extensive examples, and everything I've touched so far has just worked the way you'd expect it to. It's a gem of a project.

My impression is that bus1 appears to have taken the (copious) feedback from the kdbus debacle and actually applied it and looked at other platforms' IPC to build a novel IPC system for Linux worth using. There's a talk [1] about the design of bus1, and comparison against IPC on other platforms where IPC is saner, and how the capability model is the right design for IPC - composable, understandable, and secure-by-default. It strikes me that the bus1 devs arrived at their design after doing the things you suggested! :)

Is there something I'm missing? What might an ideal IPC API look like to you?

[1] https://www.youtube.com/watch?v=6zN0b6BfgLY

Submitted this to Our Incredible Journey [1] which documents a long string of services shut down post-acquisition.

[1] http://ourincrediblejourney.tumblr.com/

For folks looking for escape rooms all over the world: take a look at http://escaperoomdirectory.com/

http://www.freedesktop.org/wiki/Software/systemd/TheCaseForT...

In short: because having things in /usr is equally compatible, and makes some useful things like atomically snapshotting /usr to snapshot all executables, mounting /usr readonly, etc. possible.

Relevant thread from the designers of Cap'n Proto and Flatbuffers from around the date of the Flatbuffers release: https://news.ycombinator.com/item?id=7901991

Perhaps, but even a single database alone doesn't ensure atomicity.

For instance: suppose you're deactivating a user. You've got a user table, with an id column and a "deactivated" column. Then you've got an OAuth tokens table, with a foreign key column to userid.

If you had two microservices hitting the same database, then you make a DELETE to the user service, which now needs to send a DELETE to the OAuth service. Now, regardless of which transaction you logically have go first, you have a race: two services with separate transactions are modifying the same DB. Whichever transaction commits, the other could fail, leaving your external view inconsistent.

One way to solve this is to have the OAuth service check the user table to see if the user is disabled, and treat all of the tokens for that user as deactivated, but then your OAuth service is tightly coupled to your user service's schema, which means you can no longer modify the two services separately. My impression is that this isn't really what people mean when they say "microservices".

Another option is to have the OAuth service ask the user service if the user is still live, but now you have a circular dependency between services, and either one failing can effectively put the other out of commission.

Tradeoffs in all things.

Indeed.

CRUD apps (which I'll argue are the vast majority of applications) are easier to write, easier to understand, and easier to make have the behavior users expect when there's only one datastore and you never have to deal with eventual consistency or distributed or half-applied migrations.

As an anecdote, at a previous employer, we provided user accounts and OAuth for connecting these accounts to our API. We made separate user and OAuth services, with separate databases.

What resulted was an unnecessary amount of complexity in coordinating user deactivation, listing OAuth tokens for users, delegation, and doing all that while authenticating requests between microservices. Our API could not safely expose a method to deactivate a user and all of their OAuth tokens in a single DELETE. A single instance that handled both would have been easier to build and wasted less time up front dealing with complexity that we didn't need or make good use of for our scale at the time.

To solve this, we eventually merged all the data back into a single database, so we could expose sane invariants at the API level without needing to build an eventually-consistent message queue.

There's also pastebinit, which is packaged by at least the major distros and supports multiple pastebins.

http://launchpad.net/pastebinit

The nicest things about error handling in this manner are that any function that has a error condition makes it clear in the function signature, and any time you call a function that can fail, it's visible in the source where you make the call.

This makes it very easy to tell whether code is handling potential failure conditions or ignoring them - you're either ignoring the return value (which is a compiler warning) or you're handling the failure condition. I'm reminded of Raymond Chen's post from 2005 about handling failure conditions, and recognizing code which handles all of its error conditions:

http://blogs.msdn.com/b/oldnewthing/archive/2005/01/14/35294...

I think this approach will pay great dividends in both understandability and readability.

I'm in the same boat - I wind up in a situation where I find myself missing the features, stability, coherance, and integration of systemd at least once a week in my day job, where we currently use Ubuntu with upstart.

For me, it's a combination of things:

1. Proper dependency management, where I specify dependencies as they are, rather than flattening the dependency graph.

2. Proper service supervision. I've had upstart lose track of running processes, which is silly - you had one job!

3. Ability to decouple the init system's view of "process is running" from "process is ready/live". This is nice if you want to await liveness by letting the system supervisor tell you if the service is live or not, rather than writing a bunch of external scripts or checks.

4. A single place to modify the way system services are run, and unified config for e.g. resource limits, and tools for working with them.

5. Simple exploration and interleaving of logs from various communicating services via journalctl, which is insanely helpful for debugging things in a distributed system. You can see the calls come in from the network and bounce between services, and see exactly where something goes wrong. You could do this with other solutions, but I get this out of the box with journalctl, and I can look in more detail at any of the services, or when I'm checking a service's status.

It's true that many of these things could perhaps have been done in other ways, but the fact is that the systemd developers did the work to make it happen and now I can focus on my product, rather than on learning a bazillion pieces of plumbing to enable me to ship product. I'm fine adapting a few things to run via unit files instead of shell scripts if it means I can ship more correct and more reliable product in less time.

Thanks for the detailed reply!

I agree that there are certainly a number of tradeoffs involved, and reading the design goals you linked, I see that (in particular) the desire to be able to run fwknop on OpenWRT and other particularly resource-constrained setups makes a strong argument for your implementation strategy.

Your comments on MITM and synchronization are well-received. It's clear that you take the security of the fwknop implementation seriously, and I apologize if I was overly negative. I should have read more about the design goals before criticizing the project so harshly.

For what it's worth, in place of my first comment, I should probably instead say that fwknop was not a good fit for my needs at the time, but that there certainly exist cases where it will be an excellent solution, and to watch out for a practical gotcha I learned the hard way. :)

Moxie's criticism of portknocking implementations [1] has several facets that apply to fwknop. I would not recommend using fwknop in general.

Unrelated to security, but practical experience from using fwknop in the past: port-knocking tools that depend on timestamped packets are a great way to get locked out of your server if you have time skew between server and client. EC2 instances will drift a lot over their lifetime, so if you're not running NTP you will likely eventually fall outside the portknocking implementation's grace period, and then be unable to access your server at all, short of rebooting it or figuring out how skewed the clock is. Not fun.

[1] http://www.thoughtcrime.org/software/knockknock/

Not terribly, at first glance. Crypto generally uses fields with no subfields and large characteristic, whereas these attacks seem to generally be limited to fields with subfields and small characteristic.

It's interesting work, but not immediately alarming.

I really like Mozilla's recommendations, which are detailed, explicit, and well-justified: https://wiki.mozilla.org/Security/Server_Side_TLS#Recommende...

Update from https://status.digitalocean.com/ :

  Networking Issue in NY1 Facility
  
  At this time, we are currently investigating an issue in our NY1 facility.
  
  We are working to determine the exact cause/resolution of
  the issue and will provide details as soon as they
  become available.

As of this posting, https://www.digitalocean.com/login is returning CloudFlare's "Always Online" cache.

I just got woken up by two of our boxes being unreachable, but one of my other boxes is up, so maybe it's just affecting one of their datacenters?

Cert pinning only makes sense if you happen to trust multiple CAs, but want to stick to the cert issued by one particular one. We only trust one CA, and each issued cert is bound to a user and device id, so this is a non-issue. :)

A proper writeup is in the works, but to cover the basics: we know not to implement our own crypto. :)

Passwords: we apply scrypt() before any use or storage. We never store the plaintext.

Device-to-device: standard PKI. We have a CA, and the CA's cert is bundled with the client software. Devices generate 2048-bit RSA keys at setup time. They then generate a PKCS10 CSR which our CA signs, provided you give a valid username/password. When peers wish to communicate, they establish a DTLS connection (we use OpenSSL's DTLS implementation, and AES-256-CBC as the default ciphersuite), verifying that the other device:

  * is certified by our CA to represent the claimed user and device (identity)
  * is not using a certificate with a revoked serial number
  * is trusted to send and receive information about the relevant shared folder (authorization)

Revocation: When you unlink or remote-wipe a device, we mark the certificate associated with that device as revoked, and notify each of your clients either immediately (if they're online) or as soon as they come online and reconnect to our push notification service that the revoked device is no longer to be trusted. (This is one of the other tasks that our servers provide - prompt delivery of device revocation information.)

We update our libraries promptly and are subscribed to the appropriate mailinglists.

Finally, if you believe you have discovered a vulnerability in some part of the AeroFS system, please contact us at security@aerofs.com (PGP key 6E1DC9F9, if you prefer encrypted email).

Yes, we use NAT traversal techniques where possible on the WAN, and direct TCP connections over the LAN.

Your data is never stored on our servers in an unencrypted form. Moreover, the data is never stored on our servers at all.

In some scenarios (when two clients are both behind aggressive firewalls, for instance) the data may be _relayed_ by our servers, but in those cases it is encrypted (end-to-end) between the devices syncing using their respective public/private keys, so we can't eavesdrop.

Ah, I understand completely; no worries. :)

Officially, we only support Ubuntu, but in general we like to have things work for any setup that's not too exotic. Indeed, some of the more helpful bug reports I've looked at and fixed have come from Arch users.

Thanks for the feedback anyway, and let us know if there's anything else we can do. :)

Hey, this is Drew from AeroFS. Sorry to hear you've been having trouble syncing. If you want to report an issue (AeroFS -> Help -> Report a Problem in the GUI, or aerofs-sh report "Description of your issue" from your terminal of choice), we will be happy to take a look at your issues and see what's going wrong.

I'm an engineer at AeroFS, and yep, we use scrypt. The client applies scrypt with a per-user salt to the S3 encryption passphrase you enter at install-time, which gives a 512-bit key that the client stores. This is not quite the AES key itself - we apply PBKDF2 an additional time to this stored value to derive the actual 256-bit AES key (it fit more easily into our existing codebase...), but it retains the same hardness of derivation that scrypt provides (unless the stored key is stolen, as is the case in any system with stored keys).

In this manner, the AeroFS client can (and does) access the data stored on S3 directly, but if you were to lose the machine that runs the client (but still know the password you used at setup), you can still decrypt those files by deriving the same key on a different machine.

Colin Percival's writings ([1] in particular) were very helpful in picking the appropriate set of cryptographic primitives to use. Thanks, cperciva! :D

[1] http://www.daemonology.net/blog/2009-06-11-cryptographic-rig...

(edited to add that I work at AeroFS)

Clay Shirky has studied this extensively and is probably the best resource. "A Group Is Its Own Worst Enemy" [0] is perhaps the best single piece on what you're thinking about.

He offers several design guidelines toward the end of the piece, which I suspect informed the design of HN (and the posting guidelines therein).

Another technique he mentions is that of MetaFilter, which disables their new user page when they feel the effects of scale.

[0] http://shirky.com/writings/group_enemy.html

Code 511 sounds really useful. If captive portals actually implemented this, it would help efforts like Convergence[1] to handle captive portals gracefully, rather than having to play a heuristic guessing game.

It'll be interesting to see if browser vendors and captive portal vendors both implement this, and what level of adoption they both reach.

[1] http://convergence.io/

Heh. The OS landscape has changed over time, as disks have gotten much faster and much larger, and some of the design decisions made decades ago no longer make sense.

Fedora is moving everything from /bin, /sbin, /lib, and /lib64 into /usr because maintaining this separation no longer really makes sense: http://www.freedesktop.org/wiki/Software/systemd/TheCaseForT...