Digital Ocean: Ubuntu, Nginx, Unicorn, Rails

http://blog.mccartie.com/2014/08/28/digital-ocean.html

by jonmccartie • 12 years ago

53 48 12 years ago

blog.mccartie.com

meritt• 12 years ago

Great writeup. One suggestion on the nginx front I might suggest is you add an entry to drop requests for unknown hosts. e.g. http://104.131.41.220

https://github.com/h5bp/server-configs-nginx/blob/master/sit...

That github repo is a goldmine for understanding nginx configs too.

jonmccartie10• 12 years ago

Good call. Thanks, man.

latch2• 12 years ago

other things to consider:

     # if you compiled --with-http_spdy_module
     listen 443 ssl spdy;

     # amusing how many of these you'll get
     location ~*\.php {
       add_header "Not Found" 404;
     }

     # pretty sure you need to do this to have keepalive working
     # between nginx and the upstream
     proxy_set_header Connection "";
     proxy_http_version 1.1;

     # buffer writes to disk (for a busy site, you can use much larger values than 1K)
     access_log /var/log/nginx/access.log buffer=1K;

     # cache the ssl connction parameters
     ssl_session_cache shared:SSL:20m;
     ssl_session_timeout 10m;

jonmccartie10• 12 years ago

The .php block gave me a good laugh. Great idea. Just rejecting /wp-admin.php should reduce load significantly. :)

thejerz• 12 years ago

I wouldn't recommend running Nginx, Unicorn, Rails, Redis, and PostgreSQL all on one instance. Better to offload the databases onto their own VPS's.

jonmccartie10• 12 years ago

Agreed. This is just a temporary solution for a small-ish app. Looking forward to solving those problems...

threeseed2• 12 years ago

Even better idea is not to host the databases at all.

Only those who have never experienced a corrupted backup or failed slaves think a database is something that is relatively trivial to manage.

You're much better off looking at platforms like RDS, MongoHQ, Cloudant etc.

sespindola• 12 years ago

In my experience, hosted database providers are almost always either cost or latency prohibitive.

Nowadays, in PostgreSQL is quite simple to replicate a db via log shipping[1]. You can even stream the WAL to an S3 bucket.

[1] http://www.postgresql.org/docs/9.3/static/warm-standby.html

davidlumley4• 12 years ago

I used to prefer the flexibility and efficiency of having my VPS setup, and it's certainly cheaper. Over the past 2 years though, I've saved so much time and sanity using Heroku.

I certainly can understand saving thousands of dollars and improving performance by moving from Heroku to a VPS or bare metal solution, but $90 is not uncomfortable enough for me to warrant the change.

Interesting article!

jonmccartie10• 12 years ago

Thanks, David. We use Heroku a TON at work and I absolutely love them. For my small-ish app, I just couldn't afford the extra cash to pay Heroku to manage my app. Thanks for reading!

davidlumley4• 12 years ago

Ah, that makes a lot more sense! I quickly skimmed through SproutMark and it seemed great so assumed it was a full time job.

brettskiii2• 12 years ago

Heroku apps aren't performant in many areas of the world e.g Australia like where I am, Amazon EC2 has an Australian region which is amazing

davidlumley4• 12 years ago

I'm also Australian, but we have a fairly international customer base so the ~150ms difference for requests that hit the application hasn't been that big for us in the scheme of things.

donw6• 12 years ago

Question -- do you have any background as a sysadmin? If not, then I can totally understand the attractiveness of something like Heroku. It really does take care of a lot of nastiness.

From my side, I've got a fantastic Ansible setup for databases, security, ssl certificate distribution, and so on, and so Heroku doesn't really buy me anything for the extra money.

davidlumley4• 12 years ago

None as a fulltime sysadmin, most of my experience has been in operations, and in configuring servers for my own projects.

I've used Puppet and Chef before, haven't checked out Ansible. My main dissatisfaction with those tools was that there wasn't a super easy way to get started on a new VPS. I had to configure roles and settings to install Puppet or Chef, and _then_ I could get started automating things. It's certainly not hard, but I've never been in a position where it was something I did full time so have really valued platforms that do a lot of this for me.

akbar5012• 12 years ago

Also worth noting that you should run a firewall as part of the basic configuration. AWS includes this via the Security Groups, but with DO you'll need to use iptables or ifw.

jonmccartie10• 12 years ago

Good point. I didn't mention it in this post, but DO has a great article on getting started with firewalls: https://www.digitalocean.com/community/tutorials/how-to-set-...

donw6• 12 years ago

Or `ufw` if you're on Ubuntu -- very easy to set up, much easier than crafting rules by hand.

I'm also a fan of running `sshd` on an off-numbered port to add another layer of protection against zero-day attacks. Most worms spread by compromising a service on a host, and then hitting everything around that host, but (to my knowledge) most of these depend on targeted services living on their default ports.

It won't buy you anything against a direct attack, but security is all about layers of defense, not just having a hard outer shell.

MaxGabriel• 12 years ago

I think this is a typo; "deafult" instead of "default"

    nano /etc/nginx/sites-enabled/deafult

jonmccartie10• 12 years ago

Ack! Good catch! Fixed!

xtrumanx2• 12 years ago

Quick question; I don't know much about setting up a Linux server but I found it interesting the post had nothing related to security besides setting up a ssh key and separate user for deployment.

What security-related tasks do you do when setting up a new server? Besides the above, the only things that come to mind for me are:

1. Change ssh port from default. 2. Block unwanted traffic via iptables. 3. Protect ssh with fail2ban.

latch2• 12 years ago

#2 and #3 in your list are rather sweeping (fail2ban does more than protect ssh). I'm no expert either, the only thing I'd add is to disable password-based logins and root-login [1]

[1] http://www.unixlore.net/articles/five-minutes-to-more-secure...

yeukhon• 12 years ago

I personally have a mixed feeling about disabling password-based login. What if you've lot your key and you need to access the server for reasons such as getting the data out, how do you do it without root and without login password? Is there a way to go around this issue? Let me know if there is because right now, I have login password that I don't re-use. Well, I guess for real deployment, serving real users, you'd have shading and a good central logging system so losing an instance is not a big deal, but for me I run a personal server and this is so far the way I protect myself (have login and ssh) in the case of lossing my key :(

Anyhow, DO droplets allow people to "reset" root password and it is important to protect your DO account.

donw6• 12 years ago

Something that a lot of Ruby deployment setups seem to be missing is setting up proper user permissions for apps and deployment.

The first step is to have separate user accounts for everybody that's going to be deploying apps. You really don't want shared accounts (like a single 'deployer' user), because you get no audit trail on who-does-what, and you effectively lose access control to your machines.

The second step is to run the app itself as a limited-privilege user -- no write permissions to anything except for logs and tempfiles. A lot of attacks depend on your app being able to overwrite parts of its own code; if that can't happen, the attack fails.

e12e• 12 years ago

Most important: disable password login to ssh. Optional: disable direct login for root. Optional: limit hosts that can login with whitelist for sshd in hosts.allow (and ALL:ALL in hosts.deny). On Linux I generally don't lock down ports w/iptables - better to not have anything listening on a public interface that's not supposed to be publically available (most distros have sane defaults when set up as a server config these days).

As for DO upload your/a pubkey for root access to the control panel and have it installed for you in new images.

akbar5012• 12 years ago

Some very basic security:

1. Change SSH port

2. Block all password logins

3. Disable root login (namely, a non-root user must login via a key, then su or sudo to root)

4. Lock down everything with IPTables

5. Allow only encrypted connections to the few ports open via IPTables (normal exception is port 80 for http).

6. Segment the network (with DO private networking your machines are open to anyone within the same facility, so you need to isolate your private network from other DO customers)

7. Lock down each running service/daemon per it's standard configuration. For example, if you're running Postgres, then lock it down per pg best practices.

8. Create individual, non-privileged users for each service that you're starting. ex. If you're running an app server, then it should have its own user.

9. Sandbox everything that can be.

10. Turn off everything that is not being used, then uninstall it.

11. If you don't need them, then uninstall build tools such as compilers. Have a separate build machine.

12. Only open edge of network machines to the outside world. On DO, turn off eth0 on all machines that customers do not directly interact with.

13. All services that are internal user only should have public networking totally disabled. Ex. Redis should only run on a private IP, should have the public interface disabled, and should only be accessible via a secure connection (ideally with some type of true authentication, replay protection, etc. Or, just run these services on a trusted network and pray that nobody penetrates your network deep enough to find out that you're totally unprotected once inside your network.

14. Don't keep private keys/certs in places that they are not needed.

15. Have a map of each service, the user that it runs under, what resources that user has access to, and a general map of what sequence someone has to follow to control your network. If you catch someone mid-attack, you can start to lock off parts of your network.

16. Logging! And monitor them for attacks. And ship logs off the machine in question so that you can review logs even if a machine is compromised.

There's lots more, but this is a basic list that should be configured via Ansible/Chef/Puppet/Salt for each box.

One other item I've heard is that some people like to run heterogeneous networks.

xtrumanx2• 12 years ago

Thank you so much for all that. I took a great introductory Linux course by the Linux Foundation over at edx[0] which really filled me in on a lot of stuff about Linux one misses out on when they learn on their own. Sadly, the subsequent Linux Foundation training courses are way out of my price range. Even the intro course offered I took for free on edx is offered by the Linux Foundation for thousands of dollars.

The good thing is I can mostly learn stuff on my own but have a hard time figuring out what I should be learning. Your comment helps a lot on that department. Even if I don't quite understand how to do some of the stuff your saying e.g. segment the network? Sandbox? Isn't eth0 my primary port to connect to my server? Which logs should I be monitoring (I've only heard of auth.log)?

Thanks.

[0] https://www.edx.org/course/linuxfoundationx/linuxfoundationx...

dmourati• 12 years ago

I find the references to the nano text editor endearing.

jonmccartie10• 12 years ago

LOL! It was intentional. :)

locusm• 12 years ago

What about backup, check out Duplicity for backing up to S3. Rackspace has quite a nice built in tool for backups too if you ever leave your current setup. Duplicity http://duplicity.nongnu.org/

michaelbuckbee• 12 years ago

While I appreciate the time and effort put into both this hosting move (and the subsequent writeup), I really can't help but feel the time and effort would have been better spent on gaining more traffic and users than on the move.

OP is saving $90/mo, less than the cost of adding just one new monthly subscriber at his 'Premium' plan.

jonmccartie10• 12 years ago

It didn't take long ... and I wanted to take a break from staring at my non-performant Facebook ads and making cold calls.

frik2• 12 years ago

I heard Ansible over SSH is good to automate the installation. Has some experience with it? I am interested in a good tutorial similar to OP's article.

elithrar• 12 years ago

Have you looked at these? https://www.digitalocean.com/community/tutorials/how-to-crea... + https://www.digitalocean.com/community/tutorials/how-to-inst...

castell• 12 years ago

Thanks for sharing! Great that DigitalOcean has many useful tutorials.

The tutorial doesn't touch the "replace" command (http://docs.ansible.com/replace_module.html), that seem to be useful to modifying existing config files.

buttermint• 12 years ago

Would love to see a part two of this, for when you app gets popular... covering three simple (but perplexing for beginners) things:

1) dedicated database server 2) two application servers 3) git push to multiple application servers

jonmccartie10• 12 years ago

Great idea, thanks!

tovmeod• 12 years ago

sudo apt-get install nginx

no, the ubuntu repository is outdated, you should add the nginx team ppa:

sudo add-apt-repository ppa:nginx/stable

now you can 'sudo apt-get install nginx'

xxdesmus• 12 years ago

Another potential repo:

https://launchpad.net/~chris-lea/+archive/ubuntu/nginx-devel

Digital Ocean: Ubuntu, Nginx, Unicorn, Rails

Related Stories