New Year's Resolutions for SysAdmins

https://www.usenix.org/blog/new-years-resolutions-sysadmins

by zdw • 13 years ago

31 44 13 years ago

jaryd4• 13 years ago

I'd also recommend any sysadmin to start looking into an automation framework like Chef, Puppet, Ansible, or any other of the myriad options.

ams61105• 13 years ago

Absolutely. Automate relentlessly. As a bonus, the recipes/playbooks you develop serve as documentation for how things are configured.

jaryd4• 13 years ago

Yeah that's a huge bonus that I didn't anticipate before beginning. My cookbooks are essentially dynamic documentation that morph with my changes in system requirements.

sergiosgc3• 13 years ago

Better yet, version controlled changes to recipes are an excellent logbook for sysadmin teams.

protomyth4• 13 years ago

Automation is a fine thing except for some of this damn "enterprise software" that not only doesn't automate real well[1], but actually cannot be installed on VMWare[2] or even go to a hot swap machine[3]. Sysadmins always love government required software and stuff bought before we got there.

1) luckily backup seem to be reasonably automation friendly

2) "We do not support virtualization of any kind"

3) tied to hardware on one machine - f'n brilliant.

jaryd4• 13 years ago

You might find this interesting: http://www.slideshare.net/opscode/chef-vmwarewebinarslidesse...

I personally haven't tried building environments on VMWare as I run pretty much everything in AWS. However, this presentation looks promising.

WestCoastJustin• 13 years ago

start looking into an automation framework

Cannot stress this enough! There is a sea change happening and sysadmins run the risk of being swept away unless they learn about automation (puppet/chef) software and cloud infrastructure services (openstack/aws). My new years resolution is to release screencasts on these very subjects. I have already released one on Learning Puppet with Vagrant @ http://sysadmincasts.com/episodes/8-learning-puppet-with-vag... if anyone is interested.

nikhilji• 13 years ago

Good stuff! Thanks!

gmjosack• 13 years ago

I don't know if you're aware, but configuration management is not a new concept. CFEngine was started in '93. CFEngine2 which is popular among many sysadmins came out in '98. While configuration management has definitely had some improvements it's not THAT far from what we've been using for decades now.

pault• 13 years ago

I just started digging into chef with the intention of using with AWS opsworks and... holy complexity. There just doesn't seem to be any obvious entry point, as far as I can tell. I've spent two entire days searching and there doesn't seem to be anything in between "hello wordpress" and "read this 300+ page user manual". Can anyone recommend a hands on guide for setting up a multi-node stack with opsworks that isn't just plain vanilla templates? I was really impressed by Richard Seroter's videos on AWS at pluralsight, but they are more of a 30,000 foot overview and gloss over implementation details, for the most part (even at 10+ hours). The project I'm working on is mostly a pretty straightforward node service running on postgres, but it uses some custom services that need to scale horizontally, and host sensitive data that should probably be running in a VPC.

Edit: I'm also having a difficult time trying to figure out how to get up to speed with chef in the specific context of opsworks, since most of the chef-specific courseware I can find assumes I'm using a hosted chef server, and I'm not quite sure how that overlaps with the opsworks flavor. It's starting to feel like I'll be at this for another week before I can get even a simple stack running.

mateuszf• 13 years ago

Try with ansible.

blueskin_2• 13 years ago

Chef is a bad bet if you don't want unnecessary complexity - it's practically designed to be unnecessarily complex.

talkingquickly• 13 years ago

Not aimed at exactly this use case but the first two thirds of my book Reliably Deploying Rails Applications https://leanpub.com/deploying_rails_applications is heavily chef focussed.

It's intended for use with Chef Solo but potentially the early sections on writing custom chef cookbooks may be of use.

jnsaff2• 13 years ago

We just built a simple tool that generates Ansible playbooks for you. Try it out at https://cmify.com

vidarh2• 13 years ago

It's what always kept me away from these systems. So many of them are hopelessly over-engineered to the point where it's often easier to write custom solutions.

sergiosgc3• 13 years ago

Chef smells over engineered from the get go. Try salt stack, or ansible. I've settled with Salt, but I believe they are both equally good solutions. Puppet is also OK, even if we moved away from it (towards Salt) because of poor performance (on a deployment of about 100 physical servers plus the VMs running on top of them).

pchander• 13 years ago

Excuse my ignorance, but what's wrong with nslookup and ifconfig?

jaryd4• 13 years ago

short answer: `/bin/ip` replaces `ifconfig` -- it's newer and more powerful

more info: http://www.tty1.net/blog/2010/ifconfig-ip-comparison_en.html

nailer• 13 years ago

Asides from ifconfig not being maintained (which is reason enough not to use it), I always wondered specifically what was broken with it.

Then I worked for an arbitrage desk at an investment bank. They used virtual addresses for different IPs, on top of vlans connected to different exchanges, redundantly (ie bonded).

Not a single IP-having interface appeared using ifconfig.

adamnemecek• 13 years ago

OT but ip is a really unfortunate name that will be hard to google for.

ams61105• 13 years ago

They're a bit dated. Use dig instead of nslookup, and right from the ifconfig man page:

NOTE

This program is obsolete! For replacement check ip addr and ip link. For statistics use ip -s link.

protomyth4• 13 years ago

This is Linux specific advice for ifconfig, correct?

blhack2• 13 years ago

Where does it say that?

http://www.openbsd.org/cgi-bin/man.cgi?query=ifconfig

Pedantry, but keep in mind that there isn't really "the" man page for this.

Ubuntu might want to deprecate ifconfig, but that doesn't mean everyone does.

blhack2• 13 years ago

Nothing.

zdw2• 13 years ago

They've been replaced with ipadm/dladm: http://omnios.omniti.com/wiki.php/GeneralAdministration

And dnsip/dnsipq/dnsmx/dnsname/dnstxt: http://cr.yp.to/djbdns/tools.html

teddyh2• 13 years ago

What? Your reply seems purposefully disingenuous and misleading. The ipadm and dladm commands seem to be specific to Solaris and derivatives. And I can not believe that you are seriously proposing that the new standard commands for doing DNS is to use DJB’s tools.

What the original post most likely referred to is to use ip instead of ifconfig, and to use dig instead of nslookup.

petejansson• 13 years ago

On nslookup, in particular, the program has some ambiguous behavior and can be problematic when trying to debug certain DNS issues. Dig was developed to be a reliable replacement that has predictable behavior at all times. There is a companion to dig called "host" that is a simple lookup, including looking up in-addr.arpa entries ("host 8.8.8.8"). In most cases, you will get more consistent results by using dig and host, which is why nslookup is deprecated.

ams61105• 13 years ago

3. "Check that your backups are working the way you think they are."

3.1 "Check that you can actually restore a file/recover a database"

protomyth4• 13 years ago

3.1.1 "Do it daily to a box in another building while dumping the backup to offsite storage"

Ecio78• 13 years ago

I should "Finally learn IPv6" but I know that I will procrastinate again and again..

AmVess• 13 years ago

I learned #2 very early on. Further, the specs I give for a project are the very minimum. If you aren't willing to do things correctly the first time, I'm not willing to be a client.

reitzensteinm2• 13 years ago

"Check that your backups are working the way you think they are."

I'm a bit horrified to read this here. If you're a sysadmin and don't have both automated and manual testing of backups, it's hard to imagine what else was a more important use of your time.

There aren't many things that could bankrupt a healthy business overnight, but catastrophic data loss is certainly one of them.

An analogous entry for a lawyer might be to pay attention while reading contracts.

bowlofpetunias• 13 years ago

what else was a more important use of your time

Keeping shit running now is always more important than keeping shit running in case of catastrophic failure.

Because a catastrophic failure is only catastrophic if you actually have something of value in the first place.

In the real world it's a balancing act in which there is no room for absolutism. Choosing which compromises to make is the hardest part of any job that comes with a level of responsibility.

mattmanser• 13 years ago

Indeed.

Last place I worked with sysadmins, they were fire fighting like crazy, not enough of them, bad choice of suppliers, poor code bases meant things fell over and took far to long to run and a dev team begging to get off the internal network on to their own because of all the restrictions.

I needed a backup of the DB of one of the three systems to test a major setting change that no-one was 100% sure would work because no-one understood the bloody awful contractor code with the even more awfully designed DB.

Took them 2 weeks to get it to me.

Crazy place, quit after 2 months. Really thought I'd asked the right questions too.

reitzensteinm2• 13 years ago

I'm talking specifically about dedicated sys admins, since that's what the blog post is about; I'm no stranger to cowboy coding on a project that may or may not ever be worth anything. I've lost data once before to corrupted backups and I don't regret not investing more engineering effort. It was an MVP, and making better products gives me more leverage.

But by the time you're hiring full time for the position, outside of a few edge cases where maybe you're SnapChat and you are growing 20% a week, it's probably time to settle down a bit and be sensible. At which point, if testing your backups to completion to avoid catastrophic data loss isn't #1 on the todo list, it's #2.

blueskin_2• 13 years ago

#2 is the most important in that list.

New Year's Resolutions for SysAdmins

Related Stories