Command line tools for web developers

http://www.coderholic.com/invaluable-command-line-tools-for-web-developers/

by coderholic • 15 years ago

322 68 15 years ago

coderholic.com

gluecode• 15 years ago

siege is a very useful tool for benchmarking. I use it a lot.

ez773• 15 years ago

How come sending concurrent requests, as explained in the article, doesn't flag you as a DoS offender?

mnutt• 15 years ago

I was surprised that he ran siege against www.google.co.uk without some kind of "don't do this" disclaimer to new users. Running it against other people's websites is pretty poor form.

idonthack• 15 years ago

If you run it against a public site, that may be what happens. However it's intended for stressing your own sites in a test environment, in which case you have full control over the config. Setting up your test environment to block your stress tester as a DoS offender would just be silly.

rorrr2• 15 years ago

If it brings down the site, you might get into trouble. It's totally consider DoS.

dstein• 15 years ago

Does siege support benchmarking gzipped HTTP responses? This is something ab doesn't seem to do and it's starting to annoy me.

flexterra• 15 years ago

screen is another tool I use a lot, specially when connecting to remote servers over ssh.

trb• 15 years ago

Screen is a bit like a window manager for consoles. I use it to start multiple (software) servers without daemon mode. I can then switch between their outputs with screen.

Screen also detaches the console from your ssh session. In my case this means that the servers keep running if I loose my ssh connection to the (hardware) server.

It's a very handy tool and definitely belongs in the articles list of tools.

yuvadam• 15 years ago

    screen -d -m processName

is practically the easiest way to run and daemonize a process.

reinhardt• 15 years ago

screen is one of those things that's been on my todo list for way too long, along with its alternatives(?), tmux and byobu. Anyone that uses/used all three and can offer a comparison?

ez773• 15 years ago

Consider the more lightweight dtach and dvtm. Following a well-known tradition, they each do one thing only, and do it well ;-).

thristian• 15 years ago

byobu is something like a theme or packaged config for screen, so it doesn't really need to be treated separately.

The biggest difference between tmux and screen is that tmux is a lot more flexible with regard to laying out groups of sub-terminals within the main terminal. screen is mostly limited to one-at-a-time, horizontal splitting or (in very recent versions) vertical splitting, while tmux lets you go nuts: http://tmux.sourceforge.net/tmux3.png

Also, GNU screen is very old and stable, while tmux is (so far) still new and flexible. For example, tmux very quickly added support for handling Unicode characters beyond U+FFFF, a feat that (so far as I know) screen still can't manage. That's only one example, but I'm sure there'll be more as time goes on.

EDIT: One other thing that tmux does that makes it better than screen: when I start up a tmux session from within an X11 session, tmux clears the $DISPLAY variable so that processes running inside tmux don't try to connect to the original X server - which may very well have gone away at that point. It's a small thing, but incredibly annoying when it happens.

revscat• 15 years ago

I have used screen for many years and decided to finally give tmux a try about 2 months ago. I really enjoy it. It took some adjusting, but overall I like it better.

Things like this are largely due to personal preferences, though. All I can tell you is that I am happier with tmux than I was with screen. It is more modern, and the split screening capabilities are better.

Pistos2• 15 years ago

I switched from screen to tmux a while back. The key thing for me is that configuration and manipulations/operations in general are more user-friendly. Otherwise, in terms of the basic use cases, they are essentially the same.

mcantor• 15 years ago

I switched from screen to tmux about three months ago. It's easier to configure, does vertical splits, and is under more active development (apparently).

cosgroveb• 15 years ago

GNU screen is probably the most indispensable command-line tool I've learned since vi.

alinajaf• 15 years ago

(Sort of) in the same vein, I've recently started using xmonad as my window manager and so far it's a lot more comfortable than the Ubuntu default. You may need to learn a teeny but of Haskell to get up and running, but so far I've been OK copy-pasting from sample configs and muddling through.

chrishenn• 15 years ago

When I bought a new Macbook and wasn't able to run linux, XMonad was really the only thing I missed. No matter how much Apple focuses on it's interface and making it easy to use, it's still incredibly slow and unintuitive to me.

pyrhho• 15 years ago

Agreed using 'screen -x' is a trick that routinely blows minds. (and is occasionally even useful)

jodoherty• 15 years ago

I hate that the default escape key sequence clobbers over Ctrl-A though, so the first thing I always have to do when I log into a new server or account is this:

$ echo 'escape ^uU' > ~/.screenrc

Or I quickly start tearing my hair out and screaming profanities every time I try to do something.

pavel_lishin• 15 years ago

I prefer ` (backtick) as my escape key sequence for both screen and tmux.

If you do this, though, be careful what you paste - if it has backticks in it, screen will interpret them as escape characters.

DevX101• 15 years ago

Are there any graphical interfaces for load testing, in lieu of siege?

pbogdan• 15 years ago

You might want to have a look at JMeter (http://jakarta.apache.org/jmeter/).

hasenj• 15 years ago

`curl ifconfig.me` is pretty awesome

But I didn't find anything else to be interesting.

ams6110• 15 years ago

ngrep was news to me. I think I've heard of it once before, recently, but didn't really realize what it was. Having not tried it, it sounds like a really nice option when something like Wireshark or Ethereal would be overkill or just too much effort to bother with.

cultureulterior• 15 years ago

http://freshmeat.net/projects/apachetop/

I find it amazingly useful.

Pewpewarrows• 15 years ago

I prefer curl'ing icanhazip.com, and I make it a quick alias in my .bashrc:

    alias myip="curl icanhazip.com"

Apachebench (ab) is a decent alternative to siege, and knowing how to use tcpdump and netcat comes in handy for debugging. Other than that, my new favorite command line tool over the past several months has to be vagrant, which lets you script and streamline VM creation and builds from the command line. If I need to completely reproduce my production environment on a test box, it's my utility of choice.

RyanKearney• 15 years ago

That only returned my IPv6 address so I can't get my IPv4 address with that.

madars• 15 years ago

Try ipv4.icanhazip.com and ipv6.icanhazip.com .

LiveTheDream5• 15 years ago

I use http://whatismyip.org/ frequently.

mwill• 15 years ago

Theres also http://ifconfig.me/ which acts similarly. The page has a list of curl commands you can issue and what your output will be.

tecoholic• 15 years ago

Coderholic.com is Timing out, Anyone else having issues?

sarp• 15 years ago

Database Error

brs• 15 years ago

Perhaps everyone is trying out siege.

LiveTheDream5• 15 years ago

Works for me, but here's a cache just in case: http://liveweb.archive.org/http://www.coderholic.com/invalua...

perlgeek• 15 years ago

curl -I and wget -S are particularly helpful when debugging redirects.

Sometimes I migrate URL schemes, and set up permanent redirects in my .htaccess files. Testing them in a browser is a real pain, because browsers cache the redirect (which is the point of having a permanent redirect), so even if you change the .htaccess, you still get the old response. And pressing the refresh button is no help, because that reloads the destination page, not the source of the redirect.

That's when a non-caching command line client saves your day.

vld• 15 years ago

also useful:

wget post-file.it --post-file /path/to/file

dexen• 15 years ago

lftp is a practical SFTP, FTPS and FTP transfer program, including automatic upload/download resumption and synchronization (mirror) mode. Good for both interactive use and scripting.

nikcub• 15 years ago

lftp can also continue downloads and run multiple concurrent download threads, acting as a download accelerator.

for eg.

  lftp :~> pget -c -n 10 http://warez.com/large_file.mp4

will start 10 concurrent download processes (n) and continue (c). lftp is full of hidden gems.

LiveTheDream5• 15 years ago

Synchronization (mirror) mode is absolutely invaluable when you only have access to a machine via FTP.

NathanKP2• 15 years ago

For me the most invaluable command line tool that I have ever used is S3 Tools:

http://s3tools.org/s3tools

It provides a great suite of tools for interacting with S3, and is best used on an EC2 instance you are connected to via SSH. It is also ridiculously fast, much faster than trying to interact with S3 from a local FTP browser, or even from Amazon's own S3 dashboard. For example on my computer using Amazon's own web facing dashboard it will take about 30-45 minutes to make 14000 files on S3 public, versus via the command line tool downloaded and running on one of my EC2 instances it can make those files public within minutes.

I assume this is because it is local network traffic. Anyway, if you are ever in a bind and need to move a bunch of files to S3, I highly recommend S3 Tools. It has saved me many times.

Along those lines wget is the most powerful command line tool I've ever used, with so much capability. It is simply incredible when combined with S3 tools, allowing you to easily grab gigabytes of images off of a personal server or staging location and upload them to S3 very quickly.

And if you need to do more than move files around you can manage even more aspects of AWS including EC2 instances from the command line using this powerful command line tool:

http://www.timkay.com/aws/

Its great for writing automated bash scripts to manage your instances via cron jobs.

epi0Bauqu2• 15 years ago

My main complaint with s3cmd is you can't easily throttle it, and on a busy machine its fast speeds can cause issues.

NathanKP2• 15 years ago

Good point. I guess in the past I have spun up a micro instance to run it, so I didn't have to run my S3 operations using one of my main production servers, and hence I didn't run into any problems with its unthrottled speed.

delinka• 15 years ago

Perhaps lowering its priority would help.

nice s3cmd ...

spiffytech• 15 years ago

Have you tried trickle[1]? It can limit the bandwidth of most programs I've tried it on.

  trickle -u 10 -d 20 ncftp

This runs ncftp capped at 10 KB/s up, 20 KB/s down.

[1] http://monkey.org/~marius/pages/?page=trickle

sfoguy• 15 years ago

iMacros firefox addon for web automation via command line (when curl is not enough): http://wiki.imacros.net/iMacros_for_Firefox#Command_Line_Sup...

ryduh• 15 years ago

Note: ngrep can also be installed using brew on OS X.

LiveTheDream5• 15 years ago

Also available via MacPorts.

genieyclo2• 15 years ago

My most used cli tool outside of the default nuts and bolts is dtrx, the best and easiest file extractor for *nix. No more fiddling with flags or looking up, handles issues with putting lots of things in different directories or the wrong permissions on files. It has saved me a ton of time over the years.

http://brettcsmith.org/2007/dtrx/

divtxt• 15 years ago

Wow! Just wow! Bookmarked! A mere upvote is not enough for this! Thank you!!!

Questions:

- Is there a site pointing out amazing command line tools like this? (their existence as opposed to usage examples like at http://www.commandlinefu.com/)

- Is there a site listing the OS X equivalents for Linux command line tools? (e.g. what's the OS X equivalent for dos2unix / flip?)

(edit: merge 2 comments into 1)

genieyclo2• 15 years ago

Yep, check out http://onethingwell.org which is exactly that. Software (incl cli) that do one thing well, as according to the UNIX philosophy. It's nicely tagged as well, so you can click the 'osx' tag and get tools just for OS X and so forth.

I actually found dtrx on hacker news though a few years ago, but it's also on onethingwell I believe.

ludwigvan• 15 years ago

Actually, for extracting tar files, using xf is enough.

    tar xf archive.tar.gz
    tar xf archive.tar.bz2

Still, this will be useful for other formats, I think. Thanks.

ez773• 15 years ago

Don't you need the z and j flags, respectively? The way you write it works for uncompressed tars.

marcusramberg• 15 years ago

This was worth it for me for the reminder about siege alone, as 'ab' seems to have some serious issues in Lion.

bretthopper• 15 years ago

Ever wanted to know the progress of a long running operation? Copying a file, importing a mysql db, etc. Pipe Viewer (pv) is what you need: http://www.ivarch.com/programs/pv.shtml.

It lets you monitor the progress of any piped command and gives you time elapsed, speed, time left, and a progress bar (wget style).

Copy a file:

  pv sourcefile > destfile

Import mysql db:

  pv dump.sql| mysql -uroot -p db

More tricks: http://blog.urfix.com/9-tricks-pv-pipe-viewer/

edit: To install on OS X just do

  brew install pv

brendano• 15 years ago

Pipe Viewer dramatically increased my productivity for large scale data processing. In particular, it lets you quickly know whether something will take 5 minutes, or 2 hours, so you can plan accordingly. It's painful watching people try to do this without pv.

nobuff• 15 years ago

Oh-my-zsh with git and git flow plugins work well so far. https://github.com/robbyrussell/oh-my-zsh

HerraBRE• 15 years ago

It's been mentioned on Hacker News a few times before, but my project PageKite (and showoff.io and localtunnel) is designed to help out with quick web demos and collaboration.

Getting started is now two lines:

  curl http://pagekite.net/pk/pagekite-0.4.py >pagekite.py

  python pagekite.py 80 SOMENAME.pagekite.me

... answer a few questions and whatever is running on port 80 will be visible as https://SOMENAME.pagekite.me/ within moments, almost no matter what kind of network connection you have. :-)

There are also .deb and .rpm packages available for heavier users.

Command line tools for web developers

Related Stories