Curl vs Wget

http://daniel.haxx.se/docs/curl-vs-wget.html

by helwr • 16 years ago

149 59 16 years ago

daniel.haxx.se

dlsspy• 16 years ago

"libcurl does not put any restrictions on the program that uses the library."

I wonder how much of the activity could be attributed to such things.

henning• 16 years ago

Probably wget's major reason for being is mirroring websites. It does this well. Curl is a more general HTTP library with a command line front end.

And of course there are GUI frontends for wget, but it's not really intended to be used as a library. I don't see a problem here.

ighost• 16 years ago

kinda banal if you ask me. it's two simple tools that do approximately the same thing. even grep vs ack would be more interesting.

no offense to the authors of course, they did a great job in both cases.

sorbits• 16 years ago

As someone who have always used curl and wondered why I often see wget quoted in READMEs (as how to obtain some remote file), I quite enjoyed reading the write-up.

My take-away was that a) I am not missing out by staying with curl and b) the use of wget is likely due to being part of GNU.

Maskawanian• 16 years ago

I use wget for one shot downloads because it is less to type. I use libcURL when developing.

This is the difference between "wget http://www.google.ca/ and "curl -O http://www.google.ca/

Just a simple small but significant difference.

AlisdairO• 16 years ago

Well, plus the recursive download facilities. This is the reason why I mostly use wget on the command line, although I'd use curl in scripts.

ZeroGravitas2• 16 years ago

You did notice this was written by curl's author? I only noticed after I read it and without that knowledge thought it was rather a pathetically biased article.

But you can't expect developers to be impartial about their own code and it was my fault for not noticing the disclaimer the first time. It reads completely different with that framing in mind, but still if I was looking for the definitive reason that one tool was recommended over another I'd consider the point of view of the developers as a starting point rather than an open and shut case.

arethuza• 16 years ago

I'm pretty sure awk is Turing Complete while grep isn't.

ZeroGravitas2• 16 years ago

He said ack, not awk:

"ack is a tool like grep, designed for programmers with large trees of heterogeneous source code."

http://betterthangrep.com/

mikeytown2• 16 years ago

What I do

  * curl for any bash scripts
  * wget for downloading via command line

The error codes that curl output make it very powerful when creating a script. Something that gets run on cron & you want to know why it failed if it did.

Wget is simple, and for a shell it's what I always turn to. The only time I would use wget in a script is if I was making some sort of simple web crawler script.

sliverstorm• 16 years ago

same here. I use wget to do stuff, and I use curl to do complicated/fancy stuff. One is simple, one is powerful. I'll use simple whenever I can get away with it.

statictype2• 16 years ago

>Wget requires no extra options to simply download a remote URL to a local file, while curl requires -o or -O. However trivial, this fact is often mentioned to me when people explain why they prefer downloading with wget.

Funny because that's exactly the reason I installed macports and then wget. It's stupidly trivial, but there you have it.

telemachos• 16 years ago

I'm not criticizing, but wouldn't a simple alias be easier?

    alias co='curl -O'

statictype2• 16 years ago

True, much simpler. I guess a larger reason would be that 7+ years of using wget has wired me to its commandline. I'm so used to typing `wget -c "<middle-click>"` or something on the command line, that I just never considered creating aliases for alternatives.

mahmud• 16 years ago

'co' is already taken though :-)

rix0r• 16 years ago

Another stupidly trivial reason:

I can type 'wget' and then do command-V entirely with my left hand, keeping my right hand on the mouse. Having to type 'curl' would require me to move my hand back and forth, slowing me down.

Groxx2• 16 years ago

I love reasons like this, they make non-geeks look at us like we're insane. I've only used curl personally, so this wouldn't be enough to switch, but it's an excellent point.

oconnore• 16 years ago

Bash fu? alias qwer='curl -o'

godDLL2• 16 years ago

I'm guessing QWERTY? Dvorak and Colemak are not too good for single-hand typing.

danudey2• 16 years ago

stupid unix tricks:

curl http://somesite.com/download/latest.tar.gz | tar zx

(saves having temporary files lying around everywhere)

Freaky• 16 years ago

OS X probably ships with FreeBSD's fetch(1), which should do for most <command> <url> needs.

Malus• 16 years ago

  mini[~ 111] uname -a
  Darwin mini.local 10.3.0 Darwin Kernel Version 10.3.0: Fri Feb 26 11:58:09 PST 2010; root:xnu-1504.3.12~1/RELEASE_I386 i386
  mini[~ 112] which fetch
  fetch: Command not found.

Unfortunately, it does not. I use fetch all of the time on FreeBSD, so it is a little obnoxious to have to go and install wget from MacPorts.

sumeeta• 16 years ago

Wget seems to be better at naming saved files:

`curl -O http://host/my%20file.txt` annoyingly saves to `my%20file.txt` while `wget http://host/my%20file.txt` will save to `my file.txt`.

Kadin• 16 years ago

I don't think it's all that trivial; whatever a program does when run without options should be a big clue towards the primary use case that it is designed to handle or fulfill.

In the case of curl, it writes to stdout; wget writes to a file. While the mechanics of each are the same, curl lends itself towards use in scripts while wget (at least in my experience) is better if you're kicking it off interactively.

Also, wget has a very handy batch mode, where you can feed it a file containing nothing but a list of URLs and it will fetch each one. I am not familiar with a similar one for curl -- although admittedly you could do the same thing with a minimal wrapper script, that's a lot more keystrokes than "wget -i <file>".

X-Istence5• 16 years ago

I use curl for almost everything. Especially for debugging caching and whatnot, using curl -D - -o /dev/null http://example.net/ makes it extremely simple to see the headers the server is sending for a certain file.

Not only that, but curl is installed standard on Mac OS X whereas wget is not, also, most attacks that attack PHP tend to use wget to download further files to the target system, which means that when wget is not available no extra files are downloaded and it generally stops the exploit cold in its track as the attacker does not try again.

vog3• 16 years ago

> Not only that, but curl is installed standard on Mac OS X whereas wget is not

On a vanilla Debian, Wget is installed but Curl is not.

vog3• 16 years ago

> most attacks that attack PHP tend to use wget to download further files

Nowadays, neither Wget nor Curl is required for downloading in PHP, because functions like file_get_contents() are able to handle URL arguments! So a missing Wget protects only against script kiddies who don't know PHP very well and copied over some outdated code snippets.

Also, not installing Wget "protects" you just from those kind of primitive attacks to which your system shouldn't be vulnerable in the first place.

Security is achieved by sealing entry doors, not by removing tools.

X-Istence5• 16 years ago

file_get_contents on most hosts does not allow external URL's, including the one where I am an administrator.

http://www.php.net/manual/en/filesystem.configuration.php#in...

Also, installing wget is just one more way to defend a server that is hosting multiple sites with multiple scripts all of which can not be trusted all the time. I don't know if tomorrow a new bug is found in phpBB that allows a remote exploit that runs as the user php runs under, if not having wget stops one such exploit it is a win in my book.

The other thing I have in my firewall is all outgoing access is blocked for the user that PHP runs under so that in the event something does happen and something is executed under that user it won't be able to make an outbound connection or get inbound connections.

Yes, security is achieved by sealing entry doors, but removing an entry door in the first place may be even better, wget being an entry door, unfortunately the front door is a huge gaping mess you could drive a truck through, but that is unfortunately required for the business I am in, administrating a server with multiple websites running various PHP scripts.

cpach• 16 years ago

The -D option was really neat, thanks for this useful tip!

X-Istence5• 16 years ago

You are welcome. It has made life so much easier for me in the past when I wanted to know what the server was sending exactly.

prakashk• 16 years ago

> Especially for debugging caching and whatnot, using curl -D - -o /dev/null http://example.net/ makes it extremely simple to see the headers the server is sending for a certain file.

Working on web stuff, I'd needed to see server headers many times, and I have always used the HEAD command, that is part of the libwww-perl package on ubuntu. It's just: HEAD http://example.com.

The libwww-perl package also comes with GET (which is similar to curl in that it sends the retrieved content to standard output) and POST (which sends a POST request to a server) commands which are very handy too.

danudey2• 16 years ago

For that, I usually use 'curl -I http://example.com/, in case you don't want to install a separate package (or, for some reason, dislike perl).

pibefision• 16 years ago

Does anyone knows how to mirror a site using wget or curl, and get all the assets? (specially images linked on stylesheets).

Tichy• 16 years ago

Some time ago I filed a bug report because of the stylesheet images. So it still isn't fixed? It's simply a bug.

I have tried and liked pavuk in the past, don't know if it supports stylesheet images.

megamark16• 16 years ago

I use HTTrack, usually handles the job just fine.

mjs2• 16 years ago

wget has a better-designed command line API and documentation, although its inability to PUT or POST is rather unhelpful. For example, curl uses --include to output full headers (wget uses --save-headers) and, more inexplicably, --request to change the request method, even though --method is available.

barrkel2• 16 years ago

Use --post-data or --post-file to POST with wget.

mjs2• 16 years ago

--post-* is pretty sensible for POST. Unfortunately wget doesn't do PUT or DELETE.

cowsandmilk• 16 years ago

lwp-download should be in this comparison. I've found lots of random cases where it does stuff right and the others don't.

godDLL2• 16 years ago

As should aria2. But that would make it a completely different type of an article, more like a Wikipedia sub-section. We already have Wikipedia.

Groxx2• 16 years ago

That's a nice breakdown... I've been a curl user, and never really knew what the major differences were. And it doesn't seem overly biased to me. Many thanks to the author!

barrkel2• 16 years ago

I find wget is almost always what you want, unless you're interested in development details like headers. If I want output to a pipe, I use wget -O -.

In scripts, I use wget -o <log-file> (or -a) so that I can tail -f to keep track of it. I don't see off hand what curl's log file support is, so I can't comment too deeply on it.

whyme• 16 years ago

When downloading via HTTPS curl seems to barf error codes often so I try wget first, and use curl only if wget doesn't work or curl is easier (which isn't often).

1amzave4• 16 years ago

My main complaint about curl is that it prints a bunch of crap (well, statistics technically) to stderr by default. Especially for something claimed to be (at least in this article) "traditional unix style", that seems undesirable.

It's annoying when I do 'curl $URL | less' and end up with garbage in my pager that I have to scroll around a bit to make disappear.

It seems like default behavior should be more like 'curl -s -S' -- silent, but print errors if they occur, just like...well, every other sensible command-line tool in the world (so that's what I have curl aliased to).

wget of course prints statistics too, but it's not such "pipe-oriented" tool, as the author points out, so this isn't as big of a deal.

sophacles2• 16 years ago

Actually, it is very much unix style. With any modern shell you can simply turn off stderr, or only put stdout in the pipe, or put 2>/dev/null and so on. If both went to stdout then it would be a reasonable complaint.

1amzave4• 16 years ago

> With any modern shell you can simply turn off stderr, or only put stdout in the pipe, or put 2>/dev/null and so on.

Or I could type fewer extra characters by using the '-s' flag, as I described in my comment. My point was about its default mode of operation: "noisy" (which is not, as I see it, a characteristic of "traditional unix" tools).

Edit: Additionally, if both went to stdout the tool would be completely unusable -- I'd simply file that under "data corruption".

tlrobinson• 16 years ago

My version of curl only prints the progress junk to stderr if you pipe stdout somewhere. I guess I've never had a problem since I don't use a pager usually.

BTW, how do you detect whether your program's stdout is being piped to something other than the shell?

1amzave4• 16 years ago

> My version of curl only prints the progress junk to stderr if you pipe stdout somewhere.

Hmm, good point -- I actually hadn't noticed that, and it seems to be the case on mine as well. (Though I rarely use it without piping or stdout-redirecting to something, so it still strikes me as inappropriate.)

> BTW, how do you detect whether your program's stdout is being piped to something other than the shell?

In the three languages I've used recently enough to remember:

C: int isatty(int fd); /* from unistd.h */

Python: os.isatty

Shell (bourne): [ -t 1 ] # exits 0 if stdout is a tty, 1 if not

Curl vs Wget

Related Stories