Show HN: A Python Script to Download Thousands of Wallpapers at Once

https://github.com/GeekSpin/Wallhaven-dl

by geekspin • 8 years ago

27 24 8 years ago

github.com

kpenc• 8 years ago

Every function has side effects. Could've written the whole thing in one function. Also, PEP8.

apetresc2• 8 years ago

One function? Could've written the whole thing in one `wget` call.

pknerd• 8 years ago

teach me how.

mynewtb2• 8 years ago

Every code has side effects. Splitting code into appropriate functions makes those side effects much more managable.

jfz• 8 years ago

Using i and j in loops makes code much less self-explanatory, especially when you could use "page_index" and "image_index" instead.

lou13062• 8 years ago

Even better, replacing

     for i in range(len(imgid)):

and similar lines with:

    for i, img in enumerate(imgid):

would allow one to get rid of all these list accessors.

myroon52• 8 years ago

It looks like this downloads them one-by-one instead of downloading at the same time.

Is there a simple Python equivalent to Java's .parallelStream().forEach() that would allow these calls to easily be run in parallel?

mynewtb2• 8 years ago

Sure thing...

    from multiprocessing import Pool
    with Pool(8) as p:
        p.map(function, sequence))

pddubs• 8 years ago

You could just use threads here too - CPython releases the GIL during IO.

  from concurrent.futures import ThreadPoolExecutor
  with ThreadPoolExecutor as executor:
      executor.map(function, sequence)

myroon52• 8 years ago

Neat.

I see that you explicitly specify 8 as the number of processes here. In Java, parallelStream() will pick a sane default for you if you haven't previously specified (based on the number of available processors, I believe). Is something like that possible in Python?

shubb2• 8 years ago

You might want to async the downloading part. It is often faster to download 5 images at once than to download them one after another.

In the choices section you use input("text") properly one place but not others. You use a couple different ways of decoding codes, and the dict way is nicer, but also consistency is nice. Also, I am not sure you handle bad input (default all?)...

Personally, I'd pull the meat out of the loops in main() into functions - GetImageList() and GetImage(). Relatively complex there so it would be easier to read and spot errors in those bits of code in isolation.

ASalazarMX• 8 years ago

Unless you're in dire need of thousands of wallpapers (most of them you are going to delete anyway) it's better not to hammer the website. I'd even limit the download rate.

ejdanderson• 8 years ago

Agreed - I looked at the code hoping for a good example of async in python as suggested by the “at once” in the title.

slenk2• 8 years ago

Side question - are there any websites like wallhaven, but with less people and anime? I'm thinking the type of content Chromecast uses, or /r/TechnologyPorn?

xemoka• 8 years ago

Have you seen http://interfacelift.com/ ?

kspy• 8 years ago

I enjoy https://papers.co

umeshgmrl• 8 years ago

What about https://unsplash.com/ ?

Show HN: A Python Script to Download Thousands of Wallpapers at Once

Related Stories