BeGoneAds – A Python script that blocks ads by installing common hosts files

by anned20 • 7 years ago

129 92 7 years ago

Python scripts to modify system files make me a little skittish, even with source code available. I think I would just as soon grab the hosts file from https://www.someonewhocares.org/ and drop it in myself.

Redoubts• 7 years ago

I also don't understand systemd integration in the todo.

anned209• 7 years ago

The goal here is to make it automatically update the hosts file once in a while

joyjoyjoy3• 7 years ago

Why?

LASR• 7 years ago

Why not?

Modifying system hosts configuration requires privileged file system access.

The mindset here should be default deny.

joyjoyjoy3• 7 years ago

I don't understand the downvote.

1. What is so bad about python in specific?

2. If you worry about root privileges, required for modifying the host file, you can use app armor to put the thing on a leash

digitalni• 7 years ago

That hosts file is depressing. Blocking trackers by a1.tracker.name, a2.tracker. name etc. Today it seems easier to have a whitelist rather than blacklist...

barbecue_sauce• 7 years ago

Anybody have a sense of the performance overhead of using hosts files versus a detached hardware solution like a pihole?

LyndsySimon• 7 years ago

I don't have any evidence (I've not attempted to benchmark it or anything), but my gut says that the stack is checking the hosts file first anyhow, so it shouldn't be much. It might actually be an improvement over a separate appliance.

NikolaNovak• 7 years ago

My understanding is that difference is in scope, not performance.

Hosts files will only affect the host (workstation/desktop/laptop etc) they're installed on.

Things like piHole try to make it easy to apply the solution to all members of your network - which even in household cases these days can number in dozens, making it impractical to manage hosts files for all of them (This includes items like phones which are typically unfeasible to mess with hosts file).

ycombonator3• 7 years ago

It would be nice to see a performance hit based on the number of hosts entries.

ImprovedSilence• 7 years ago

Pie hole also has a nice browser interface to debug blocked requests that are breaking the site you don’t want To be broken. Which happens inevitably when you pull together 10 different sources of blocked lists.... or just one persons whose ideal blacklist doesn’t match yours.

memco• 7 years ago

I used one of the popular hosts files on my local machine for a while: the networking didn’t seem to suffer, but the boot time for my machine slowed noticeably. And manual updates were painful because loading the file in an editor is slow so if you use your hosts file for other reasons it can inhibit your workflow. I would recommend automated process on some dedicated device so you don’t impact your normal usage.

Another experience I had was that certain sites failed to work correctly. I didn’t do extensive testing but when I disabled the hosts nocking the sites worked, when I enabled it they broke. These were companies with whom I was trying to do account related business: so it wasn’t just that something didn’t render correctly it actively prevented me from updating my accounts when I tried to submit requests.

I still like the approach and will continue to use it, but it hasn’t been frictionless.

dredmorbius2• 7 years ago

Not much.

62,448 line (63,370 actual '0.0.0.0' entries) /etc/hosts file, 100x resolving 'www.google.com', Debian GNU/Linux, Thinkpad with spinning rust.

The short version has 32 lines, with 14 active entries, mostly defaults and local systems.

Short hosts:

    $ for i in {1..100}; do time host www.google.com; done 2>&1| grep real |  sed 's/^real[       ]*//; s/0m//; s/s$//' | mean
    n: 100, sum: 2.209, min: 0.015, max: 0.052, mean: 0.022090, median: 0.02, sd: 0.007450
    %-ile:  5: 0.016, 10: 0.016, 15: 0.016, 20: 0.016, 
    25: 0.0165, 30: 0.02, 35: 0.02, 40: 0.02, 45: 0.02, 
    55: 0.02, 60: 0.02, 65: 0.02, 70: 0.021, 75: 0.022, 
    80: 0.0245, 85: 0.029, 90: 0.033, 95: 0.0385

Big hosts:

    $ for i in {1..100}; do time host www.google.com; done 2>&1| grep real |  sed 's/^real[       ]*//; s/0m//; s/s$//' | mean
    n: 100, sum: 2.517, min: 0.016, max: 0.063, mean: 0.025170, median: 0.023, sd: 0.009818
    %-ile:  5: 0.016, 10: 0.016, 15: 0.016, 20: 0.016, 
    25: 0.017, 30: 0.0185, 35: 0.02, 40: 0.021, 45: 0.022, 
    55: 0.024, 60: 0.0255, 65: 0.0265, 70: 0.028, 75: 0.029, 
    80: 0.03, 85: 0.0325, 90: 0.0395, 95: 0.042

The delta of means is .003080s -- call it 3ms slower for the large hosts file.

("mean" is an awk script for computing univariate moments.)

As others have mentioned, the main benefit of a centralised LAN service is that all devices on the LAN are protected. The hosts file on this system (a laptop) is effective regardless of where I am. It also pre-dates my configuring OpenWRT's adblock package about a month ago, though I'd had a hand-rolled DNSMasq configuration earlier. The laptop hosts file is almost certainly a few years out of date -- another occupational hazard of such things.

The OpenWRT solution runs on the Knot Resolver (kresd) caching nameserver. I've not noted any lag for it. The blocklist there is currently 231,627 hosts/domains (roughly doubled: specific + wildcard matches), from 0-29.com to zzzpooeaz-france.com.

unethical_ban• 7 years ago

When the network goes down, it can take several minutes for it to come back up. I was having DNS and connectivity issues at a LAN party. Wouldn't get connectivity for minutes after a link bounce.

Then I removed the hosts file, and it worked instantly.

Maybe for a static workstation it wouldn't be bad, but for a laptop or something that loses link frequently, it could be an issue.

sherincall2• 7 years ago

I get that this is just someone's side project, I'm glad it exists and they're free to write it in their favorite language/environment and all; but the effort to actually run this is equivalent to actually copying the hosts files manually, and I already have all the dependencies installed. I could never get my non-techy parents to run this properly.

If the goal of the project is actual adoption, a native executable without external dependencies would have been a much better option.

anned209• 7 years ago

This is a todo, It's already on PyPI and I'm working on getting it packaged for all the main distros of Linux/Windows and MacOS.

sherincall2• 7 years ago

I saw the todo item, but didn't realize it also included providing an executable for Windows and macOS. Thanks!

inlined3• 7 years ago

Since accepting Host files from someone on the internet can be dangerous I dug into the code:

The list of hosts to exclude comes from several sites here: https://github.com/anned20/begoneads/blob/2c90fcee221edf71f8...

The actual application of the hosts file is here: https://github.com/anned20/begoneads/blob/2c90fcee221edf71f8...

I missed something though. Is a simple domain name per line enough to send that content to /dev/null? I haven’t used that form in /etc/hosts.

My primary concern was that this technique could be used to send ad traffic to a site that returns 404 but gathers metrics on the web regardless.

anned209• 7 years ago

This is actually sent to the IP address 0.0.0.0, it roughly means that all the traffic of the listed hosts is routed back to localhost

lelf• 7 years ago

0.0.0.0 ≠ localhost

unfunco• 7 years ago

In this context, it actually means a "non-routable meta-address used to designate an invalid, unknown, or non-applicable target" [1] - 127.0.0.1 is localhost, 0.0.0.0 is its own thing.

[1] https://www.howtogeek.com/225487/what-is-the-difference-betw...

js2• 7 years ago

I think you may be misreading the code. It concatenates the host files at the various URLs and then inserts their contents into /etc/hosts. I only looked at a couple of those files but the ones I did used either 0.0.0.0 or 127.0.0.1 as an address combined with a domain.

To be honest, this code is over-engineered. It could be a single script with a handful of functions. At the same time, it’s missing functionality such as deduplicating entries from the different lists.

anned209• 7 years ago

Those are things that are still on a small personal todo list. I agree that it could be much simpler, but I like the elegance of it.

daverobbins12• 7 years ago

How is this different or better than Steven Black's project?

Repo: https://github.com/StevenBlack/hosts

DyslexicAtheist3• 7 years ago

my first question too. Steve Black's updateHostesFile.py is extensible and can be automated and is well trusted and gets tested by a large community. I don't want to sound critical but would like to understand the value add in comparison. Both are in python as well so I don't get it

fuzz4lyfe• 7 years ago

I assume you wrote a hello world once or twice, what was the point of that?

anned209• 7 years ago

To be completely fair with you, I didn't know this was out there. Then again, I think my solution is more elegant, especially if all the todo's are finished.

tempsolution• 7 years ago

Patching a critical system file with python on a regular basis... What could go wrong?

joyjoyjoy3• 7 years ago

I use host flash: http://host-flash.com/

Does anyone know an up-to date list for blocking social networks?

DyslexicAtheist3• 7 years ago

Steve Black's hosts file ... just specify "-e social" or "--extension social" option, or use a "myhosts" file to name your own domains for a subset (e.g. all of facebook or whatever)

https://github.com/StevenBlack/hosts

jakeogh• 7 years ago

Nice to see projects using click!

Here's another one to toss on the pile (works, I use it, supports wildcards, *nix only): https://github.com/jakeogh/dnsgate

zactato• 7 years ago

Serious question. We all realize that the economics of the internet is largely fueled by ads, so why are we so keen to block them? It’s ad revenue that have allowed technology to flourish so strongly over the last two decades.

dsswh2• 7 years ago

Not long ago, the economy was largely fueled by slavery. Yet we got rid of that.

layoutIfNeeded• 7 years ago

Citation needed

briandear• 7 years ago

Not exactly Godwin’s law, but pretty close. Comparing advertising and marketing to the ownership of human beings? Slavery infringed on the inalienable of human beings, the existence of advertising doesn’t take away my freedom or potentially subject me to beatings.

It’s a ridiculous comparison. I am not a friend to intrusive ad-tech, but making a moral equivalence to slavery is to trivialize slavery. It’s like comparing parking tickets to the death penalty.

jormungand• 7 years ago

You are partially right. There are ads that are truly useful and those which are malicious, such as popunders which often include scammy pops. I use adblock to prevent the latter. There are popunder ad networks which are trying to fight against adblock by introducing "solutions" like anti-adblock, see here: https://propellerads.com/blog/anti-adblock-3-monetize-99-per...

I side with the adblock solutions in this war.

pinguinFromY• 7 years ago

I would not mind contextual ads that don't get in the way of viewing the actual content a website or mobile app is offering. But no, we get huge banners that cover the whole background, pop-ups, every click opens a new tab redirecting to an ad and "hot chicks in your area". If I'm seeing an intersting ad I'll click on it by myself, don't need your help really. Conclusion: it's not the ads themselves but how they get in your way. See reddit ads, google ads. They are part of the actual content.

mtgx• 7 years ago

Whether or not ads on the internet are good is not that clear cut. But I would say 99% of tracking is both evil and mostly useless, probably with the 1% being first-party analytics to track some very simple stuff like page views.

I'm glad Firefox is now blocking third-party trackers by default (not that I needed it for myself, but it's important for others to have this).

geggam• 7 years ago

Internet was a better place when the economics of the internet was driven by porn.

</2 cents>

aagd• 7 years ago

I'd say it's not so much about the ads, more about the way they're delivered with all those shady tricks for tracking and fingerprinting. IMHO ad blocking is more about privacy than anything else...

ycombonator3• 7 years ago

Ads have gotten to a point where they hamper user experience. My problem with most of the ads is the amount of js tracking junk and 3rd party A/B calls that grinds the browser to a halt.

cm20122• 7 years ago

Do you mind FB ads which are pretty smooth experiences?

rafaelvasco• 7 years ago

Reading the code one clearly sees why Python is so well suited for these kinds of applications, one-shot script executables: Really nice string ops, regex, file io etc. One of my favorite languages. The other is C# for everything else, that Python is not that suitable for: Huge complex codebases, type safeness, more strict performance requirements etc. Specially the static typing. The dynamism and lack of type annotations of Python really bothered me when I was developing a somewhat complex desktop app in it some years ago. I guess I'm a static typing guy with optional dynamism kinda person.

misterdoubt• 7 years ago

If you haven't checked back lately, type annotations in Python are getting better and better. Built-in support via the typing module and a strong community package in mypy.

firefoxd• 7 years ago

I wish the hosts file could have an include directive. Since I regularly add or remove entries, the file becomes a mess.

ris• 7 years ago

You ran WHAT script on your machine?!

hlau3• 7 years ago

I'll be the first to admit that the existing advertising ecosystem is broken, primarily due to misaligned incentives across the board. But, given a choice, would you rather have a clearly labeled thing that you know is an ad transparently trying to influence you or a sneaky human billboard, err "influencer" coming up to you with an agenda along with tons of product placement in whatever you watch/read/listen to?

harry82• 7 years ago

There's no either/or decision to be made here. You get compromised, paid for content with our without ads as well. Critical thinking I'd a requirement always.

hlau3• 7 years ago

There definitely is an either/or because blocking of one channel will naturally necessitate money/barter flowing to the other channel. One is at least transparent and regulated, the murky world of influence peddling isn't since it's hard for anyone to tell in the moment whether something is "organic" or not.

mehrdadn• 7 years ago

Hosts files slow down the system as well as the browser itself. Get/create a browser extension to actually block the request (at least while your browser supports this) so you get immediate results.

bigend• 7 years ago

If you let someone else manage the hosts your computer resolves, you are trusting that someone as much as your ISP. A man in the middle.

gregw2• 7 years ago

I have cron jobs on my mac that update my hosts files (to block "addictive" sites in my case (not ads)). It doesn't really work.

Browsers cache and use outside DNS servers despite the hosts files. Chrome and sometimes Safari don't really honor the hosts files 100% of the time. Every once in a while I google around to try and restore my control, try to tweak my browser settings but I have yet to find anything that makes using hosts files bulletproof.

mywittyname• 7 years ago

I think firewall rules would be your next line of defense. I'm not sure how configurable most home routers are though.

appleflaxen• 7 years ago

how does the list compare to the pi hole hostfile?

BeGoneAds – A Python script that blocks ads by installing common hosts files

Related Stories