krizhanovsky

Most open-source L7 DDoS mitigation and bot-protection approaches rely on challenges (e.g., CAPTCHA or JavaScript proof-of-work) or static rules based on the User-Agent, Referer, or client geolocation. These techniques are increasingly ineffective, as they are easily bypassed by modern open-source impersonation libraries and paid cloud proxy networks.

We explore a different approach: classifying HTTP client requests in near real time using ClickHouse as the primary analytics backend.

We collect access logs directly from Tempesta FW (https://github.com/tempesta-tech/tempesta), a high-performance open-source hybrid of an HTTP reverse proxy and a firewall. Tempesta FW implements zero-copy per-CPU log shipping into ClickHouse, so the dataset growth rate is limited only by ClickHouse bulk ingestion performance - which is very high.

WebShield (https://github.com/tempesta-tech/webshield/), a small open-source Python daemon:

* periodically executes analytic queries to detect spikes in traffic (requests or bytes per second), response delays, surges in HTTP error codes, and other anomalies;

* upon detecting a spike, classifies the clients and validates the current model;

* if the model is validated, automatically blocks malicious clients by IP, TLS fingerprints, or HTTP fingerprints.

To simplify and accelerate classification — whether automatic or manual — we introduced a new TLS fingerprinting method.

WebShield is a small and simple daemon, yet it is effective against multi-thousand-IP botnets.

It's useful to store a web server access logs in an analytics database, e.g. to fight against bot attacks. We store structured access logs in Clickhouse, which is already good, but compression and data ordering from the post may improve performance even more - we'll try this.

The thing is that a web server, especially under DDoS, may produce much more records than Clickhouse can ingest. But there is good news: for Nginx, if you build a fast pipeline to feed access logs to Clickhouse, you can increase performance, I'd say up to x2, thanks to faster access logging.

uRPF prevents IP spoofing used in volumetric DDoS attacks. However, it seems uRPF is vulnerable to route hijacking on its own

This is quite insightful, thank you.

This particular project, WebShield, is simple and it didn't take too long to develop. Basically, with this project we're trying to figure out what can be built having fingerprints and traffic characteristics in an analytic database. It's seems easy to make PoCs with these features.

For now, if this tool can stop some dummy bots, we'll be happy. We definitely need more development and more sophisticated algorithms to fight against some paid scrapping proxies.

It's more or less simple to classify DDoS bots because they have clear impact - the system performance degrades. For some bots we also can introduce the target, for the bots and the protection system, e.g. the booked slots for a visa appointments. For some scrappers this is harder.

Another opportunity is to dynamically generate classification features and verify resulting models, build web page transition graphs and so on.

This is a good point about possible blocking of ~50% of the Internet. For DDoS we _mitigate_ an attack, not _block_ it, so probably for bots we should do the same - just rate limit them instead of full blocking.

Technically, we can implement verification of client side certificates, but, yes, the main problem of adoption on the client side.

That's a good advice, thank you.

In our approach we do our best to not to affect user experience. E.g. consider an example of a company website with a blog. The company does it's best to engage more audience to their blog, products whatever. I guess quite a part of the audience will be lost due to requirement of authentication on website, which they see first time.

However, for returning, and especially regular, clients I think that is a really simple and good solution.

Hi,

thank you for the reply!

You can read about JA5 at https://tempesta-tech.com/knowledge-base/Traffic-Filtering-b... .

But the thing is that the hashes were just inspired by the work of John Althouse and there is no any relation.

Unfortunately, we didn't realize what "JA" stands for at the time we were designing the feature. We will rename it https://github.com/tempesta-tech/tempesta/issues/2533 .

Sorry for the confusion.

The benchmarks https://github.com/ncm/computed-goto/blob/master/benchmarks/... benchmark is not applicable to this discussion because it compares _too_ small state machines. I reference my talk and presentation once more: http://www.tempesta-tech.com/research/http_str.pdf - slide 23 discusses that the goto FSM makes sense for _hundreds_ of states.

I'm unable to find measurements. I mean, the amount of code duplication is not an optimization parameter, the amount of Mb/s of HTTP parsed is.

The measurements are covered in slides 23-24 in http://www.tempesta-tech.com/research/http_str.pdf

goto jumping between branches is a bad thing from the point of view of maintainability.

In our parser we have DSL for the state machine, so we encode which state the machine should go. The HTTP parser is one of the most updated piece of a web accelerator code and we don't struggle on the FSM - every developer in our team updates the code, even newcomers.

Thank you, I'm glad that you enjoyed the article!

Regarding computed and simple goto I'd like to reference our early article discussing the parser in standard goto https://natsys-lab.blogspot.com/2014/11/the-fast-finite-stat... . My recent talk at scale https://www.socallinuxexpo.org/scale/17x/presentations/fast-... (watch videohttps://www.youtube.com/watch?v=LQc4er8ng64&feature=youtu.be... and slides at http://www.tempesta-tech.com/research/http_str.pdf ) discusses the parser with the compiler extensions.

We also found hat Ragel parser generator (using the goto state machine) generated somewhat faster large state machines than Bison (switch-drived state machine). However, as pointed out in the SCALE talk goto works better only on big enough state machines.

While I addressed safety in a separate section in the article, I wouldn't argue about that: it seems Rust designers made the perfect work in safety. However, C++ is moving in this directly, bu there is the "gap" as it was described in the cited talk from the CppCon 2020.

However, there article is about FAST programming languages. Which means, and I stated this explicitly at the beginning of the article, that the main factor for the article is the speed of the generated code.

This is why I compared the single Rust implementation with 3 C/C++ implementations. The question was: whether Rust does something unreachable for C/C++? And the answer is "NOT". Also please keep in mind that all the benchmark programs, using the same algorithms, are still coded in bit different ways. And the differences impact performance significantly. I analyzed two programs, in Rust and C, to show the differences.

I still don't see any misconceptions. The reason why the kernel uses SIMD with FPU save/restore is to optimize context switches. We addressed the topic in https://netdevconf.info/0x12/session.html?kernel-tls-handsha... . I guess early versions of WireGuard used the same approach: save FPU context at the beginning of softirq, process may packets with SIMD in one shot, and restore FPU state.

There are also other issues with the kernel code and I addressed them in the article, why it doesn't makes sense (while still possible) to use C++ in the kernel code.

Please read the discussion in https://www.reddit.com/r/Cplusplus/comments/jjtn5v/fast_prog... . In short, everything is doable, but before starting your project, which you're paid for, you have to grow quite a large C++ infrastructure, which double the native Linux kernel API in many ways.

Operator new is wholly compatible with kernel allocators.

Please read the referenced thread. The things aren't so simple.

There were quite a long work, for couple of years, on the parser. The first article about the parser is https://natsys-lab.blogspot.com/2014/11/the-fast-finite-stat... and next I updated it significantly in my talk at https://natsys-lab.blogspot.com/2014/11/the-fast-finite-stat... (http://www.tempesta-tech.com/research/http_str.pdf and https://www.youtube.com/watch?v=LQc4er8ng64&feature=youtu.be... ). The talk also discusses code repetitions: compiler makes duplicate code anyway, even for straightforward `for` loop and nested `switch`, but in our case we had much more duplicate code.

The thing is that we used hot/cold labels to minimize jumps over the code and improve instruction cache usage, so calling functions isn't good at all. We wanted one big function with is executed as linearly as possible to improve instruction cache prefetching.

I didn't mean anything condescending here actually. The point is that when people starting programming, it's good to put them into a restricted environment, so that they get used to use the right tools and in the right way. For example, it's quite easy to misuse goto in normal programs and make the programs unreadable. However, if a developer is already perfect with writing good code, but (s)he needs to do something special, like the state machine in the article, the restriction just makes the developer's life harder.

But did you? My FreeBSD knowledge is quite outdated, I believe the last version I worked with is 7, but I believe at some point, it was possible to compile C++ kernel modules for FreeBSD. This time I can not find the proof, but plus to that post I also found links like

https://lists.freebsd.org/pipermail/freebsd-hackers/2009-Feb... https://lists.freebsd.org/pipermail/freebsd-hackers/2006-Jul...

The last one mentions: "There's actually a fair amount of experience with people doing C++ in FreeBSD kernels. People have been doing things with it for about 8 years now."

Also if you don't like my example with quick fix of some dirty project, then consider InnoDB storage engine which was developed in pure C for 2 decades, but now it's C++.

Well, we didn't mention this in the article, but this is very practical actually. About 10 years ago we had a request significant reworking an open source DNS server and the main problem with the project was that almost whole logic was placed in about 10 functions and most of the functions exceeded 1000 LoC. Each code update involved plenty of headache with dynamic memory freeing. We spent about 1-2 days to compile it with C++ and replace the most crucial pointers with smart pointers, that we didn't have to care about memory freeing. That time we had to make the job done ASAP, surely we should have spend time in proper code refactoring.

Surely you need to modify C code to compile it with C++ and in the article we made an example of the modification. But the point is that the changes are pretty straightforward and small.

Unfortunately, there is no good explanation of the algorithm which you refers, also I didn't find a source code for this.

This is a good question. Currently we've integrated the queue in two projects in production state: user-space proxy server and VPN capturer (Linux kernel module).

In both the cases we enhanced the queue with conditions like is the queue empty or is it full, so we can drop packets if consumers are overloaded and there is no sense to put a packet to the queue.

Secondly, the queue is designed to work on multi-core environments and this is usually multi-node NUMA systems for modern hardware. So we adjusted both the applications in such manner that administrator can assign different number of cores for all processors for consumers and producers depending on current workload. This makes the system mode balanced, so queue is empty or full rarely.

And finally, for kernel space we also implemented lightweight also lock-less condition wait for the queue (http://natsys-lab.blogspot.ru/2013/08/lock-free-condition-wa...). It also gives us lower CPU consumption when there is no workload (so the system eats less power) and even better performance due to reduced cache bouncing.

The algorithms are simply different and solve different problems: disruptor is a messaging queue (if a producer P0 emits M0 into the queue then all consumers C0..CN receive M0) while our queue is a classical work queue (if P0 and P1 emit message M0 and M1, then only one consumer Ci receives M0 and only consumer Cj receives M1 (i could be equal to j)).

Our implementation competes with boost::lockfree::queue and it's much faster since Boost implementation uses more heavy synchronization techniques. The benchmark on GitHub also has Boost implementation, so you can compare both the queues.

Unfortunately, there is no adequate algorithm description for disruptor queue, only some indistinct descriptions mostly suitable for business people rather than engineers. So it was not easy to dig it's source code. However, I learned it and there are some notes about the implementation.

The implementation is bit inaccurate: there are a lot of branches without branch prediction information available at compile time, to avoid cache line bouncing it wastes two cache lines instead of simple align an item on cache line (I mean vrt_padded_int). I didn't pay too much attention to memory barriers usage, but giving that X86-64 provides relatively strict memory ordering, probably some of them also could be eliminated. Disruptor uses very cleaver ideas, but I believe its performance can be improved after good code review.

One more point is that while our queue implementation is C++, it's still self sufficient and can be easily ported to C for using in kernel space. It's doubtful (in my humble opinion) that generic container depends on non-standard libcork and moreover logging library (clogger).