Struggling with Container HTTP Logs? Meet Logmink

https://github.com/mutairibassam/logmink

by mutairibassam • 2 years ago

13 16 2 years ago

The Issue: When working with Docker containers, capturing and visualizing HTTP logs can be a pain. You usually need to modify your code or set up complicated logging systems just to see what’s happening with your app's HTTP requests.

The Solution: Logmink! Logmink is an easy, plug-and-play tool that lets you capture HTTP traffic from your Docker containers without touching your app code! Just add the Logmink agent, and all your logs are sent to a central hub where you can store and visualize them.

Why Use Logmink?

Quick Setup: Get started in minutes, no code changes needed. Centralized Logs: View all HTTP requests from all containers in one place. Easy to Extend: Built on Node.js, so you can easily add your own features. Try it out and make monitoring your containers a breeze: Logmink GitHub Repo

#Docker #DevOps #OpenSource #Logging #Containers #NodeJS

euph0ria• 2 years ago

What is the performance like when doing this? Doing pcap capturing and processing in node.js feels slow, but that is just a feeling and I don't have any data to back it up.. are there any performance impact studies?

FooBarWidget2• 2 years ago

I also wonder whether the code actually works reliably. They have an "onpacket" handler on the pcap object, but this handler is, I think, called on every TCP packet. Which may not correspond to an HTTP session.

Consider an app that starts an HTTP session by sending two TCP packets:

1. GET /foo

2. HTTP/1.1\r\n...etc...

Their code would not catch this.

mutairibassam5• 2 years ago

I agree with you, currently I am only catching the request that have valid http method. I mentioned this in code comments there is todo task to capture different cases as well.

Moreover, I am happy to learn more from you and if you have any recommendation please mention it. This simple implementation has met my needs.

Regarding the performance, to be honest I've not tested it as I am not expert in this domain, so any hit is appreciated.

mutairibassam5• 2 years ago

I did some fixes, now it should capture all TCP packets. Appreciate your support to validate.

firesteelrain4• 2 years ago

I am probably missing something. We just attached a Splunk UF container and forwarded all of our logs (events) to our centralized Splunk instance.

mutairibassam5• 2 years ago

Please clarify the point or the challenge if this related to the package?

firesteelrain4• 2 years ago

Both tools (Splunk UF running as a sidecar and Logmink) serve the purpose of forwarding logs for centralized analysis. While Logmink has a narrow focus of just HTTP logs, Splunk is log general and can serve the same purpose.

Splunk is considered more enterprise grade. Logmink focuses only on HTTP logs. But Splunk seems more appropriate here for enterprise environments.

PeterZaitsev• 2 years ago

Hm. Storing logs in MongoDB. Does it really scale ?

szszrk3• 2 years ago

Can't tell if this is a joke or not [1].

I think it actually makes sense. I did a few implementations of Elastic and while it was great, but even with all aspects fully managed via code, at some point it consumed insane amount of brainpower to keep expanding, adding features, securing and patching it. It also consumed very large amounts of hardware - which I was fine with mostly, as this was onprem and costs were accepted.

I tried something similar in a more cloud-based environment (both private and public) and I was shocked how much infrastructure I have to pay for just to create a reasonable PoC. Tried same with Openshift/Vector and it just gave me a headache.

There are plenty of situations where you can accept a small risk of even losing some log data (due to low redundancy, scale vertically) and deploy something that is much simpler. Small apps with small volumes, or apps with large volumes, but you are OK with high latency of the solution and you are ok with slow (and cheap) VMs hosting it. It could work well with environments where you prefer to have more of the platform tools as dedicated nodes assigned to separate teams/apps (due to easier charging, or easier handling of noise neighbors, etc).

[1] https://www.youtube.com/watch?v=HdnDXsqiPYo

Struggling with Container HTTP Logs? Meet Logmink

Related Stories