whost49

I hope that I would be able to finish this quickly because I want to focus on my other project (https://github.com/yousseb/atfal-ai & https://github.com/yousseb/atfal-site) which is an effort to track missing children/people and use AI to help find them. I love meld, but as a person who has lost his brother in a way that he couldn't come back - and I keep losing family in Gaza -, I want to help reunite families who still have the chance to be reunited. Everyone is welcome to help.

Drew, thanks for sharing your settings. It looks like you are using a 2GB RAM system.

How many unicorn workers are you using? For 2GB systems, I'd recommend at most 2 rather than the default 3. Some discussion about this is here: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1279

We'll be doing more work in these coming months to profile and reduce the memory usage needed by GitLab so that all your tools can run comfortably in the 2GB range.

For your use case, I think Fluentd may work fine. LogZoom currently deals with structured JSON log data received from hundreds of hosts around the world. It could be modified to handle arbitrary logs (and wrap a structure around it) and integrate with Docker, but that was not the goal here.

Thanks for that information. There's no doubt Logstash can do a lot, and it sounds like with the multiple pipeline feature Logstash will make it easier to do what we wanted to do in a single process.

In the past, we've also been burned by many Big Data solutions running out of heap space that adding more processes that relied on tuning JVM parameters again did not appeal to us.

If you're around I'd love to hear specifically what you mean by this. Internally Logstash is very thread friendly, we only recommend multiple processes when you want either greater isolation or greater fault tolerance.

Right, we considered using multiple Logstash processes, but we really didn't want to run three instances of Logstash requiring three relatively heavyweight Java VMs. The total memory consumption of a single VM running Logstash is higher than running three different instances of LogZoom.

We looked at the Filebeat Redis output as well. First, it didn't seem to support encryption or client authentication out of the box. But what we really wanted was a way to make Logstash duplicate the data into two independent queues so that Elasticsearch and S3 outputs could work independently.

Heka looks good and does a lot more. It doesn't appear to support Redis and S3 out of the box, so we would have probably had to evaluate, learn, and change the third-party plugins had we known about Heka beforehand.

No, we did not consider Gollum--it definitely looks like a possible solution and one we might have considered. I think the name of the project makes it hard to find, unfortunately.

What does Brubeck do that the collectd statsd plugin can't?

https://collectd.org/wiki/index.php/Plugin:StatsD

Is there a reason why you didn't just use collectd, which is written in C and has lots of plugins, including statsd, InfluxDB, Graphite, etc.?

Aclima - http://www.aclima.io - San Francisco, CA

==========

Aclima is an early-stage company based in San Francisco that designs and deploys distributed, large-scale sensor networks to empower people with actionable environmental quality data. Our end-to-end solutions collect, process and analyze real-time streaming data from thousands of sensors, enabling a level of environmental awareness that has never been possible before. We believe our technologies can redefine the way we imagine and manage our buildings, communities, and cities, helping us improve our collective well-being. We are looking for passionate engineers to help build, scale, and improve our platform.

We have no required list of skills or years of experience. Instead, we’re looking for engineers who are smart and get things done. Our engineering culture values rapid iteration, continuous improvement, and as much automation as is sensible. We work in a relaxed, purpose-driven atmosphere with flexible hours and competitive perks.

Positions open:

* Full-time Backend Engineer

Our stack includes: Python, Git, MariaDB, Cassandra, Nginx, NSQ, Redis, Ansible

- solid understanding of functional programming languages, distributed systems

* Full-time Frontend Developer

- solid understanding of core JavaScript, HTML5 and CSS3

- experience building well-structured web applications

- a passion for user-driven interaction design and delightful user experiences

- excitement about data visualization, mobile design, and responsive design

* Full-time UI/UX Designer

- At least 2-3 years of experience who has heavy UX chops and expertise in CSS-based design, peppered with JavaScript-based interaction design/development.

- Someone who is excited and passionate about data visualization, mobile design, and responsive design and has strong opinions about all three.

* Full-time DevOps Engineer

- some experience writing shell and Python scripts

- Debian/Ubuntu, Jenkins, Locust, JMeter, Google Cloud/AWS, Ansible experience a plus

==========

If you’re up for the challenge, contact us: jobs@aclima.io

Apply directly: http://boards.greenhouse.io/aclima

Aclima - http://www.aclima.io - San Francisco, CA

==========

Aclima is an early-stage company based in San Francisco that designs and deploys distributed, large-scale sensor networks to empower people with actionable environmental quality data. Our end-to-end solutions collect, process and analyze real-time streaming data from thousands of sensors, enabling a level of environmental awareness that has never been possible before. We believe our technologies can redefine the way we imagine and manage our buildings, communities, and cities, helping us improve our collective well-being.

We are looking for passionate engineers to help build, scale, and improve our platform. We have no required list of skills or years of experience. Instead, we’re looking for engineers who are smart and get things done. Our engineering culture values rapid iteration, continuous improvement, and as much automation as is sensible. We work in a relaxed, purpose-driven atmosphere with flexible hours and competitive perks.

Positions open:

* Full-time Backend Engineer

Our stack includes: Python, Git, MariaDB, Cassandra, Nginx, NSQ, Redis, Ansible

* Full-time Frontend Engineer

- solid understanding of core JavaScript, HTML5 and CSS3

- experience building well-structured web applications

- a passion for user-driven interaction design and delightful user experiences

- excitement about data visualization, mobile design, and responsive design

* Full-time UI/UX Designer

- At least 2-3 years of experience who has heavy UX chops and expertise in CSS-based design, peppered with JavaScript-based interaction design/development.

- Someone who is excited and passionate about data visualization, mobile design, and responsive design and has strong opinions about all three.

* Full-time DevOps Engineer

- Debian/Ubuntu, Jenkins, Locust, JMeter, Google Cloud/AWS, Ansible experience a plus

==========

If you’re up for the challenge, contact us: jobs@aclima.io

Aclima - http://www.aclima.io - San Francisco, CA

---

* Full-time Backend Engineer

* Full-time Frontend Engineer

* Full-time UI/UX Designer

* Full-time DevOps Engineer

---

Aclima is an early-stage company based in San Francisco that designs and deploys distributed, large-scale sensor networks to empower people with actionable environmental quality data. Our end-to-end solutions collect, process and analyze real-time streaming data from thousands of sensors, enabling a level of environmental awareness that has never been possible before. We believe our technologies can redefine the way we imagine and manage our buildings, communities, and cities, helping us improve our collective well-being.

We are looking for passionate engineers to help build and scale our platform. We have no required list of skills or years of experience. Instead, we’re looking for engineers who are smart and get things done. Our engineering culture values rapid iteration, continuous improvement, and as much automation as is sensible. We work in a relaxed, purpose-driven atmosphere with flexible hours and competitive perks.

If you’re up for the challenge, contact us: jobs@aclima.io