stanhu

There has been a steady march of performance issues that have been made over the last few months that may not have been mentioned in the release post:

* 11.8: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=...

* 11.7: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=...

* 11.6: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=...

With regard to memory usage, we've made some progress there as well. For example:

1. We cut 80-100 MB/per Unicorn process by removing a dependency: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21008

2. A change in our merge request processing in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22725 lowered gigabytes of runtime memory from our worst and most commonly-used Sidekiq background job.

3. In the second-worst performing Sidekiq job, switching to a faster XML processor in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23136 also dropped large spikes of runtime memory.

That being said, we still have a long way to go. I recently did an analysis to figure out, "Why does each Unicorn process take 400-500 MB RAM?" The note in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4118#not... explains why. In short, there is truth that we're paying a price for Ruby overhead.

What can we do about it? I have several ideas:

1. Upgrade to Ruby 2.6 (significant memory improvements for free; see https://youtu.be/ZfgxvNUfQdU)

2. Replace many Unicorn processes with a multi-threaded Puma server

3. Continue profiling and optimizing inefficient code paths

4. Make the Ruby interpreter more efficient (e.g. with a compacting garbage collector, see https://gitlab.com/gitlab-org/gitlab-ce/issues/54555)

5. Make Rails more efficient (e.g. https://github.com/rails/rails/pull/34711).

Lastly, there may be significant parts of the code base that will need to be rewritten to improve memory and/or performance. Gitaly is a prime example of this: many Git-related calls have been abstracted into a Go process.

I agree that memory requirements are too high, but I think there is a lot of room for improvement without a complete rewrite, which would be very costly. Also keep in mind that a standalone GitLab instance ships with all batteries included: PostgreSQL, Redis, etc., all of which add to the memory requirements. That being said, we are certainly looking at moving performance-critical functions into Go. See the GitLab Workhorse and Gitaly projects as examples.

There are two major ways to look at application memory usage: baseline and runtime usage. Most people pay attention to the first because that's what you see when you first deploy GitLab. However, runtime usage is just as important because a running instance could gobble up gigabytes of RAM.

Baseline memory: We've made some recent progress in reducing baseline memory usage (e.g. cutting 80-100 MB per Unicorn process in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21008). However, our Unicorn workers dominate most of the memory usage, and you can see more analysis of where our memory is going in https://gitlab.com/gitlab-org/gitlab-ce/issues/49702. https://brandur.org/ruby-memory is probably the best explanation I've seen why RAM usage in Unicorn processes often rises and doesn't come down. However, we can reduce baseline usage in a number of ways:

* Removing dependencies from the main application

* Reducing any unnecessary allocations at startup

* Moving to a multi-threaded application server (we've shipped Puma in experimental mode in https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/...)

* Improving Rails memory usage (e.g. ActiveRecord optimizations in https://github.com/rails/rails/pull/34711)

* Improving the Ruby interpreter (e.g. helping ship a compacting garbage collector, see https://gitlab.com/gitlab-org/gitlab-ce/issues/54555).

Runtime memory: Most of our runtime memory problems are often due to memory bloat or unoptimized code paths. For example, we put big dent in some of the most egregious background jobs over the last few months. For example, a change in our merge request processing in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22725... lowered 95% of our runtime usage by gigabytes. Switching to a faster XML processor in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23136 also dropped memory usage in another background job.

Obviously we're still far from where we should be, but we're making progress. If you or know anyone else who might be excited to help us, please let us know! https://about.gitlab.com/jobs/ Thanks for your feedback.

Could you show the output of `ps aux` inside the container? Also note that GitLab Omnibus will add more Unicorn processes the more available RAM it has in the system. You can turn off that automatic scaling by lowering the number of processes via https://docs.gitlab.com/omnibus/settings/unicorn.html.

You can sign up for an account to see the bug report status. The patch has not yet been backported.