edsiper2

Calyptia | https://calyptia.com | Full-time / Remote (Americas / Europe) preferred

Calyptia is the company behind Open Source project Fluent Bit[0], with a strong DNA in Open Source, we solve Observability challenges around scalability and processing in data pipelines.

We have couple of open roles:

- Engineering

  - Senior Support Engineer (Linux, Kubernetes)

  - Senior DevOps Engineer (Cloud, Kubernetes, CI/CD, Automation, etc)

  - Senior Software Engineer (Golang)

  - Senior Software Engineer (C / Linux - userspace)

  - ABM Demand Generation Manager

  - Sales Development Representative (SDR)

  - Senior Sales Engineer

  - if there is no open role for you but you think you can navigate with us and take Calyptia to the next level, send me an email: eduardo at calyptia dot com (and..tell me why ?)

- https://calyptia.com/careers/

.

[0] https://fluentbit.io

Naming is hard, but please, do not repeat the mistake of many OSS project in the last 20 years calling each project by prefixing the name with the stack/environment involved.

Now a "trending" language can catch the attention, but tomorrow?.. maybe. So the value proposition and starting from it name should be different (if you want adoption).

For my use case, for a rewrite of jq I would expect one thing only: higher performance... show the numbers ;)

Calyptia Inc.| Senior Frontend Engineer - Lead| REMOTE | Full Time

Calyptia is an open-source company that focuses on solving Observability challenges for cloud environments. We are the creators and maintainers of Open Source projects Fluentd and Fluent Bit, which are all about observability (logs, metrics & traces).

We are looking for an experienced Frontend developer that has been a tech-lead before with strong skills in architecture design, TypeScript, React, GraphQL, Git and API services,

We look for a proactive candidate with strong communication skills and able to work in a distributed team.

If you are interested, drop me a message to eduardo at calyptia dot com

First of all congratulations to Valgrind and the team behind it! This is an essential tool that help me personally over the years while developing.

What needs to be done to get Valgrind binaries available for MacOS (M1) ?, from a company perspective we are happy to support this work. If you know who's interest and can accomplish this pls drop me an email to eduardo at calyptia dot com.

ah!

[ERROR ] GitHub rate limit exceeded! If you have not done so already,

[ERROR ] you can tell kraft to use a personal access token when contacting

[ERROR ] the GitHub API. First, visit:

Calyptia | REMOTE | Full-time | https://calyptia.com

Calyptia is the company primary funding the OSS projects Fluent Bit[0] and Fluentd[1], CNCF graduated projects.

We are looking for talented individuals with outstanding discipline to work on solving challenges associated with data processing and scalability. We are looking for candidates for the following teams:

- Backend & Ops: Golang, Kubernetes, CI/CD

- Frontend: Javascript, Typescript (mandatory), React

- Fluent Bit OSS: C, Linux and familiar with networking, event loop, syscalls, protocols, etc.

Feel free to send me your github profile or resume to eduardo at calyptia dot com.

[0] https://fluentbit.io

[1] https://www.fluentd.org

my 5c about your intro: I see you can handle a bunch of technologies but your primary focus and interest in the last time has been on Rust. You might promote yourself as a Rust Developer instead of mentioning all the other tech, so you can get easily filtered for the right job.

msgpack users:

- Fluentd & Fluent Bit

- Microsoft SignalR https://docs.microsoft.com/en-us/aspnet/core/signalr/message...

the following as a 5 min intro video to one of the techniques :

- https://www.youtube.com/watch?v=FP-dNd1xTcs

if you want to learn more: https://yogananda.org/practice

Lua is just great. In Fluent Bit[0] we expose Lua filter capabilities. So end-users can write their own scripts to manipulate record logs and accommodate keys/values easily with a simple programming logic:

- https://docs.fluentbit.io/manual/pipeline/filters/lua

I am happy to see performance is being improved, as of now we stick to LuaJIT, since our use case is pretty simple we have not found a good reason to move away from it.

[0] https://fluentbit.io

Logstash alternatives with Elasticsearch support:

[1] https://www.fluentd.org (Ruby Language)

[2] https://fluentbit.io (C Language)

Log structure is really important, from the examples provided I would suggest the same approach can be used using a full 'logfmt' style, so timestamp and the event type can be set as keys, e.g:

  ts="2019-03-18 22:48:32.990" event="Request started" http_method=POST http_path=/v1/charges request_id=req_123

One interesting use-case here for 'me' is the ability to perform queries in a schema-less fashion and I will do a quick speech on what we are working on Fluent Bit[0] (open source log project), pretty much the ability to query your data while is still in motion (stream processing on the edge[1]). Consider the following data samples in a log file:

  ts="2019-03-18 22:48:32.990" event="Request started" http_method=POST http_path=/v1/charges request_id=req_123
  ts="2019-03-18 22:48:32.991" event="User authenticated" auth_type=api_key key_id=mk_123 user_id=usr_123
  ts="2019-03-18 22:48:32.992" event="Rate limiting ran" rate_allowed=true rate_quota=100 rate_remaining=99
  ts="2019-03-18 22:48:32.998" event="Charge created" charge_id=ch_123 permissions_used=account_write team=acquiring
  ts="2019-03-18 22:48:32.999" event="Request finished" alloc_count=9123 database_queries=34 duration=0.009 http_status=200

  $ fluent-bit -R conf/parsers.conf \
               -i tail -p alias=data -p path=canonical.log -p parser=logfmt \
               -T "SELECT * FROM STREAM:data WHERE user_id='usr_123';" 
               -o null -f 1

  [1552949312.991000, {"event"=>"User authenticated", "auth_type"=>"api_key", "key_id"=>"mk_123", "user_id"=>"usr_123"}]

One of the great things of the stream processor engine is that you can create new streams of data based on results, use windows of time (tumbling) for aggregation queries and such.

[0] https://fluentbit.io

[1] https://docs.fluentbit.io/stream-processing

Yes

https://byuu.org/library/libco/

A really good library that we use is libco[0], it supports different architectures and mechanisms. We use it extensively on Fluent Bit[1] to manage async IO network operations (epoll + coroutines).

- [0] https://byuu.org/library/libco/

- [1] https://fluentbit.io

Since your goal is to have a career switch I would recommend you the following approach:

- Essential books and papers are good as a reference to learn and improve but focus first on hands-on stuff.

- Get involved in an open source project that fits the 'system level' category, learn how it works and try to contribute to it (again, put your hands-on)

- Start with development in C language, a good tool to build would be a TCP client and server (kind of 'echo' program). The goal of this will be to learn: how to compile a C program, basics of memory management (malloc, free, realloc), networking (create a socket, bind, listen, connect), transfer data (write, read) and so on.

- If possible, learn using a native Linux environment.

All of the suggestions above will pay off if you are persistent and keep trying learning by building; of course, getting involved in an open source project is a must if you want to have a successful career.

Getting into open source very early

in my experience I would consider the following:

- Understand what are your takeaways for the audience. Remember it's about the content, keep in mind main areas/topics to cover.

- who's your audience ?, there are a few conferences where going very technical (low level) matters, most of them want to focus on how X works, how to solve Y, etc.

- "Start with why". Yeah, it should sound familiar, check this 5 minutes video: https://www.youtube.com/watch?v=IPYeCltXpxw .

- Note that likely in the first 3-5 minutes the audience will have a > 90% focus on what are you talking about, make sure to take advantage of that time with a good "why" (see above) for engagement.

- Share your presentation slides with a colleague and wait for "honest" feedback.

we are using libco[1] in our project[2]; it provides a good abstraction for architectures and provide different backends using ucontext, setjmp and native register management for x86 and x86_64.

[1] https://byuu.org/library/libco/

[2] https://fluentbit.io

check the GIT commit changes listed in the post, you will see the details there, e.g:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Definitely:

5. The odd Redis persistence model: fork() and fighting the copy-on-write.

FYI: The tutorial guide is missing: http://lcui.org/guide/

is there any big difference with the Debian image provided by Google ?

   launcher.gcr.io/google/debian8

https://speakerdeck.com/petere/how-postgresql-is-tested

HTTP Pipelining should not be the base for a performance benchmark. It's an unrealistic scenario.

To make the benchmark a bit fair, add a graphic of memory usage, pipelining might have a huge impact on resources usage in the server side.

- Treasure Data (current): 1.5 hours a week (100 ppl company)

- Oracle (past): 1.5 hours a week (a few thousands ppl)

Both cases my team and management avoided unnecessary meetings, the 1.5h is for two meetings: 1:1 and team meeting. It's not up to the company size, but instead how optimized your team wants to behave.

note: I am in a SW Engineer position. Not sure about Marketing, HR, Finance..etc.

The whole project is in C, there is a planed Lua support for the future versions that can help to filter/modify records, more news about it in the incoming weeks ;)

The decision about "why C" is: flexibility, performance and adaptability (note that it was originally designed for Embedded Linux targets, but now going everywhere). In order to make things easier for output plugins, every time a set of records needs to be flushed through some output plugin, a co-routine is created so any plugin can yield/resume at any time. For example out_http, out_es and out_forward relies on network I/O, having an event loop and a coroutine associated allows to simplify the plugin development and state management: connect, write, read, etc. This model is the foundation and allow the next step to integrate scripting more smoothly. For environments without co-routines support (old compilers), a POSIX thread model exists.

What are the specific "fragility"/concerns you see in Fluentd plugin model?

For the last year we have been working in a lightweight log shipper product called Fluent Bit[0]. Originally made for Embedded Linux now is taking it place in common environments.

It's pretty similar to Fluentd in architecture, some features are:

- Event-Driven (async network I/O).

- Input / Output plugins.

- Data routing based on Tags.

- Optional SSL/TLS for networking operations when required.

Next major version 0.9 will come with buffering support (memory/file system). Ah, it's fully made in C.

[0] http://fluentbit.io

http://fluentbit.io/documentation/0.8

http://github.com/fluent/fluent-bit

please do a favor:

- rename tarballs to Marp-0.0.8...tar.gz

- put the code inside a main directory

As someone said, it's time to move on but take it easy. Try to reduce your "extra working hours", at least try to get enough free time and make sure weekends "are weekends". You need time for you.

Now it's time to start thinking "what would you like to do next" and invest some time in a daily basis looking for companies and positions that you will feel motivated to apply. That research and process may take some time.

Ah, don't get into the "Engineer -> Senior Engineer -> Manager" thinking, titles are not important, what really matters is what you do and how much you enjoy it. If you enjoy what you do, you will succeed.