petrosagg

Materialize | Staff Software Engineer (Sources/Sinks) | NYC or Remote (US/Canada/EMEA) | https://job-boards.greenhouse.io/materialize/jobs/5357900004

Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.

We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.

Note: the linked position is focused on the data integration aspect of Materialize but we're also looking for wider database roles so feel free to apply if the general area of Materialize seems interesting to you!

What we’re looking for:

- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).

- Strong CS fundamentals (B.S./M.S. equivalent).

- Bonus: distributed systems, SQL databases, CDC, or stream processing experience.

Compensation: $200K–$225K + equity.

Backed by Kleiner Perkins and Lightspeed ($100M+ raised), we’re remote-friendly and committed to building a diverse, world-class team.

Materialize | Staff Software Engineer (Sources/Sinks) | NYC or Remote (US/Canada/EMEA) | https://job-boards.greenhouse.io/materialize/jobs/5357900004

Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.

We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.

What we’re looking for:

- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).

- Strong CS fundamentals (B.S./M.S. equivalent).

- Bonus: distributed systems, SQL databases, CDC, or stream processing experience.

Compensation: $200K–$225K + equity.

Backed by Kleiner Perkins and Lightspeed ($100M+ raised), we’re remote-friendly and committed to building a diverse, world-class team.

Materialize | Staff Software Engineer (Sources/Sinks) | NYC or Remote (US/Canada/EMEA) | https://job-boards.greenhouse.io/materialize/jobs/5357900004

Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.

We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.

What we’re looking for:

- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).

- Strong CS fundamentals (B.S./M.S. equivalent).

- Bonus: distributed systems, SQL databases, CDC, or stream processing experience.

Compensation: $200K–$225K + equity.

Backed by Kleiner Perkins and Lightspeed ($100M+ raised), we’re remote-friendly and committed to building a diverse, world-class team.

Thank you for the detailed response!

However, while checksums can be used under the “Crash Consistency Model” to solve consistency through power loss, PAR showed that checksums are not sufficient to be able to distinguish between a torn write at the end of the (uncommitted) WAL caused by power loss, and a torn write in the middle of the (committed) WAL caused by bitrot.

The PAR paper states that "although Crash preserves safety, it suffers from severe unavailability". I assume that when TigerBeetle loads state from RAM into a CPU cache/register it operates under the NoDetection consistency model or the Crash consistency model if ECC RAM automatically resets the CPU on read errors. At the same time it doesn't suffer from severe unavailability so what gives?

The answer is probably that ECC RAM is just reliable enough that the NoDetection/Crash models are fine in practice.

I can believe that off-the-shelf checksum and redundancy options offered by filesystems like ext4 and ZFS or systems like RAID don't hit the required error probabilities but why does the argument stop there? Couldn't a distributed database generate error correcting data on every write in the application layer so that the probability becomes low enough such that NoDetection/Crash become a non-issue for storage, just like RAM? Is there some other fundamental difference between reading and write data from RAM versus a disk?

Sure, we’re not yet injecting storage faults, but then formal proofs for protocols like Raft and Paxos assume that disks are perfect, and depend on this for correctness? After all, you can always run your database over RAID, right? Right? If your distributed database was designed before 2018, you probably couldn’t have done much. The research didn’t exist.

I'm trying to understand this part but something seems off. It seems to imply that those proofs do not apply to the real world because disks are not perfect, but neither is RAM. The hardware for both RAM and disks has an inherent error rate which can be brought down to arbitrary levels by using error correction codes (e.g ECC RAM).

I'm assuming that the TigerBeetle correctness proofs are predicated on perfect RAM even though in reality there is a small probability of errors. This tells me that there is an error rate which they consider negligible. If that's the case what is the difference between:

* TigerBeetle's storage approach

* Paxos or Raft with enough error correction on disk writes that the probability of errors equals that of ECC RAM which is considered negligible

I've probably made a logical error in my reasoning but I can't see it. Can someone enlighten me?

The feature you're referring to is const generics which is different than generic associated types (GATs) which is the feature being stabilized.

min_const_generics have been on stable for while

I must be missing something. The paper describes the algorithm of the CRDT and mentions that "timestamps ’t need to be globally unique and totally ordered".

Then it mentions multiple times that Lamport clocks/timestamp can be used as the timestamp in their system but as far as I know these only give a partial order of events. How is this reconciled in their system?

Here you go https://www.youtube.com/watch?v=WJj_NMhYwf0

The OP link is over https, which appears to work fine but only for redirecting you back to http

transcript: https://www.infoq.com/presentations/netflix-five-level-owner...

Thanks for explaining this

Thanks for explaining this, I have a much better idea about those systems now

Yes, and this problem is exacerbated by the raspberrypi's microUSB power input. We've seen countless number of cases where a Pi exhibiting SD card corruption was also experiencing undervoltage. If you're using raspberrypis in production it's paramount to have a good microUSB cable and power supply

I'd be interested to hear about the failure modes of I2C that you've observed. Unfortunately I'm not familiar with CAN or SPI. Do these alternatives provide error correction as part of the protocol? Do they offer superior reliability in some other way?

balena founder here. balenaOS does use the hardware watchdog available on the raspberry pi to detect CPU lock ups and automatically reset the board. On top of that we're also running software watchdogs that check the health of key system components and restart them if they become unhealthy.

It's true that SD cards are known for getting corrupt. balenaOS separates the partitions of the device and keeps the userspace in a readonly one while keeping mutable OS state in a separate partition. We are very conscious of writing as infrequently as possible to the SD card for this reason. The partition that accepts the most writes is the one holding the user container, which will get written to during an update, and also in case the user container stores any data on the device.

I'm aware that SD cards will internally swap blocks and don't really care about partition boundaries but assuming you're using an SD card with a well designed firmware it shouldn't lose a block during wear leveling.

That said, the SD card problem is one reasons we designed balenaFin :)

balena founder here. balenaOS comes with all the infrastructure needed for robust host OS updates. We expose this functionality to our users via a button in the web dashboard. We don't yet have an automated, rolling upgrade style mechanism.

The main consideration for a feature like this is that sometimes containers have dependencies to interfaces exposed by the operating system which are not always stable. This is especially true for IoT usecases because containers will typically interface with some device connected to the system.

Tangential to this, we're working on an extended support release schedule (a la firefox) for balenaOS. I could see us building an automated OS update mechanism on top of that. We'll definitely think about it, thanks a lot for your feedback :)

Resin.io | Fullstack, frontend, security, golang engineer, Etcher maintainer, software engineer, technical sales lead | remote | full-time | salary + equity

Resin.io provides a software platform that helps developers build, deploy and manage code on connected devices. We also maintain a variety of successful open source projects including etcher.io, balena.io, and resinOS, and make contributions to high-exposure projects such as Docker, Electron, and AppImage.

Fullstack Engineer - https://resin.workable.com/jobs/209405

  You will work on core resin.io features throughout the whole stack on a
  service infrastructure for IoT devices, and solve complex technical
  challenges that have an impact on a worldwide network of GNU/Linux embedded
  devices

  You will develop web and command-line frontends using cutting edge
  technologies (React, Redux, TypeScript), and contribute back to our open
  source UI toolkit

  You will shape the future of container technologies in embedded Linux and
  IoT. You will work in Balena, our open source Moby-based container engine.

  We take process seriously. You will have a crucial role in defining how
  resin.io writes, tests, and deploys code. 

  You will take the lead of the Etcher image flasher, one of the most popular
  open source Electron applications, and make it a core part of the IoT
  provisioning and development workflow. You will also work on Etcher Pro, a
  standalone drive duplicator product based on the Etcher software.

  We run sales like we run engineering. While closing deals is important, we’re
  more interested in helping customers succeed with resin.io and solving their
  technical challenges.

The problem is that when a user refreshes the page using the reload button the browser will revalidate even cached resources. This results in a lot of requests that get carry the `If-Modified-Since` header and 304 responses. Cache-Control: immutable avoids this revalidation altogether since it's guaranteed that the source won't ever change.

No more jQuery from Google CDN, no more analytics/exception tracking/fonts from your favoured provider and no more advertising conversion tracking

You don't have to trust the CDN for any static resource you load as long as you use the subresource integrity feature of modern browsers[1]. You basically include the hash of the content on your main domain and the browser will validate it when it loads it from CDN. So you only need to trust your main site.

[1] https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

Edit: It was already mentioned by michaelmior https://news.ycombinator.com/item?id=12138277

I think the main problem is that there are a lot of SSH servers already deployed without using the CA approach and migrating HostKeys is a pain without causing horrible warnings to the clients.

Take GitHub for example. If this feature was mainstream they would be able to phase out their DSA key without people noticing.

Ah, I just realised what you meant. No, we're not affiliated in any way with the application server.

I'm one of the co-founders

Resin.io - Remote - Full-time Site Reliability Engineer

About us

We are building a seamless code delivery platform for embedded devices (like the Raspberry Pi). Think of it as Heroku for the Internet of Things if you must. We've already ported Docker to ARM CPUs, launched the alpha version of our product are about to release our beta version during this month. We are a well funded company founded and led by developers and have a strong engineering-led culture. As long as there is progress, we are flexible with regard to location and work hours.

About you

We are looking for a Linux talent to work on our cloud deployment which is a constellation of Docker-ised services deployed on AWS. Besides strong Linux skills, familiarity with Docker will help a lot. Much of our codebase (on cloud and device) is in node.js so JavaScript skill is a plus. You will be responsible of our server infrastructure, making sure we have seamless code deployment across all of our services and responding to any incidents and emergencies that may arise.

The Process

Send us your CV (a pointer to source code online is a great plus). Those that pass a phone interview will receive a small but paid project as a test. If we like working with you and you like working with us, you'll be offered a full-time job.

Get in touch

send your CV to hello@resin.io

Actually, you don't have to trust the casino. As pointed out by fryguy there are ways to ensure no party knows the deck [1].

[1] http://en.wikipedia.org/wiki/Mental_poker#Shuffling_cards_us...

For the first method to work the encryption function must be commutative. Are there any commutative encryption functions?

Given that it's a two year old post they may have changed their PRNG.

No, that's only if you "delete" your account through the settings menu. If you fill out the form in the link it's permanent.

You could always permanently delete your account on Facebook.

https://www.facebook.com/help/www/224562897555674

You're right, the instructions are a lot and not trivial but are very useful for the Docker team and hackers that may want to modify the procedure. We are planning to release pre-built binaries and kernel soon though.

Actually performance is suffering if you run it on a system with a not-so-good GPU. Intel GPUs are known for their poor WebGL performance.

The performance with wireframe rendering would be the same. The difference is that Andrew is simulating 150 points in CPU whereas this simulates ~260k in the GPU. On my 2008 laptop with NVidia GPU it runs at 60 FPS.