HN user

notemaker

183 karma
Posts4
Comments100
View on HN
Vim Racer 2 years ago

I've had a similar idea on my backburner for two years, as a fun project to dabble in Elixir, but this is just so much more _fun_ than my version which looks like an admin page that just happens to have a Vim canvas on it.

Well done!

In hindsight, yes that was a bad move (especially considering that my work laptop is still locked to my banned ID…)

As an Apple noob at the time, I assumed that if my MDM-managed device prompted me to log in with my Apple ID, that it of course would be an allowed action.

With regards to data being shared, the only thing I noticed was wifi passwords and peripherals pairing (apple keyboard).

True, but Apple specifically in my case was abhorrently indifferent to the consequences and their support structure is entirely unable to help you. If you have a dedicated storage provider, you lose their single service only. With Apple, you lose everything you have attached to your account which carries a much higher risk.

See https://skogsbrus.xyz/dont-put-all-your-apples-in-one-basket... for more info.

Read Deep Learning with Python [1]. It's ok not to understand everything, but doing the labs will be invaluable.

That book should set you up with the fundamentals. Pytorch is the defacto standard right now for training; during deployment you will use whatever your deployment setup allows (e.g. embedded devices typically have their own inference frameworks).

Python is the lingua franca for anything in machine learning, but other languages are used where necessary for performance or for ecosystem benefits.

[1] https://www.manning.com/books/deep-learning-with-python

I'd recommend moving to development that interfaces with ML instead of moving to being an ML practitioner. You have much faster feedback cycles, your work is predictable (engineering, not science), and you don't feel the pressure of never reading enough papers.

To do that you only need to understand the fundamentals of tensors, some basic knowledge on what the big no-nos are within ML development so you can course correct your peers if they break them, and either focus on the operations side of things or deployment. In both cases, having a knack for optimizing bottlenecks will be very helpful since they will be present during both training and inference.

Tacit programming 2 years ago

I've never found this style readable unless with pipes (bash / elixir), where I love it. With any other syntax, I find it just adds mental overhead. Maybe because you have to read it backwards?

Pretty happy with my existing workflow. It started out with a photo organization script a couple of years ago that I ran manually and has now developed into a fully automatic system.

1. syncthing uploads photos from my phone to my NAS (receive only)

2. scheduled systemd service (hourly) on my NAS reads newly received photos, renames them by date, moves it to my photos catalog

3. photoprism starts a scan of the photo library (triggered by systemd when 2. finishes)

4. weekly systemd service syncs the library to backblaze

This is a great talk that answers many of your questions, https://youtu.be/JvBT4XBdoUE

I am so far merely an enthused observer, so I can't give any insights into the experience of delivering business software with it. Here's my observations though

- Engineers that work with this stack love it. The two most common issues I've seen are lack of static typing and difficulty in hiring

- Great for IO bound tasks due to collaborative scheduling. For CPU bound tasks performance is better offloaded to e.g Rust (see rustler)

- Exception handling is done by processes having supervisors. My mental model for this is similar to pods & deployments à la K8s, but at any abstraction level you want inside your component.

- Elixir pleases many crowds: ruby, FP, performance, resilience

- Phoenix (an elixir library) is seen as a next step from ruby on rails.

- Concurrency model is considered to be simple & powerful

It is meaningful, but maybe I didn't give enough context.

The service ingests data from a timespan T spanning M minutes. Before the next timespan T+1 is ingested, the service has to process the data from timespan T in <M minutes otherwise we fall behind. What I mean with "faster than real-time" is that it has to process the data faster than it is ingested.

I've written a couple of C-extensions for a Python service that needed to run in real-time. We started out 100x slower than real-time and managed to make it slightly faster than real-time in a hectic couple of weeks.

The main bottleneck that C-extensions solved for us then was marshaling. Instead of doing something like Numpy --> custom algorithm --> Numpy --> custom algorithm in Python, we moved it all to C. This way, we decreased the number of times we had to marshal data & we reduced the amount of data that needed to be marshaled.

IIRC we gained 10-100x improved performance in all scenarios where we used C-extensions. Before doing so we tried to use Numba, as it seemed to offer a "free lunch", but we never got it to work.

The final cherry on top that let us surpass real-time speed was replacing a graph algorithm with another that had a better time complexity for larger graphs. Before doing that switch, we'd still be faster than real-time for _most_ inputs but would lag behind severely whenever we had a dense input graph come in.

They update, look for, and provide documentation. They realize the value of good documentation.

They review their own PRs, with a focus on making them easy to review.

They are vocal on public channels and aren't afraid to ask questions outside their team - making sure that communication that could be relevant for other people isn't sent over DMs.