d4l3k

We're working on making these composable. torchft is largely focused on the model integration and algorithms where as Monarch is handling more of the orchestration/monitoring. They operate at a bit of a different layer but the plan is to have torchft have the fault tolerant algorithms that can be used both in Monarch or a standard PTD job

We want to be tolerant to application bugs and host/GPU failures that can be solved by replacing/restarting the machine. External services and network failures we don't have much control over so aren't aiming to solve that.

For specific types of failures check out the section on "Reliability and Operational Challenges" from the Llama 3 paper https://ai.meta.com/research/publications/the-llama-3-herd-o...

Let me know how it goes! If you're interested in chatting / run into any problems feel free to reach out via the links in my profile

Recently there's been a lot of interest and improvements in semi-synchronous training. The Streaming DiLoCo paper came out this year and is a big step forward for datacenter semi-sync.

Historically it's been limited to areas like federated learning for low power/low network training but with the massive increase in number of GPUs it's becoming relevant even for training in datacenters.

It is another variable ML researchers have to tune so does add some complexity and I expect most folks just aren't familiar with it yet.

On "typed language": all of torchft is typed! The coordination/quorum layers are written in Rust w/ GRPC and the front-end is typed Python with Pyre since it has to interact with PyTorch and model code.

Hey Tim, how's it going?

Interested in lending PyTorch some compute? :)

torchft can handle much larger scales but for public multi-day demonstration run this is what we had available. Point of this blog was to demonstrate correctness of the quorum algorithm and recovery with a stock PyTorch stack and not so much peak flops.

Stay tuned though -- planning on doing some much larger demos on B200s!

Hey, nice to see this here!

I'm the primary author so happy to answer any questions you might have!

It seems to work just fine with SIP enabled. I just switched and it seems to be a lot better than Amethyst. Amethyst had a lot of issues with focus follows mouse and dropdown dialogs that seems to just work with Yabai

Seems like SIP is only needed for system dialogs etc so has the same limitations as Amethyst

Not sure why they aren't loading, seem to be fine now

They're also at https://github.com/d4l3k/fn.lc/tree/master/static%2Fdiy-self...

Supercharger auth is between the car and the charger and doesn't require an internet connection. I get billed the normal way via my Tesla account since the VIN is registered

There's some functionality loss but it's mostly been mitigated. I have a custom app I wrote since I can't use the stock app.

The one feature I miss is that there's no voice commands since that requires Tesla's servers but at the same time I also haven't been bothered enough to plug in a custom backend

I've got a blog if you're interested haha https://fn.lc/post/

I've been hacking on my car and creating my own self driving models

Code is at https://github.com/d4l3k/torchdrive

If the map can't talk to Tesla it'll use Google maps directly. I usually don't allow connections to Tesla on my rooted Model 3

I just tried to set this up and couldn't. Seems like it's invite only with a waitlist :/

Adding port forwarding to Mosh has a $600 bounty -- highest OSS bounty I've ever seen

https://www.bountysource.com/issues/4471419-ssh-port-forward... https://github.com/mobile-shell/mosh/issues/337

They measure torque on the wheel from the drivers hand. It is possible to fool via defeat devices etc (ex www dot autopilotbuddy dot com).

There is a WIP system that uses the selfie camera to monitor the driver but it's still possible to fool (image taped in front or block it with tape etc) so unlikely it can catch all cases of drivers being willfully being dangerous. https://twitter.com/greentheonly/status/1379928419136339969

They are working on a camera based solution though it's imperfect. You can see examples of it running at https://twitter.com/greentheonly/status/1379928419136339969

Post can also be accessed via IPFS https://gateway.ipfs.io/ipns/Qmea45XwFtdwaCGAPLRMxFmoUP5YLnk...

I've spent a lot of time digging into the Tesla software and hardware. Let me know if you have any questions! :)

I've spent a lot of time digging into how the Model 3 works internally (not just the security aspects) so happy to answer any questions you might have! :)

The post is also available on IPFS at:

https://gateway.ipfs.io/ipns/Qmea45XwFtdwaCGAPLRMxFmoUP5YLnk...

Pretty much all of the things you'd want in a relational database are now present in Mongodb too.

The real benefit of MongoDB at this point is the ability to easily scale beyond a single machine with shards and high availability using replica sets.

Postgres will get you pretty far, but beyond a certain point the scaling story breaks down and you have to hack some sort of user space sharding solution. At that point all the schema update and backups become a nightmare.

Schemas: https://docs.mongodb.com/manual/core/schema-validation/

Transactions: https://docs.mongodb.com/manual/core/transactions/

SQL: https://docs.mongodb.com/bi-connector/master/

They recently added multi document transactions though.

https://docs.mongodb.com/manual/core/transactions/

The tracing Python objects technique is very similar to how pytorch traces models to produces a graph representation that can be executed entirely in C++ for performance reasons.

The pytorch one is a little bit more general since it can trace arbitrary models + have custom high level operations that might not be supported by default. You just pass it a python method and an example input.

  import torch
  def foo(x, y):
      return 2*x + y
  traced_foo = torch.jit.trace(foo, (torch.rand(3), torch.rand(3)))

Seems pretty clear https://developers.facebook.com/blog/post/2018/12/14/notifyi...

Firefox has almost the exact same thing. https://support.mozilla.org/en-US/kb/how-search-from-address...

It still wouldn't get the drug to the people who need it. Most of the cost involved is in getting it approved by the authorities for human use.

There's nothing stopping people from manufacturing the drug for personal use, other than that it's likely extremely hard to make.

Pixel trackers are just small embedded images in the email that when they are fetched from a server, the server logs it. Pretty simple way to track whether or not someone has opened an email. It's my understanding that gmail however downloads all images and then serves them from Google servers to avoid this issue entirely.

Traditional email clients like Outlook/Thunderbird are susceptible to this kind of attack. That's why they often ask you before loading images.

https://smallbusiness.chron.com/set-email-tracking-pixel-493...

Oh, that's pretty neat! I hadn't seen that before. Does dnslink work with browsers at all?

The nice thing about React Native is that you can always write native code if necessary. Though, I've used plenty of laggy native apps, so it's more about the software engineering behind it than the technology used.

Additionally, with JS most code runs on the UI thread unlike most native apps. If you use a web worker with RN you'd get similar behavior to your native app.

https://github.com/devfd/react-native-workers

The issue with just sharing the /ipfs/ version instead of /ipns/ is that I can no longer update the site and I just made a small edit to clarify some stuff so that link is already out of date.

As an alternative, I'd recommend either my website: https://fn.lc/post/dropbike/ or the cloudflare AMP cache https://fn-lc.amp.cloudflare.com/c/s/fn.lc/post/dropbike/

You can also access it via the normal website: https://fn.lc/post/dropbike/ (or via AMP cache: https://fn-lc.amp.cloudflare.com/c/s/fn.lc/post/dropbike/)

I've had IPFS support on my website for a while now and figured since one of the top HN posts was about porting a blog to IPFS I'd share the IPFS link.

Oh, the guy from support was fairly polite. I cut down his responses a bit for the sake of brevity but I'll clear that up.

For example the initial exchange was:

Hey Tristan, thanks for reaching out to us about this. We really appreciate you telling us about this. We don't have a bug bounty scheme, but I've forwarded your feedback to our software team. Thanks again, Tristan.