HN user
right now, I am scoping it to nearby people; within X miles distance. Saying strangers look fishy - might have to add more verifications.
How open would you be to trying go out?
anyone has any videos?
lol, that is what we all needed. AI Boss
anything tbh. as long as you have runbook - you can try to automate actions through nvsx; it sits on top of NVSentinel. restarting will work mostly for smaller jobs - distributed training, pretty common will need more fault tolerant methods to continue rather than just restarting.
One of the feedbacks we got while testing was - we might even reduce the duration of checkpointing. Which was a huge insight.
We are testing out & would love more collaboration from the community - if your team is running training jobs, hit us up.