HN user
Hi HN, I hope you enjoy our research preview of interactive video!
We think it's a glimpse of a totally new medium of entertainment, where models imagine compelling experiences in real-time and stream them to any screen.
Once you've taken the research preview for a whirl, you can learn a lot more about our technical work behind this here (https://odyssey.world/introducing-interactive-video).
All of this is really great work, and I'm excited to see great labs pushing this research forward.
From our perspective, what separates our work is two things:
1. Our model is able to be experienced by anyone today, and in real-time at 30 FPS.
2. Our data domain is real-world, meaning learning life-like pixels and actions. This is, from our perspective, more complex than learning from a video game.
Great questions!
Why are you going all in on world models instead of basing everything on top of a 3D engine that could be manipulated / rendered with separate models?
I absolutely think there's going to be super cool startups that accelerate film and game dev as it is today, inside existing 3D engines. Those workflows could be made much faster with generative models.
That said, our belief is that model-imagined experiences are going to become a totally new form of storytelling, and that these experiences might not be free to be as weird and whacky as they could because of heuristics or limitations in existing 3D engines. This is our focus, and why the model is video-in and video-out.
Plus, you've got the very large challenge of learning a rich, high-quality 3D representation from a very small pool of 3D data. The volume of 3D data is just so small, compared to the volumes generative models really need to begin to shine.
Additionally, curious about what exactly the difference between the new mode of storytelling you’re describing and something like a crpg or visual novel
To be clear, we don't yet know what shape these new experiences will take. I'm hoping we can avoid an awkward initial phase where these experiences resemble traditional game mechanics too much (although we have much to learn from them), and just fast-forward to enabling totally new experiences that just aren't feasible with existing technologies and budgets. Let's see!
is your hope that you can just bake absolutely everything into the world model instead of having to implement systems for dialogue/camera controls/rendering/everything else that’s difficult about working with a 3D engine?
Yes, exactly. The model just learns better this way (instead of breaking it down into discrete components) and I think the end experience will be weirder and more wonderful for it.
If I had to choose one, I'd easily say maintaining video coherence over long periods of time. The typical failure case of world models that's attempting to generate diverse pixels (i.e. beyond a single video game) is that they degrade to a mush of incoherent pixels after 10-20 seconds of video.
We talk about this challenge in our blog post here (https://odyssey.world/introducing-interactive-video). There's specifics in there on how we improved coherence for this production model, and our work to improve this further with our next-gen model. I'm really proud of our work here!
Compared to language, image, or video models, world models are still nascent—especially those that run in real-time. One of the biggest challenges is that world models require autoregressive modeling, predicting future state based on previous state. This means the generated outputs are fed back into the context of the model. In language, this is less of an issue due to its more bounded state space. But in world models—with a far higher-dimensional state—it can lead to instability, as the model drifts outside the support of its training distribution. This is particularly true of real-time models, which have less capacity to model complex latent dynamics. Improving this is an area of research we're deeply invested in.
In second place would absolutely be model optimization to hit real-time. That's a gnarly problem, where you're delicately balancing model intelligence, resolution, and frame-rate.
I agree! Check out outputs from our next-gen world model here(https://odyssey.world/introducing-interactive-video), featuring richer pixels and dynamics.
Hi! CEO of Odyssey here. Thanks for giving this a shot.
To clarify: this is a diffusion model trained on lots of video, that's learning realistic pixels and actions. This model takes in the prior video frame and a user action (e.g. move forward), with the model then generating a new video frame that resembles the intended action. This loop happens every ~40ms, so real-time.
The reason you're seeing similar worlds with this production model is that one of the greatest challenges of world models is maintaining coherence of video over long time periods, especially with diverse pixels (i.e. not a single game). So, to increase reliability for this research preview—meaning multiple minutes of coherent video—we post-trained this model on video from a smaller set of places with dense coverage. With this, we lose generality, but increase coherence.
We share a lot more about this in our blog post here (https://odyssey.world/introducing-interactive-video), and share outputs from a more generalized model.
One of the biggest challenges is that world models require autoregressive modeling, predicting future state based on previous state. This means the generated outputs are fed back into the context of the model. In language, this is less of an issue due to its more bounded state space. But in world models—with a far higher-dimensional state—it can lead to instability, as the model drifts outside the support of its training distribution. This is particularly true of real-time models, which have less capacity to model complex latent dynamics.
To improve autoregressive stability for this research preview, what we’re sharing today can be considered a narrow distribution model: it's pre-trained on video of the world, and post-trained on video from a smaller set of places with dense coverage. The tradeoff of this post-training is that we lose some generality, but gain more stable, long-running autoregressive generation.
To broaden generalization, we’re already making fast progress on our next-generation world model. That model—shown in raw outputs below—is already demonstrating a richer range of pixels, dynamics, and actions, with noticeably stronger generalization.
Let me know any questions. Happy to go deeper!
We've been working on the prediction problem for some time. We should have a blog post coming out about our approach (it's pretty neat and in-line with this post) soon.
Voyage | Palo Alto, CA | Full-Time | Visa | https://voyage.auto
Voyage’s mission is to super-charge communities with driverless vehicles. Our fleets power essential, everyday services designed to enhance each resident’s quality of living. At Voyage, we strive to become part of every community we serve. Voyage’s first product is an autonomous taxi service located within a 160,000 resident retirement community in Florida. Here, our fleet delivers on the promise of autonomous driving - solving the mobility needs of residents who need it most. Whether a resident faces mobility restrictions, or just wants to take a ride, we take pride in getting every Voyage passenger to their destination safely, efficiently, and affordably.
We're a team of 40 engineers that have raised $23m from world-class VC's to build a massive and meaningful transportation company. We're growing the team rapidly, and are searching for engineers across multiple disciplines (machine learning, robotics, consumer software, devops, and more). If you love to ship, I think you'll love working at Voyage.
We are based in Palo Alto, CA. We are also open to remote work for the right person! Please refer to oliver@voyage.auto.
Unfortunately not just yet, but we are open to remote work for the right folks. Ping me! oliver@voyage.auto
Ping me when you're ready to start a job search in the space :) oliver@voyage.auto
LIDAR all the way. High-resolution, long-range physics-accurate depth is so crucial.
We will one day have a fully autonomous car with just a few cameras, but we may also have ultra-low cost high-performance LIDAR then.
Hello!
What makes Voyage different?
We are going to market with autonomous vehicles in a very different way, focusing on large private cities first and foremost. We intend The Villages, Florida to be the first (retirement) city that's traversable end-to-end (all 750 miles of road) in an autonomous vehicle.
We'll eventually make the leap to public cities, and it will feel gradual when it does happen.
We think about our technology quite differently, leaning on lots of partners for the infrastructure (mapping, simulation, sensors, tele-operation, middleware, and more) behind the scenes. This enforces a real focus on the un-solved autonomous algorithms.
We'll also be sharing later this year a project we're in the middle of that's dramatically different technologically to what we've seen elsewhere, utilizing the community itself to make a leap in autonomous performance.
From what I understand, you pick canonical routes inside private communities.
We design our autonomous systems to traverse _any_ point-to-point route within an entire private (retirement) city. We intentionally don't just focus on a single, shuttle-like route. It turns out that pretty much any route in a place like The Villages is far less complex than other city-like environments, but that the business opportunity is just as large.
What prevents Google from coming in and mapping the area in a week and run you out of business?
Voyage has exclusivity clauses in our agreements with our communities, where we also grant the community a slice of Voyage in the form of equity. Contracts are unfortunately meant to be broken, which means that we put a lot of effort into making sure relationships with these locations are great. We frequently host Town Halls and make sure the community is heard. This is crucial.
I'm a self driving car engineer, why would I pick Voyage over other big players who have a lot more capital and much bigger team with a lot more people like Drew Gray?
It's a lot of fun here. Contrary to the hype, there's relatively few full-stack self-driving car startups at the Series A level. We believe our people, our technology, and go-to-market to be the best of that group.
Most importantly, when searching for new Voyage team members, we don't optimize for specific degrees or backgrounds. One of our greatest strengths is the team we've built with that philosophy.
Exactly!
Come for a ride (oliver@voyage.auto)! I think you might be impressed with where our technology is. It's really quite good. A lot has happened since August 2017.
We're heavily focused on a retirement city in Florida (125,000 residents on 750 miles of road) and on our G2 vehicle[1]. We recently signed a deal with Enterprise to commercially lease many, many more vehicles than our three initial prototype G1 vehicles.
1. https://news.voyage.auto/introducing-the-voyage-g2-autonomou...
Hello! Great questions.
1) If you already have a good grasp of Python, I always advise to start with the AI for Robotics MOOC at Udacity, which is my favorite class of all time. Once that's done, I'd take a look at their Deep Learning classes and the Self-Driving Car Nanodegree.
2) I think it's crucial to get to grips with how the whole stack works, so I always advise to get to grips with a middleware like ROS. Also, don't be afraid to dabble in algorithms (think problems in motion planning, computer vision, etc.)
3) The traditional programs (think PhD programs) create a lot of specialists focused on a single domain, but the industry is in dire need of more generalists. An engineer who is able to dive into any part of the stack is a huge value-add!
Hello everyone! @olivercameron, CEO of Voyage here.
Drew is currently busy at our Testing Grounds shipping a new release, but if there's any Voyage or self-driving car related questions I can answer, I'd love to hear em'!
My previous life was at Udacity, where I spent 4 years working on their self-driving car and machine learning curriculum. I learned a ton from working with Sebastian Thrun and the rest of the Udacity team.
Voyage | Palo Alto, CA | Full-Time | Visa | https://voyage.auto
Voyage’s mission is to super-charge communities with autonomous vehicles. Our fleets power essential, everyday services designed to enhance each resident’s quality of living. At Voyage, we strive to become part of every community we serve.
Voyage’s first product is an autonomous taxi service located within a 160,000 resident retirement community in Florida. Here, our fleet delivers on the promise of autonomous driving - solving the mobility needs of residents who need it most. Whether a resident faces mobility restrictions, or just wants to take a ride, we take pride in getting every Voyage passenger to their destination safely, efficiently, and affordably.
We're a team of 30 engineers that have raised $23m from world-class VC's to build a massive and meaningful transportation company. We're growing the team rapidly, and are searching for engineers across multiple disciplines (machine learning, robotics, consumer software, devops, and more). If you love to ship, I think you'll love working at Voyage.
To start, I wanted to share more about the Velodyne VLS-128 LIDAR. As far as I'm aware, this is the first 128-channel LIDAR on the market, and it really is nuts.
• 9.6M points per second
• ~300 meters of range
• 360° coverage
Long-range and high-resolution LIDAR has historically been a unique advantage for Waymo, and my feeling is that the VLS-128 dramatically closes the gap for the rest of the field to have the same quality of vision.
Some video:
1. https://cdn-images-1.medium.com/max/2000/1*1qQPsYeu8suHJn412...
2. https://cdn-images-1.medium.com/max/2000/1*D13g1k2Ugq0NJJ3CN...
We pride ourselves at Voyage on being open about our technology and process. This post reads a little like a press release, but I'd love to answer any candid questions about Voyage or the autonomous vehicle field. I'll be here all afternoon!
We take our responsibility with any passenger _very_ seriously. We strive to be world-class in how we operate 100% of the time, and we learn everyday from the best customers we could find as to how we should handle situations like these.
No one has figured this out yet, so we have some ways to go, but rest assured we take our responsibility super seriously.
1) It's not an immediate priority for us, and not really a strength. We are partnering with an amazing company for our second generation vehicle that will go a long way towards this, though. In general, we are excited to see LIDAR sensors become slimmer.
2) One of our engineers builds and maintains Carloop.io, which I can verify as awesome!
3) Safety and performance. One project to keep an eye on is ROS2, which is prioritizing development of real-time features (RTOS, DDS support).
We’re ready! You can do those sorts of rides today (totally driverless) entirely legally in Florida and Arizona.
We design our algorithms to not be necessarily reliant on more data for better performance. We recognize that fleet size and large-scale data collection just isn't going to be a key strength for Voyage, so we design our algorithms accordingly. For example, our classification stack required only 9,000 point cloud frames to achieve pretty amazing performance.
Thank you Dhruv!
Hi marcell, great questions.
1) Yes, at least today. We call them Operations Specialists, and they're highly trained in operating autonomous vehicles and customer service.
2) This is a tough one, since it's not apples-to-apples. That said, Waymo is clearly the industry leader in this regard, and we're glad that at least some of their formula is observable from the outside. We've been at this a little less than a year, and today many of our passenger rides are 95+% autonomous. A long way to go, though!
3) We love these communities for a number of reasons:
A) Speed limits in these communities tend to be a little more restricted than everyday public road. The faster you (and others) can go, the more complex it becomes. The roads are also spectacularly well-maintained.
B) Customer acquisition is a non-trivial problem, and with these sorts of partnerships we gain (exclusive) access to 125,000 potential riders. It would be brutal for us to achieve the same in San Francisco, and not necessarily cost effective to retain them.
C) If we wanted to try out concepts that may accelerate the timeframe to remove the Test Driver, a private community is easier to collaborate with.
D) Above all else, there's just a lot of demand for what we offer in these sorts of communities. Self-driving cars will lower the cost of transportation, and for those with vision impairments (or Parkinson’s, Alzheimer’s etc.), it enables such residents to get around with less friction.
A typical ride is 95% autonomous.
A long way to go for sure, but having been at this just less than a year, we're feeling good!
It depends on how fine-grained we're talking. If you rely on your map for absolute perception (every object must exist in your map in its current real-world state), then yes, you're in trouble. If you rely on the map as a basic prior, and perceive the world through different means, then a map (to me) is the right way to go. You avoid a whole category of computer vision in doing so.
The good news is that a point cloud map is relatively trivial to create, and the pipelines to update such a map over time are becoming more understood. There's lots of startups taking on this challenge.
Thank you! We think so too.
Correct! We think people would be surprised at how big some of these private communities are, and how many places look like them other than retirement communities.