AirSim: open-source simulator for autonomous vehicles

by jonbaer • 7 years ago

320 53 7 years ago

kuu• 7 years ago

Neat!

therobot24• 7 years ago

looked at using airsim a few months ago to replace a product we'd been using for some time, problem was the actual vehicle dynamics simulation capabilities were subpar

grenoire2• 7 years ago

Did not do a code review but I would not be surprised if they were using Unreal's default vehicle simulation; which, as you noted, is indeed sub-par. It's honestly a very challenging task to get vehicle physics to accurately replicate real cars, and the systems which are out there (commonly used by manufacturers to make advertisement renderings) are proprietary and expensive.

draugadrotten• 7 years ago

If using Unreal, the assetto corsa competizione video game is pretty good for simulations and it offers a commercial license. https://www.assettocorsa.net/assetto-corsa-pro/

As does rFactor2 but it may not be so easy to use with unreal.

http://www.vesaro.com/store/pc/viewPrd.asp?idproduct=130

Stevvo2• 7 years ago

It the demo video it popped up"Advanced Physx car model". Physx is designed for a good enough approximation in games, not an accute simulation.

mey• 7 years ago

There is a physical car designed to be repainted (like a motion capture actor) in CGI. It solves physics by using actual physics.

http://www.themill.com/portfolio/3002/the-blackbird%C2%AE

akoumis• 7 years ago

Airsim also has poor facilities for stepped simulation needed for many RL algorithms (e.g. insert control, advance simulator by 30ms, observe state, repeat). It looks like this is still being worked on: https://github.com/microsoft/AirSim/issues/600

lutorm2• 7 years ago

Weird, I'm surprised that isn't the default mode of operations for something that's designed to plug in your own control system.

Stevvo2• 7 years ago

Awesome; it looks like GTA gone serious.

bArray• 7 years ago

If you are interested in this, I would recommend checking out DroneSimLab [1]. Both are developed for the Unreal Engine and both have SITL (Software In The Loop) for drone control. One thing DroneSimLab will allow you to easily do is add multiple drones. If I remember correctly it uses Gazebo to create semi-realistic control for the drone.

[1] https://github.com/orig74/DroneSimLab

jcims2• 7 years ago

Does this support uav swarms? Would you need one instance per vehicle?

agusgun• 7 years ago

I think it only support 1 uav. The issue have already stated about that: https://github.com/microsoft/AirSim/issues/1540

gexla• 7 years ago

I just ran across Real Engine looking for Architectural Visualization software (note that Twinmotion is free until Nov!) This is an amazing space which I imagine will explode in variety of application, especially once VR goes mainstream.

Are there any open source alternatives which even come close?

I wonder if I would attract attention if I were to upload a rendering of Saudi oil infrastructure for drone simulation. ;)

Can we use this thing as a simulation for shooting drones out of the sky? This could be an interesting massively multiplayer research project creating an arms race between attackers and defenders.

robmiller• 7 years ago

I just ran across Real Engine looking for Architectural Visualization software (note that Twinmotion is free until Nov!)

Thank you!

daenz• 7 years ago

Super cool. I've always been a big fan of generating synthetic training data. As long as you can degrade/distort the data to match real world conditions, you can have your cake and eat it to: accurate inputs with perfect automated labels.

At my last job, we read the contents of receipts and parsed them for purchase information. I was really interested in working there, so I built a docker-based CGI receipt-generator[0] that mimicked real-world conditions of a receipt captured with a camera phone. After getting the job, I expanded it to provide the perfect labelled data of bounding box and character[1]. I then used this data to train custom TensorFlow models.

0. https://github.com/amoffat/metabrite-receipt-tests

1. https://i.imgur.com/gkdnAkt.jpg

legulere• 7 years ago

In medical imaging synthetic training data get called phantom.

https://en.wikipedia.org/wiki/Computational_human_phantom

desideratum• 7 years ago

This is really cool! Nice work. Do you remember how you went about creating the original receipt generator? I'd be really interested in this kind of stuff without any ML/DL.

abetlen• 7 years ago

Not OP but I was just looking through their repo and I've done something similar in the past for a different application. We both used Blender for generating the rendered synthetic data. Blender has a really simple python API that was originally made for simple scripts but you can run it in a headless mode. You start with a base scene that has all of your objects and use the api to randomly generate samples and render them out to a file.

LyndsySimon• 7 years ago

In theory, could I use this (or something like it) to develop algorithms for controlling autonomous vehicles (aerial, ground, and/or aquatic), then if I ever get my experimentation to the point that I’m interested in doing so, building the vehicle itself and deploying it with a reasonable degree of confidence that it would actually work?

jafingi• 7 years ago

There would be a reality gap, going from simulation to real world. Good read: https://ai.googleblog.com/2017/10/closing-simulation-to-real...

jcims2• 7 years ago

AKA 'sim2real' gap for googling purposes.

currymj• 7 years ago

getting things like this to transfer well is not necessarily easy, it’s a major active area of research.

even if you somehow had a perfect physics model, there are problems like the fact that motors and joints behave differently as they wear down, etc.

it might be easier for wheeled vehicles — usually the examples of failure are complicated, error-prone things like bipedal walking, hands with lots of joints.

lutorm2• 7 years ago

Yeah the trick is knowing what the performance of the as-built vehicle will be before building it. You can to some extent marginalize over this by running randomized trials with different parameters for your mechanisms so that you verify that you're not too sensitive to the exact setup. The control system needs to be fairly robust to such variations. But you still don't know that you've modeled all the physics correctly until it's actually flight-proven. Picking the correct degree of fidelity in a simulation model isn't super straightforward.

TaylorAlexander4• 7 years ago

My thinking is that if your sim is kinda realistic, then once it works you can fine tune on some (expensive) real hand labeled data. But the sim helps you develop the training process etc.

jedberg• 7 years ago

If you're interested in this stuff but don't want to spend the time setting everything up, you should check out Amazon DeepRacer[0]. They have a whole environment set up to do reinforcement learning (although right now it's only their tracks). You can get your feet wet with the free tier.

[0] https://aws.amazon.com/deepracer/

pornel• 7 years ago

Slightly worrying that the car simulation doesn't appear to have any cyclists or pedestrians.

uoaei4• 7 years ago

Or pictures of people on the sides of sprinter vans.

Or bikes on trailers on the backs of cars.

Or pickup trucks stuffed to the gills with Lime scooters.

Or...

toxicFork2• 7 years ago

It's good that it's open source then ;)

kuanbutts• 7 years ago

Yeah and in the car video on https://microsoft.github.io/AirSim/, they have the car driving on the wrong side of a barricade on a bus only lane. :(

knolan• 7 years ago

I remember people using Grand Theft Auto V to generate synthetic data. It’s got pretty much all of that.

asdfman123• 7 years ago

I know, I wanted to train a neural-network piloted killer robot car too.

Ithildin• 7 years ago

How is this different than CARLA? Just curious.

cheschire• 7 years ago

For C# devs, there’s unity integration as well.

cultofmetatron• 7 years ago

anyone else want to use this but can't because they use a mac?

kebman• 7 years ago

Flight sim is totally out. Drone sim is what's up now! :D

TaylorAlexander4• 7 years ago

I just downloaded this last weekend! I’m trying to find a good simulator where I can import models generated with photogrammetry. I’ve been using a cell phone camera and OpenSfM + OpenMVS to make 3D models of real world environments, and I want to be able to run them in sim. I see this is built on Unity. In the past I tried UnrealEngine but it was such a big mess of software for simple RL tasks. I’ll have to see if Unity is cleaner.

I’m building a mostly 3D printed four wheel drive Rover vehicle I designed, and I’m using an Nvidia Jetson Xavier to try to make it autonomous using only cameras for sensors. Currently I’ve got some stereo cameras but I’m considering a synchronized 4 X 4K camera rig. Hopefully I can release some scans once I get something good.

The rover is public domain open source so please check it out if you’re curious! It can support a 10-20lb payload with the right springs:

https://youtu.be/ToGT3KokPZA

heyitsguay• 7 years ago

This sounds super cool! Can you say a bit more about your OpenSfM + OpenMVS stack? I've been idly interested in building 3d models from cell phone video for a while.

TaylorAlexander4• 7 years ago

Thanks! Yeah I’m essentially running OpenSfM exactly as described in their documentation usage example. Then on that page they also show how to export to OpenMVS, and from there I follow similar example usage documentation from OpenMVS. OpenSfM has a config file and the defaults are fine, but I’d recommend reading the description for each possible parameter so you know what to tweak if you don’t like the default results. Oh and OpenMVS uses a lot of memory if you use lots of photos, but remember you can use swap. At first I used GCE instances with huge amounts of physical ram and that was a silly waste of money in hindsight. I simply didn’t think of using swap at first.

ClumsyPilot• 7 years ago

This is quite cool, why cameras only though? Intel realsense, ultrasound? Are you avoiding them due to cost?

TaylorAlexander4• 7 years ago

Hello and thanks! No I am not avoiding either of those due to cost. I’ve worked with ultrasonics on a robot before and I found them pretty unimpressive. I have seen a demo of some advanced ultrasonics but the guy also told me he patented the technology. Normal ping pong ultrasonics return such ambiguous information they aren’t very useful. Regarding intel realsense, they have two different technologies. Some realsense devices are stereo cameras, so they are “just cameras” in the sense of this discussion. But I’d rather run the algorithms myself and build a complete perception system than use the built in computation of for example the realsense tracking camera. The other technology they use is structured light, and such technology doesn’t work well or at all in sunny environments. I want to build a complete perception and navigation stack for outdoor robots, and aside from some sensors inside the robot, cameras are the technology I’m most interested in learning how to use. I just think cameras are the future. Passive vision is how most every living thing on earth perceives the world.

mfsch• 7 years ago

That looks interesting, but I wonder how accurate the flight behavior is when they are not including any atmospheric dynamics. The wind in the atmospheric boundary layer is characterized by strong semi-random fluctuations that arise from the turbulent interactions of the air with the surface. These fluctuations occur on scales both larger and smaller than a drone, so they should have a significant effect on its aerodynamics.

__sy__• 7 years ago

Aerodynamic simulation is a really tough problem to take on. I'd venture to guess that it's in part due to not having a complete set of theories/equations around fluid dynamics. From what I hear, X-Plane comes really close to this. But yes, skill-transfer aside, I'd guess any airborne vehicle trained in a simulator that lacks environmental effects, ground effects, induced drag effects...etc in its physics calculation is not going to perform well in the real-world.

alasdair_• 7 years ago

I've been using LiftOff (https://store.steampowered.com/app/410340/Liftoff_FPV_Drone_...) to test my drone handling skills in a simulated environment for a while. It's remarkably accurate and allows for custom drone models to be downloaded that match up very well the the handling of the real drone in question.

Weather, wind etc. are all taken into account.

gkolli• 7 years ago

Any companies working on this?

mijail• 7 years ago

(Disclosure: I work at one)

There’s a few working specifically on simulating the autonomous vehicle domain. air sim is unique because you can do both ground and air. From the business side of things I’ve seen enterprises find it as a good way of proving the value of simulation. The company I work for CVEDIA, develops SynCity which we use both for simulating autonomous systems and generating training data for object recognition and classification tasks.

thomasedwards• 7 years ago

Just looks like an improved Midtown Madness to be honest.

https://en.wikipedia.org/wiki/Midtown_Madness

/s incase it wasn’t clear

timtu7• 7 years ago

What is the practical application of this?

AirSim: open-source simulator for autonomous vehicles

Related Stories