Lane Following Autopilot with Keras and Tensorflow

https://wroscoe.github.io/keras-lane-following-autopilot.html

by yconst • 10 years ago

305 72 10 years ago

wroscoe.github.io

This is nice work, but anyone wanting to try it for themselves should be warned that you shouldn't unpickle data received from an untrusted source.

https://blog.nelhage.com/2011/03/exploiting-pickle/

str33t_punk• 10 years ago

glad to see this as the first comment

kordless• 10 years ago

Anti-pickler.

wroscoe10• 10 years ago

I copied this method of loading datasets from Keras. https://github.com/fchollet/keras/blob/master/keras/datasets.... What's a better alternative.

Drdrdrq2• 10 years ago

Another serialization format which doesn't create objects, like JSON, XML, CSV,...

Cyph0n3• 10 years ago

Nicely done! But I'm assuming that this is more of an exercise rather than a real-world application of ML? I say this because the task of keeping a car between two lines is trivially done using control algorithms. Of course, the CV part -- "seeing" the lines -- requires some form of ML to work in the real world.

webaba5• 10 years ago

Obviously, "Lane Following Autopilot using my brain and controls theory" would not make it to the top of HN. Welcome to the new era where Tensorflow replaces Lyapunov and ML spares you the need of understanding hard problems... until you need guarantees and safety... but but it's ok let's add more data.

gugagore4• 10 years ago

I agree with you. If you can leverage control theory from the 1950s to solve your problem, what's the point?

However, I will state that using e.g. Lyapunov functions to prove the stability of the system requires a model of the system. And even if you need a guarantee for your system, that guarantee is only as good as the fidelity of your model. For an inexpensive RC car, with slippage and saturation, without torque control or inertial sensing, you're going to have a hard time doing something that sounds as principled as what you suggest.

yessql• 10 years ago

Why learn to walk when crawling is effective? When crawling you have hard guarantees that you won't fall down.

amelius• 10 years ago

Also, general autonomous driving isn't as hard as it sounds. Given a list of time-dependent coordinates of obstacles, it should be pretty easy to navigate around such that no collision occurs. The hardest part is testing, but this is just a matter of tedious work and doesn't require great intellectual effort.

aub3bhat4• 10 years ago

Rather than demeaning someone's effort, learn why specific methods are considered appropriate/state-of-the-art when solving certain problems.

This is an unbelievably wrong comment. All the Lyapunov and traditional Process control theory in the world won't help you solve autonomous driving. Also regarding "Guarantees and Safety" they don't magically appear out of thin air when you use traditional process control especially in noisy domains like autonomous driving. This comment is equivalent of "I can write code to solve Atari Pong in any programming language deterministicly so any post showing Deep Reinforcement Learning is stupid"...

seertaak2• 10 years ago

trivially done using control algorithms

Is it really trivial? Honest question... Which control algorithms are you speaking of?

gumby• 10 years ago

umm...say, a PID loop?

Cyph0n3• 10 years ago

As u/gumby said, I was thinking of a PID controller. Basically, the car would continuously measure how far away it is from the line and compare that with the "expected" (computed) value. Based on the error between the two values, the controller would adjust some variable (e.g., wheel angle).

Computing how much of an adjustment is required is where the PID part comes in. The controller uses the Derivative (rate of change) of the error as well the the Integral of the error to improve its estimate. These two values can intuitively be thought of as the predicted error and history of the error, respectively.

[1] https://www.wikiwand.com/en/PID_controller

dbecker• 10 years ago

This is primarily a fun toy problem.

It uses a Raspberry Pi and ~50 lines of code. So I don't think anyone should expect it to do something that's impossible with other approaches.

cr0sh3• 10 years ago

But I'm assuming that this is more of an exercise rather than a real-world application of ML?

While this example is simplified - and I wouldn't recommend it for a real-world full-size vehicle trial - it does implement everything (scaled down) described in NVidia's paper:

https://images.nvidia.com/content/tegra/automotive/images/20...

In short, the project uses OpenCV for the vision aspect, a small CNN for the model, uses "behavioral cloning" (where the driver drives the vehicle, taking images of the "road" and other sensor data like steering - as features and labels respectively - then trains on that data), and augmentation of the data to add more training examples, plus training data for "off course" correction examples...

If you read the NVidia paper, you'll find that's virtually all the same things they did, too! Now - they gathered a butt-ton (that's a technical measurement) more data, and their CNN was bigger and more complex (and probably couldn't be trained in reasonable time without a GPU), plus they used multiple cameras (to simulate the "off-lane" modes), and they gathered other label data (not just steering, but throttle, braking, and other bits)...but ultimately, the author of the smaller system captured everything.

Furthermore, NVidia's system was used on a real-world car, and performed quite well; there are videos out there of it in action.

This is virtually the same kind of example system that the "behavioral cloning" lab of Udacity's Self-Driving Car Engineer Nanodegree is using. We're free to select what and how to implement things, of course, but I am pretty certain we all understand that this form of system works fairly well in a real-world situation, and so most of us are going down the same route (ie, behavioral cloning, cnn, opencv, etc). Our "car" though is a simulation vehicle on a track, built using Unity3D.

Of course, the CV part -- "seeing" the lines -- requires some form of ML to work in the real world.

Actually, it doesn't. The first lab we did in the Udacity course used OpenCV and Numpy exclusively to "find and highlight lane-lines" (key part was to convert the image from BGR to HSV, and mask using the hue). No ML was required.

That said - I wouldn't trust it for real-world vehicle driving use - but it possibly could be used as part of a system; however, as NVidia has shown, a CNN works much better, without needing to do any pre-processing with OpenCV to extract features of the image - the CNN learns to do this on its own.

wroscoe10• 10 years ago

Yep, this was an exercise to compete in the DIYRobocars race in West Oakland last weekend. There were 9ish cars with 7 running end to end Tensorflow autopilots and the others using OpenCV/line detection. Open CV one the race.

agumonkey• 10 years ago

I remember seeing keras in commaai source. Who else uses it ?

wroscoe10• 10 years ago

Tensorflow anounced last week that it will incorporate Keras as its default higher level abstraction. It's a pleasure to use.

ipunchghosts• 10 years ago

X is in the range 0, 255. They don't show code converting it to a much saner range for the network they've chosen. Is the full source somewhere?

cr0sh3• 10 years ago

Have you already looked at the pickled data? Because it looks like the model is outputting a single label value out of 256 labels; depending on the training data (steering angle) and how it is represented in the data (signed float or integer?), each one of those 256 learned should (?) be similar - I think.

Again, I'm not an expert. Or - maybe it is outputting a number 0-255, and then taking that number and converting it (and maybe other operations) into values suitable for the servo on the car (perhaps centered around 0 - so -128 to 127 or something like that - then scaled for servo PPM width or whatever values needed)...

All guesses, of course.

wroscoe10• 9 years ago

The input values are image arrays 120x160 pixels with 3 channels for red,green,blue. The values range from 0-255 and are not normalized before they are fed into the convolution layer. I found this did not make a difference.

The output of the model is a single real number between -90(left) and 90(right). I believe a better approach would be to bin the outputs and use a classifier. This way you'd know when the model was getting confused (ie, approaching a perpendicular line.

cr0sh3• 10 years ago

Also the full repo is mentioned (I think the article is just a general highlighting of the full repo):

https://github.com/wroscoe/donkey

zxcvvcxz• 10 years ago

Great summary, I always think it's best when machine learning projects have visuals and videos to showcase what is actually being learned.

This simple project is a good example of supervised learning from what I can tell - the network will learn to steer "as good as" the human that provides the training data. For a different (and more complex) flavor of algorithm, check out reinforcement learning, where the "agent" (computer system) can actually learn to outperform humans. Stanford's autonomous helicopters always come to mind - http://heli.stanford.edu/

ramshanker• 10 years ago

Before even opening the link, I was thinking, which jurisdiction would it be legal to program "personal" autopilot.

Awesome tutorial.

nilkn• 10 years ago

Was the track changed at all during the training? I'm wondering if there's some subtle overfitting here where the car learned to drive along only this specific track. It mentions this but I'm not sure what concrete actions were taken to avoid overfitting:

The biggest problem I ran into was over fitting the model so that it would not work in evenlly slightly different scenarios.

Regardless, a very cool project.

wroscoe10• 10 years ago

The method to avoid overfitting was to use the model with the lowest validation loss, not training loss. I was able to change the track around my house with reasonable success. I think would need many more example turns in the training data to become robust.

grandalf2• 10 years ago

Consider the fairly massive changes to the competitive landscape ushered in by the combined factors of self-driving and electric vehicles:

- For liability reasons, most of the algorithmic IP will likely be open sourced. Either because it's required by regulators or because it's the most efficient way for car makers to socialize risk of an algorithmic failure.

- Electric vehicles have many fewer moving parts, which means that the remaining parts are likely to be converged upon by the industry and used widely. This breaks a lot of platform-dependency issues and allows for the commoditization of parts like motors. As these become standardized and commoditized, and easily comparable on the basis of size, torque, and efficiency, there will be virtually no benefit to carmakers to manufacture their own. The same applies to aluminum monocoque frames, charging circuitry, etc.

Tesla currently differentiates its models based on how many motors and what size batteries, but beyond that it's mostly just cabin shape, along with new innovations like the hepa filter cabin air cleansing which will likely be a standard part of all future models.

- Battery tech works the same way as motors, with little competitive advantage to be gained by automakers, especially since most of the IP in this area is already spoken for.

Compare the number of patentable parts in a model T vs a 1998 Taurus vs a 2017 internal combustion vehicle vs a Telsa. Tesla is one innovator, and GM has already likely patented many inventions relating to EV technology back in the original Chevy Volt era.

All this is why Tesla acquired SolarCity and is attempting to make an infrastructure play rather than a technology play. Only due to Musk's rare ability to self-finance big risks is this even possible, since infrastructure moonshots featuring $30K+ hardware units are hard to fund.

mulmen• 10 years ago

How do you see car makers differentiating their products in a world where all the parts including the frame are commoditized and the software is open source?

Also, GM built an electric car back in the 90s called the EV-1. I wonder how much innovation was in that car vs the Volt.

Drdrdrq2• 10 years ago

Interior design. Cars will evolve into mobile living spaces so the quality of interior becomes more significant to exterior.

JoshTriplett• 10 years ago

Quality, maintenance, cost of ownership, style, comfort, physical construction, actual innovation, accessories, existing brand preference; all the things they differentiate themselves with today.

rsp1984• 10 years ago

I like your analysis here but how is this related to the article?

grandalf2• 9 years ago

This DIY lane following algorithm made me think about it, and after reading Elon Musk's tweet about retrofitting I'd been thinking about the market and figured this thread was as good a place as any to deposit my thoughts.

nojvek• 10 years ago

What I would love to see is an end to end neural network soln. On one end camera input comes through, on the other outputs for speed and steering angle.

But rather than a black box, it's explainable what the different layers are doing. If neural nets are turing machines then we should be able to compile some parts of the net from code.

Then the net is a library of layers. Some Layers trained with back prop, some compiled from code.

wroscoe10• 10 years ago

I'm working on adding the throttle. This is difficult because you need to drive the correct speed and stopping or running off course can mess up the training data. This project was inspired by Otavio's carputer which does predict throttle, steering angle, and odometer.

argonaut4• 10 years ago

Almost all neural nets are not Turing complete. Only very specific RNNs are; most RNNs aren't, including pretty much any RNN model used in the real world right now (https://uclmr.github.io/nampi/talk_slides/grefenstette-nampi...).

Also, this is a useless fact, because so many other random things are Turing complete.

deepnotderp• 10 years ago

The end to end approach of regressing steering wheel angle already exists, check nvidia's paper.

sja2• 10 years ago

I might be missing it, but I don't see instructions for installing TensorFlow/Keras on the Raspberry Pi in the Donkey repo or in this blog post (needed to actually run the trained model, it looks like). For TensorFlow, there are pre-built binaries and instructions to build from source here:

https://github.com/samjabrahams/tensorflow-on-raspberry-pi

Note: I am the owner of this repo

wroscoe10• 10 years ago

Donkey runs a client on the Pi and a remote server that runs Keras/Tensorflow.

sja2• 10 years ago

A ha! Very cool- apologies for not seeing how it worked at first; I assumed you used the server to control/collect data manually, and then loaded the model onto the device. Thanks for the demo!

billyzs3• 10 years ago

Not to put down the OP's work (I think it's a great project), but I'm just wondering what advantages might an ML approach have over "traditional" CV algorithms. In a really well controlled environment lanes will be easy to detect, and computing the difference between the current heading and lane direction should be doable; maybe if we're talking about complex outdoor environments and poor sensors then ML would have an advantage? Or if we're teaching the robot what the concept of a lane is?

I think back to the days when I basically implemented lane following with an array of photo resistors, an Arduino, a shitty robot made from Vex parts and some c code. The problem is much simpler than the one presented in this article, but then the computational resource used was order of magnitudes less. At what point then, do you decide that "OK I think the complexity and nature of the problem warrants the use of ML" or "Hmmm I think neural network is an overkill here"?

wroscoe10• 10 years ago

Traditional CV approaches are much easier to debug as well. I chose the ML approach with the assumption that it would be easier to build a robust autopilot that would work in many lighting conditions. Actually my short term goal is to get the car to drive around my block on the sidewalk (no lines). From my experience CV approaches have many parameters that need to be tuned specifically for each environment. While ML approaches also have parameters that need tuning they stay constant between environments.

billyzs3• 9 years ago

I see, that makes sense. It'd be indeed worth it if we can apply a model trained on controlled environment to a more challenging one with little to no modification. Good luck with the project and keep us updated!

platz• 9 years ago

My first thought was something that used several PID mechanisms.

option_greek• 9 years ago

Because ML approaches can adapt to different environments like a forest trail. While this can probably be achieved with OpenCV, this just feels natural: https://www.youtube.com/watch?v=umRdt3zGgpU

argonaut4• 10 years ago

Two major errors: 1) This doesn't seem to be controlling overfitting on the right validation set. 2) There isn't a test set at all (separate from validation).

Using Keras' "validation_split" parameter will just randomly select a validation set. This is not the right thing to do when your data is image sequences, because you will get essentially identical data in training and validation.

Because of this, the numbers/plot here might as well be training accuracy numbers.

mhanus• 10 years ago

Keras uses the end of the data set as validation, and only randomizes it if the "shuffle" argument is set to True [1].

[1] https://keras.io/getting-started/faq/#how-is-the-validation-...

argonaut4• 9 years ago

Except the second half of the data is the flipped of the first half (X = np.concatenate([X, X_flipped]))

feelix• 9 years ago

Apologies if I'm being stupid, but I can't find the details on how to physically connect the hardware together anywhere. Is this still on the todo list? I'm interested in applying this tutorial and making an autonomous RC car.

wroscoe10• 9 years ago

I updated this post with some of the great feedback from the comments. Also I just ported the algo used by the last DIYRobocar race winner, CompoundEye. Here's that post: https://wroscoe.github.io/compound-eye-autopilot.html#compou...

Thanks!

Lane Following Autopilot with Keras and Tensorflow

Related Stories