Tesla's Robotaxi error

And a concern that autonomous vehicles must mimic what goes on in drivers' minds--even though we have no training data

Here is a clip from a Tesla Robotaxi making and then correcting a mistake. (Click the image for a 30-second video.) The Tesla is following the car in front of it into the left-turn lane. The mistake was that the Tesla should have waited until the following block to make its left turn.

As the video shows, when the car is about to begin the left turn, it “realized” its mistake and jerked the steering wheel back and forth, uncertain about how to fix its error. Worse, the left turn lane it was in ended at the intersection, but the Tesla was unable to get back into the forward lane. The car found itself riding a block on the wrong side of the double yellow line. Eventually, it reached the next left turn opportunity and turned successfully. Watch the clip to see the event.

Assume you were Tesla, the company. How would you train the autonomous driving neural net to fix this problem? Assume you wanted to train the system to deal with its error more gracefully. (If you could eliminate the error itself, that would be best. But let’s look at the problem of fixing this error after it occurs.)

How a human would act

If I had made this error, i.e., entered a left-turn lane and begun the turn even though I was a block early, the following would be going through my mind: let’s complete the left turn—even though it’s wrong—and expect the navigation software (i.e., GPS) to determine what to do next. That happens all the time. I frequently miss a turn or decide not to follow the GPS instructions precisely. Whenever that happens, the GPS system says “rerouting” and determines the best route to complete my ride, given my then-current location.

The problem Tesla and other companies training autonomous driving systems face is that the thought process described above, including my expectation that the GPS system would adjust my route given the error, is not part of the training data. There is nothing in the record of what my car actually does that captures my expectations. It is not possible to train an autonomous driving neural net to have “expectations” based solely on a car’s movements.

Can we train a neural net to mimic human thought—and do it without having human thought data available for training?

Now that I write this, it strikes me that a driver’s thought process is central to the driver’s steering, braking, etc. actions more generally. The top row of the following diagram represents what normally happens when a human drives a car.

The bottom row represents how we want a neural net to drive a car. Such a neural net is trained by giving it many sensor-data/driving-action pairs and expecting the training result to produce a neural net that will produce appropriate car actions for all external world states. Given the amount of mind-reading expected to be trained into such neural nets, it’s striking that they succeed as well as they do.