Piloted Highway Driving

10 min readSep 21, 2021

This was a fun project that I started in my summer vacations. It is a collective outcome of the knowledge gained from Udacity’s self driving car engineer nano-degree program and the literature survey done post the course on trajectory generation and behavior planning. I love solving math problems especially, linear algebra and calculus. hence, I have tried my best to stitch some concepts together that I found interesting to solve the problem at hand.

Introduction

Driving on the Highway is a mundane yet challenging task to fully automate as it involves higher risks due to the speeds of the vehicles involved.

The goal is to safely navigate around a virtual highway with other traffic that is driving +/-10 MPH of the 50 MPH speed limit. A Unity3D simulator environment provided by Udacity was used for this purpose.

Given the car’s localization and sensor fusion data, there is also a sparse map list of way points around the highway. The car will try to go as close as possible to the 50 MPH speed limit, which means passing slower traffic when possible. The simulation environment is quite dynamic as other cars will decelerate, accelerate and try to change lanes too. The car should avoid hitting other cars at all cost as well as driving inside of the marked road lanes at all times, unless while changing lanes. The car should be able to make one complete loop around the 7 km highway. Since the car is trying to reach 50 MPH, it should take a little over 5 minutes to complete 1 loop. Also, the design constraints will include comfortable driving goals i.e. The car should not experience total acceleration over 10 m/s² and jerk that is greater than 10 m/s³.

Details of the simulator

The car uses a perfect lateral and logitudinal controller and will visit every (x,y) point it receives in the list every .02 seconds.
The units for the (x,y) points are in meters and the spacing of the points determines the speed of the car. The vector going from a point to the next point in the list dictates the angle of the car. Acceleration both in the tangential and normal directions is measured along with the jerk, the rate of change of total Acceleration.
The (x,y) point paths that the planner receives should not have a total acceleration that goes over 10 m/s², also the jerk should not go over 50 m/s³. NOTE: As this is BETA, these requirements might change. Also currently jerk is over a .02 second interval, it would probably be better to average total acceleration over 1 second and measure jerk from that.
There will be some latency between the simulator running and the path planner returning a path, with optimized code usually its not very long maybe just 1–3 time steps.
During this delay the simulator will continue using points that it was last given, because of this its a good idea to store the last points you have used so you can have a smooth transition. previous_path_x, and previous_path_y can be helpful for this transition since they show the last points given to the simulator controller with the processed points already removed. the application would either return a path that extends this previous path or make sure to create a new path that has a smooth transition with this last path.

Implementation :

I have made my implementation available in this repository while most of the implementation details are already present as code documentation, this article would just summarize the points in the readme.

Architecture:

The path planner task is split into the following functional modules as shown in the class diagram:

Map Route Generation — Map class
Scene Object Prediction — Objects class
Path Planning — PathPlanner class
Trajectory generation — Trajectory class
Cost Computation — Cost class

where the PathPlanner class is the main entry point of the planner that generates the target trajectory that the vehicle would follow.

While there can be many architectures that satisfy the requirements, the following architecture was developed that solved the requirements at hand while keeping the functional implementation in relevant modules.

Map Class

The Map class object is the first to be constructed and the last to be destructed.

This module basically improves the interpolation of the way points in the map. The cubic spline improves the accuracy of interpolation over doing a simple linear interpolation between two closest map way points.
Since, it is computed once at the start for the entire route, it does not cause any performance overhead in terms of time complexity.
The resolution is later helpful for the Jerk minimizing trajectory generation in the trajectory planner.

Objects Class

The Objects class does a basic behavior prediction of the scene around the Ego vehicle.

The class is responsible to project using the current data of the objects basically the frenet s and d , cartesian x, y , dx, dy and compute the speed, future x,y and yaw angle i.e. heading.

behavior prediction — estimation of trajectories for scene objects

After getting the speed and filtering the frenet d and frenet s the objects that are outside the kPredictionHorizon which is around +/-70 m in the frenet s coordinate and outside the d coordinate of 0m to 12m are ignored.

using the newly filtered relevant objects the closest front and rear objects in all the lanes are identified.
TTC (time to collision) and TTD (time to decelerate) are computed. This is essential to find free space and max possible safe speed in a lane segment so that both rear ending into a front vehicle and another rear vehicle rear ending into the Ego is avoided.
the free space of all the three lanes are stored and the max speed allowed in the three lane segments are stored.
the PathPlanner will then use this information to generate myriad of trajectories.

Path Planner Class

The PathPlanner is responsible to generate targets or goals without thinking much about it’s feasibility other than being legally possible. The path planner will only govern the bare minimum legal aspects of safe driving behavior such as

Traffic lane laws : which lanes to drive in for a long duration
Etiquette in switching lanes : Do not make a direct switch into a lane that would require crossing another lane a.k.a avoid double lateral lane change.
respect safety distances : this is to ensure that the ego vehicle does not end up going head on into a confusing traffic situation ending in the solver not converging requiring an emergency braking maneuver. To make it easier for my implementation, The German traffic law system is used. This was chosen to avoid much research into the legal aspects as both countries USA and Germany have at least the same traffic side.

German Traffic rules are explained as:

“Rechtsfahrgebot” — it means to prefer the right lanes to cruise/drive as a routine lane. this is the right-most lane marked as ROUTINE lane.
“Rechtsüberholverbot” — do not overtake on the right if possible. However, this does not include passing a slow moving vehicle that is traveling on the ACCELERATION lane.
ACCELERATION lane is the center lane, where vehicles shall accelerate quickly and reach the top speed. The EGO may prefer a OVERTAKING or ROUTINE lane depending on their free spaces.
OVERTAKING lane, which is the leftmost lane will be preferred when it is free for the kPredictionHorizon.

The following rules make no lane changes and follows the ego current lane.

cruise control target. The driver set speed is the target velocity.
spacing control target. The driver set speed or permissible speed set by the front and rear vehicles is the target velocity.
The Cruise control target is basically the vehicle in a open free space accelerating to reach 50MPH. in this case it is 49MPH so that the max speed is not violated.
The Spacing control target is when there is a vehicle restricting the free space of the EGO. the target velocity is set to the vehicle in the front.
Finally, one more target named the AEB — Automatic Emergency Braking target which will decelerate the vehicle at max deceleration to desperately avoid a collision is added.
A maximum of 4 targets are generated and a minimum of 3 targets are generated.

Trajectory Class

Goal is to generate trajectories. nothing is validated while generation. trajectories in both Frenet and Cartesian coordinate systems are generated using the Map object and the Objects prediction data.

Trajectory is generated using two methods.

JMT — quintic polynomial solver — jerk minimizing trajectory
frenet trajectory generation based on s and d
JMT trajectory is used as the primary generator. it is responsible to generate a minimum jerk trajectory for the given target. It is based on solving a quintic polynomial equation as shown.

quintic polynomial based trajectory generation

Where s and d are the transformed x,y from Cartesian coordinate system into Frenet system. the end result is the generated trajectory by solving the matrix expression. solving is done by inverting the T matrix as the code snippet from the project illustrates.

double T_3 = T_2 * T;

double T_4 = T_3 * T;

// make a computation of T matrix.

MatrixXd T_(3, 3);
T_ << T_3, T_4, T_4* T,
3 * T_2, 4 * T_3, 5 * T_4,
6 * T, 12 * T_2, 20 * T_3;
double T_2 = T * T;

// check if the matrix is invertibleif (T_.determinant() != 0)
{

  MatrixXd S(3, 1);
  S << (i_end[0] - (i_start[0] + i_start[1] * T + 0.5 * i_start[2] * T_2)),
  (i_end[1] - (i_start[1] + i_start[2] * T)),
  (i_end[2] - i_start[2]);
  
  VectorXd A(3);
  
  A = T_.inverse() * S;
  returnValue.push_back(A[0]);
  returnValue.push_back(A[1]);
  returnValue.push_back(A[2]); 
}

The frenet trajectory generation function is a special case to handle emergency braking maneuver because, the JMT solver that was developed will fail to provide a solution for T = 0s.
Reason : The T matrix will become singular. hence, it is not invertible to give a solution for a minimum jerk trajectory.
The final step is to compute costs for all the trajectories that were generated.
The minimum cost is selected. if all the trajectories involve collision with predicted object trajectories, then the AEB trajectory is chosen.

Cost Class

The corner stone of the trajectory generator is the Cost class. the Cost class does the following

Compute max acceleration and total acceleration of the trajectory in question.

Governs the max acceleration not exceeding +/-10 ms^-2 and total acceleration over all the steps in the entire trajectory of +/-10ms^-2

Compute max jerk and total jerk of the trajectory in question.

Governs the max jerk and total jerk to not exceed +/-10 ms^-3

Compute collision cost.

A large positive cost of 10⁵ is atleast given to collision that is compounded by trajectory point where the collision occurs.
A collision detection would straight off result in the elimination of the trajectory from selection.
Never will a collision trajectory make it’s way into the output as a selected trajectory.

Compute efficiency cost.

This is basically the difference between free space and the kPredictionHorizon. The difference is between 0m and 70m. less the better. this ensures the Ego chooses the lane with more space.
Total cost is computed after weighing the individual costs. the mandatory checks are given high cost. the less mandatory is given a lower cost.

Other helper classes

function a function(t) class that computes N derivatives of the given polynomial coefficients.
Rectangle a class which implements SAT (separating axis theorem) to find overlapping objects to compute collision on predictions of object positions and generated trajectories to compute collision cost.
Separating Axis Theorem: If a line can be drawn to separate two polygons, then they do not collide. This is used to find if the trajectory overlaps any scene object.

separating axis theorem for overlapping objects

More theory can be found in the link. The theory was used to implement collision detection on predicted trajectories in the project.

Integrating it all together

After a trajectory is selected. the path planner returns this trajectory so that the simulator vehicle can act. In order to account for the simulator latency, 5 previous points from the trajectory are re-used.

Summary:

Thus, using cubic splines to improve map data interpolation and solving for quintic polynomial equation coefficients for finding optimal trajectories, trajectories for all possible candidate targets were generated based on the situation of the Ego vehicle. the trajectory with the lowest cost was chosen. The vehicle was observed to perform lane changes and maintain a steady highway driving close to the top speed while not exceeding the comfort and safety conditions.

Scope for future work:

The algorithm was designed to lose efficiency and trade it off against jerks, uncomfortable acceleration and collisions on the predicted path thus, being defensive in it’s driving approach. The next possible iteration is to use Bayes theorem to predict lane changes for objects on the road using my implementation of a gaussian naive bayes classifier from scratch to classify lane changes. This way the Ego vehicle can be made to run closer to the upper limit of its comfort and safety capabilities.