The AI world is fascinating, and today has been used in a lot of areas to make our work easier. One aspect of AI is deep learning and has been used extensively is the Advanced Driving Assistance System. The ADAS has been included in almost 40% of the vehicles today. In this course, the students will 

• Gain insights into the AI world in context of Deep learning in ADAS
• Learn softwares such as tensorflow, numpy, opencv. 
• Learn Image Processing.

Autonomous vehicle uses maps to plan the path ahead. In addition to this, using SLAM (Simultaneous Localization and MApping),  the system detects unknown paths and adds them to the map which can be later used for path planning and obstacle avoidance. Under this course, the students will learn how to 

• Produce estimates from unknown variables when on road 
• Develop 3D models with Camera and Lidar data fusion 
• Specifically identify locations on map using Grid Mapping 

While robotics piques the interest of a majority of people, to develop one is not a piece of cake. Every movement that a robot makes is programmed and involves path planning and trajectory optimisation. This course on the same topic will help the student gain insights into robot machine planning, which is used in autonomous vehicles, warehouse robots etc. Here the students will learn 

• Step by Step description of motion planning techniques 
• Practical implementation of learned concepts using ROS, C++ and Python 
• How to develop softwares for robots

Every company uses some specific learning tools to build their own ADAS technologies. This course focuses on building a control system for driver assist technology. The students here will learn how to 

• Develop an ADAS system from Level 1 automation to Level 3
• Build control system using simulink 
• Build a level 2 adaptive cruise control project

This week ,we will be giving emphasis towards learning more about Tensor. The concept of tensor is very important in the autonomous vehicle system, and this week you will be knowing why. You will also look into tensorflow and numpy along with why and where we use them.

• What is a tensor?
• What is tensorflow and numpy?
• Tensorflow ML execution Environment.
• Why tensorflow ?
• Tensorflow basics.
• Numpy basics.

Here, we will look into the Basics of building Artificial intelligence. We’ll first try to decode an AI and learn about it. We will also look into neural networks and basic code in Tensorflow. The topics in focus for the week will be

• Decoding AI  (ML,CV,Deep learning,Reinforcement Learning)
• NNs,forward pass,backward pass, Optimizers,losses,regularizations.
• Practical basic code in Tensorflow (Basic NNs)
• CNNs,RNNs
• Practical basic code in Tensorflow (Basic CNNs and RNNs)
• Areas of AI which will be covered in the course
• Combining everything, a simple program.

Our emphasis this week will be to learn how to handle data. During the entire process of learning, our AI will be gathering, storing and processing a lot of data. It is vital that any person working with ML and AI know how to store, archive, and retrieve the data. This week, we will be looking into

• Web Scraping. 
• Types of Data we will need
• Data Annotation
• Data Validation
• Basics of DVC
• How to version and structure Data
• Datasets common - KITTI, Cityscape
• Metrics - various IOU, NMS Threshold, RMax, Efficiency, Throughput others. Ways to create metrics

During the training process, known data is provided to the neural network. The NN (neural network) makes a prediction about what the data we are showing represents. During the learning procedure as each of the images are passed to the NN, it “infers” the images and learns. This will be our focus for the week.

• Overview
• Softmax
• Cross Entropy
• Practical Code
• SVM 

Random forest is a type of “Ensemble learning” method which is used to teach a machine. This is one of the many NN learning procedures.

• Decision Tree 
• Cut-in example

Regression is a supervised machine learning technique that is used to predict continuous values during the learning procedure. This week we will learn about

• Regressing x,y location of object of interest in the scene
• Pre-trained network or Transfer learning for transfer learning or Pre-trained CODE for training dataset.

Object detection is a very vital part of automated vehicles. This helps the vehicle to identify and asses- road conditions, traffic, pedestrians, traffic signs, traffic lights and much more. This is vital for safe and smooth functioning of the vehicle. So, in this week we will be learning about the

• Basics
• Overview of Basic Object Detection model.
• Building a basic object detection model.

During this week we will be teaching our vehicle on how to identify which path it has to follow to reach its destination. This includes teaching it how to check lanes, how to shift lanes, and also how to take the shortest route from point A to reach point B. So, the specific topics we will be looking into are

• What is path policy?
• Finding lanes with deep learning and traditional methods.

This is the final week of our course. So, we have learnt all the knowledge that is required to teach and run our vehicle AI. We will be wrapping up the course by having a

• Quick Overview
• Refreshing everything
• End (Camera input) to End(Throttle, Brake and Steering input) ML
• Self Driving Car - Steering angle detection

In order to extract information from the image, image processing techniques are carried out on it. These extracted features in terms of shapes and alphabets, helps in recognizing an object. This week, we will look into the techniques that are used to do so. 

The topics, that will be discussed are:

• Image Filters 
• Gaussian Noise
• Noise removal methods 
• Convolution and Correlation operations
• Mean and Median filters 
• Template matching methods
• Edge Detection methods : Canny , Sobel , Prewitt etc. 
• Image Gradients 
• Hough transforms – straight lines , circles and curves 
• Time/Spatial to frequency domain Image conversion

The images are viewed on an x-y plane. Each of it’s pixels is traced in a manner to get information out of it. This is facilitated using image geometrics and cameras. In this week, we will learn about the different geometries that are used in computer vision:

• Image coordinate system: Different systems 
• Projective geometry 
• Perspective Projection 
• Multiview Geometry 
• Stereography and Depth Imaging 
• Stereo Correspondence 
• Epipolar Geometry 
• Rigid body transformations 
• Geometric camera calibration 
• Multiplane calibration 
• Finding corners and matching feature points 
• Performing transformations with intrinsic and extrinsic parameters 

A  video is a sequence of images. In order to extract information from it, these images will be evaluated every time with a different angle. The topics that will be discussed under motion models are:

• Applications of motion modelling 
• Motion estimation and techniques 
• Optical flow 
• Lucas and Kanade methods: Hierarchical and Sparse methods
• Full motion, general motion and affine motion models 

Filters are applied to an image to enhance it. However, applying a filter to a video would mean to continuously keep a track on all the images and apply filters to it. The process that is carried out to do this, will be discussed this week. 

• Introduction to Tracking 
• Feature trackers
• Steps in tracking, Prediction and Correction 
• Constant velocity, acceleration models 
• Kalman filter: Prediction, correction, Intuition and tracking with Kalman filters 
• Bayes Filters 
• Particle Filters and Localization using Particle Filters 
• Real life tracking issues and comparison of different methods 

There are various types of things around us, from cats, dogs, humans, tree so and so forth. One of the applications of computer vision is to differentiate one object from another.    This understanding of what falls under what category is done by image image recognition and then are further classified by image classification.  The topics, discussed under this week are: 

• Introduction to Recognition and classification 
• Supervised and Unsupervised learning methods 
• Dimensionality reduction , Principal Component Analysis , Discriminative classifiers , 
• Machine learning classifiers 
• Feature extractors: HOG , Haar Cascade and CNN models 
• Introduction to Convolutional Neural Networks 
• Building blocks of Neural Network for computer vision 

It is very easy for  a human to differentiate one object from another just by looking at it. However, this task is done by computer vision following a number of steps. This becomes, even more difficult when it comes to videos, to recognize and detect objects from noise. An understanding of how this is done will be given this week.

• Understanding background and background subtraction methods 
• Filtering methods for videos 
• Temporal templates 
• Moments in images and video 
• Hidden Markov model
• Image segmentation 
• Clustering methods for segmentation 
• Mean shift algorithms for segmentation 
• Segmentation by Graph partitioning 

In contrast to 2D vision, 3D introduces another dimension called the depth dimension. This makes the images that we are looking at more realistic. 3D vision is enabled in a number of ways. We will learn about these methods this week. 

• Introduction to depth imaging 
• Depth estimation from stereo imaging
• Role of geometry in depth imaging 
• IR based depth estimation 
• Applications of depth imaging 

To make the process of learning computer vision easier a number of libraries exist in the form of frameworks. What these are and how they can be used is what we will be covering this week. 

• Introduction to Deep learning architectures
• Alex net 
• Yolo 
• VGG Net 
• GoogleNet 
• ResNet 
• Region based CNN
• Introduction to deep learning frameworks / Libraries for computer vision 
• Tensorflow
• Keras 
• Pytorch 
• Caffe
• Constructing a model and training with the frameworks 
• Inference graph / model generation

Computer vision does not gain its learning by looking at an image from one side. Instead, multiple copies of it are generated with the help of algorithms in order to understand an image. This is one of the things that gives Computer Vision its intelligence. 

• Introduction to Data collection 
• Data acquisition : Discovery , Augmentation and Generation 
• Data Labelling : Labeling methods and tools used for labeling 
• Using Existing Data : Relabeling and Transfer learning methods
• Introduction to Synthetic data generation 
• Rendering methods 
• Procedural randomization 
• Manual Modeling 
• 2D and 3D datasets generation 

In the last two weeks, 5 research papers will be provided to the student. The student can pick the topic of their choice from them. This will be followed up with lectures on that topic. 

• 3D Object detection from 2D Images – Monocular 3D , Object Detection Leveraging Accurate Proposals and Shape Reconstruction 
• Human Pose Estimation – Deep Cut : Joint Subset Partition and Labeling for Multi Person Pose Estimation 
• Panoptic Segmentation 
• COCO-GAN : Generation by Parts via Conditional Coordinating 
• Hierarchical Multi-Scale Attention for Semantic Segmentation 

Stability plays a crucial role in determining the safety and performance of vehicles.  In the case of autonomous vehicles, it deserves even more attention. To ensure stability and to perform all the required functions in an efficient manner, autonomous vehicles employ control systems.

The second week of the course will give you an overview of classical controls. The topics that will be covered here include: 

• Stability
• Pole zero methods
• Transient performance
• Disturbance and tracking
• Other classical controls definition
• PID systems
• Analysis and solving
• Examples on P, I, PI & PID
• Gain Selection 
• A brief explanation on solvers in Simulink

Adaptive Cruise Control ensures safety by maintaining the vehicle at a safe distance from vehicles ahead. It functions by automatic alteration of the speed of the vehicle based on the circumstances. 

The third week of the course includes a  project that involves Adaptive Cruise Control. The topics that will be covered in this week include: 

• Overview of adaptive and normal cruise control
• Start of project with longitudinal model
• Longitudinal dynamic modelling
• Aerodynamic drag
• Rolling resistance
• Linearizing longitudinal dynamics
• Longitudinal dynamics block diagram in simulink 

A longitudinal controller regulates the cruise speed of the vehicle. It is a system of sensors, control computation and control actuation components. The fourth week of the course deals with the design of longitudinal controllers for autonomous vehicles. 

The topics of this week are: 

• Transfer function modeling in Simulink
• Modeling control system
• Controller design 

Other than safety, adaptive cruise control offers convenience to drivers. They keep the vehicle steady by adjusting the speed and also, they accord the option for the drivers to set their own preferences. 

Fifth week of the course covers topics like: 

• Touch base on least square method 
• Adaptive control for vehicle speed control

This part of the course deals with the ADAS modeling of Adaptive Cruise Control. Here, you will get to know about the sensors used, mathematical model, basics of Linear Quadratic Regulator, state model, etc. 

The topics of the week include 

• Introduction to level 2 automation & sensors used
• Mathematical model
• Introduction to LQR basics
• State model 

This week also covers the modeling of Adaptive Cruise Control. You will get to know about the design method and modeling of  ACC. Topics of the seventh week include: 

• Design method and modeling 
• Simulink modeling for ACC

Cooperative Adaptive Cruise Control is an extension of Adaptive Cruise Control that makes the autonomous vehicles connected. Other than regulating vehicle speed for maintaining a safer distance, CACC makes autonomous vehicles cooperate with one another by establishing communication between them. 

The Eighth week of the course deals with improvements in adaptive cruise control. This covers the topics of 

• Cooperative adaptive cruise control 
• Introduction and modeling

Proper navigation of an autonomous vehicle is achieved by means of longitudinal and lateral controls. As longitudinal control regulates the cruise speed of the vehicle, the function of lateral control is to steer the wheels to keep them in the lane. In other words, it deals with the lane keeping and lane changing control. 

The topics that will be discussed in this week are 

• Introduction to lateral control 
• Lateral model and tire model
• Bicycle model
• Lance centering logic discussion
• Model lane centering logic
• Assignment on  controller design and tuning 

Lane centering is the feature designed for maintaining the vehicle position at the centre of the lane. It automatically steers the vehicle to ensure that it travels only along the centre of the lane. This week also deals with the modeling of lateral control. 

Topics that will be covered in this week include 

• Lane control modeling 
• Lane centering feature 

The last week of the course also deals with lane centering. It covers the modifications of the lane centering features. The topics of this week include

• Lane biasing
• Introduction to logic
• Feature modeling 

In the final week of this course, the student will be working on the major project. They will have to develop a level 2 system with Adaptive cruise control and Lane Change Assist

Very often when a car passes through a tunnel or location where there is a lot of disturbance, determining their exact location becomes difficult. These disturbances are called noise, and one needs to overcome this noise to get the continuous location of the car. To do this, Kalman filters are used. 

Under Kalman filters in this week, we will be covering 

• MLE and MAP 
• Bayes Inference 
• Gaussian Distribution 
• Kalman filter

Kalman filters are further divided into Extended Kalman Filters and Unscented Kalman Filters. While EKF uses a few points to estimate the location, UKF uses a number of points to estimate a given location. How is one better than the other and why is it needed will be explained in detail in week 3. 

The topics that will be covered in this week are:

• Introduction to Nonlinear Kalman Filters 
• EKF 
• Solved example of EKF 
• UKF

Using the range sensor, odometer, and a map of the location, Monte Carlo Localization helps in estimating the position and orientation of the object of interest. 

In this week, the topics that will be covered are:

• Explaining MCL with example 
• Properties of MCL 
• Discussion problem solving using MCL

An aircraft, before landing needs to be sure of its position, a self-driving car also needs to be alert of its path. While driving, the car should be on the road and also alert of its environment. This is done by using sensors such as GNSS/INS. 

In this week, we will learn about:

• Sensor Fusion and the need for it. 
• Introduction for Pose Estimation using GNSS and  INS 
• Why and where do we need sensor fusion 4.
• Demonstration

Camera and Lidar (or Light Detection and Ranging) use two different approaches in order to detect an object. By fusing these two parameters, the distance at which an object is from the self-driving car can be estimated with accuracy. 

In this week, we will cover:

• Camera and LiDAR parameters 
• Applications 
• Demonstration

During the course of this week, you will be learning about SLAM/Mapping.  Simultaneous Localization And Mapping(SLAM) is a computational problem that constructs or updates a map of an unknown environment and also keeps a track of the robot’s location.

In this week we will cover, 

• Introduction to Mapping and SLAM 
• Real – Life examples 
• Features like ORB

During the course of this week, you will be learning about Occupancy Grid Mapping. Occupancy Grid Mapping refers to a family of computer algorithms that address the problem of generating maps from noisy and uncertain sensor measurement data for mobile robots.

In this week we will cover, 

• Algorithm 
• Demonstration 
• Examples

During the course of this week, you will be learning about EKF SLAM. EKF SLAM algorithms are used for maximum feasible algorithm for data association. It is basically a class of algorithm that utilizes the Extended Kalman Filter (EKF) for Simultaneous Localization And Mapping (SLAM). 

In this week we will cover,

• EKF SLAM 
• Comparison with EKF localization 
• Application 
• Advantages and Disadvantages

One way of detecting the position of the self-driving car or robot is to take an estimation of its movement and the distance it has from one landmark. This gives the idea of where the object is most likely to be next, making it easier to place its location on the map. 

In the last week of this course, the students will learn about :

• Graph SLAM 
• Application 
• Advantages and Disadvantages 

During the course of this week, you will be learning about C-Space i.e Configuration Space. C-space is the space that provides possible positions for the robot to move. 

The topics you will be learning this week are:

• How to use the Configuration Space
• Representing Configuration space as a graph
• Planning using Visibility Graph
• Finding the shortest path.
• Djikstra’s Algorithm, A*, Bellman-Ford

During the course of this week, you will be learning about sampling-based motion planning. This will solve the navigation queries. Instead of depending on the entire map of the C-space, the robot depends on the procedures that decide if the robot’s configuration is approaching an obstacle or not. 

The topics you will be learning this week are:

• Various types of Rapidly exploring Random Tree(RRT)
• Application of RRTs
• Path planning using the RRT algorithm
• Setting up Ubuntu environment for the next week

During the course of this week, you will be learning about ROS. ROS is a robotics middleware that manages the complexity and heterogeneity of the hardware and applications. Not only this but it also performs low-level device control, implementation of commonly-used functionality, message-passing between processes, and package management.

The topics you will be learning this week are:

• Setting up ROS 
• Following instructions on the ROS website
• Adding ROS to the docker container
• A basic introduction to Cmake
• Programming using ROS
• Introduction to 3-D visualization tool called Rviz
• Difference between 
• ROS/RTOS
• ROS1/ROS2
• DDS
• Middleware

During the course of this week, you will be learning about motion planning with non-holonomic robots. Non-Holonomic robots are built in such a way that they only travel in one direction along a given axis. To put it in simple words, Non-Holonomic robots can only move forward, backward, or sideways.

During this week you will learn:

• Path and speed planning
• Trajectory representations
• Splines
• Clothoid
• Bezier curves
• Polynomials
• Introduction to Frenet Frame a
• Planning in Frenet frame
• Boundary value constraint problem and methods
• Pointwise constraint problem and methods

During the course of this week, you will be learning about Mobile Robot collision detection. Here, the robot will detect a collision and will change its trajectory to escape the contact as fast as possible and move away safely.

During this week you will learn:

• Collision detection for static Obstacles
• Motion prediction for dynamic obstacles
• Motion prediction in Frenet frame with Kalman filters
• Collision prediction for dynamic obstacles

During the course of this week, you will be learning about Hierarchical planning for Autonomous Robot. Hierarchical planning optimizes the global path and it requires only a considerable amount of time for the path replanning operations.

During this week you will learn:

• Route planning, A*, D*, D* lite
• HD Maps, SD Maps
• Behavior planning - State Machines, Decision Tree, Behavior Tree, etc.
• Behavior and Motion Planning integration

During the course of this week, you will be learning about Trajectory planning. Trajectory planning plays a major role in robotics and paves way for autonomous vehicles. It is basically the movement of robots from point A to point B by avoiding obstacles over time. 

During this week you will learn:

• Polynomial Planners
• Lattice Planners
• Collision checking
• Trajectory selection (Cost functions)

During the course of this week, you will be learning theoretical concepts on this topic

The topics you will be learning this week are:

• Dynamic Programming for Path planning for the long horizon.
• MPC Planner for the short horizon.

During the course of this week, you will be learning about Planning in unstructured environments. Unstructured environments include the off-road, parking lot, etc. In such a type of environment, the robots should identify the optimal path between the start and the goal path. So, for the robots to perform this, a suitable path planning algorithm is required.

During this week you will learn:

• Path planning for a racing environment 
• The Autonomous Racing Software Stack of the KIT19d
• Covers essential modules like
• Perception
• Localization
• Mapping
• Motion planning and control

During the course of this week, you will be learning about Reinforcement learning for planning. Basically, it is a machine learning method that has increased application in robot path planning. The robot would explore its surrounding environment and learn using the trial and error process. The machine learning method has an advantage in path planning and requires less prior information. 

The topics you will be learning this week are:

• Markov process and Bellman’s principle of optimality
• Value and Policy iteration
• RL classifications: On/Off Policy, Model-based/free
• TD learning/ SARSA, Monte Carlo, Dynamic programming, Q learning
• DQN
• Highway driving example
• Parking lot example

Now that we have covered the major parts of the course, we will now be moving on to the concluding week.

During this week we will be 

• Recapturing on what we learned
• Tips on how to stand out in job search
• Tips on how to proceed in academics in this field