#Lucas Kanade Optical Flow method to perform sparse Optical Flow for pixels of interest.
#Pixels or features of interest are derived from Shi-Tomasi corner detector algorithm.
#Importing neccessary libraries
import cv2
import numpy as np
#defining featue detection function using Shi-Tomsi corner detection.
def find_features(gray_image):
#ShiTomasi corner detection
points = cv2.goodFeaturesToTrack(gray_image,mask=None,maxCorners =200,qualityLevel = 0.2,minDistance =7,blockSize=5)
return points
#defining lucas kanade method to find optical flow
def LK_model(gray_t_1,gray_t,prev_points):
#calculate lucas kanade optical flow
points ,status ,error = cv2.calcOpticalFlowPyrLK(gray_t_1,gray_t,prev_points,None,winSize =(15,15),maxLevel = 3,criteria =(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT,10,0.03))
return (points,status,error)
#Driver Script
#Create a VideoCapture object and read from input file
capture = cv2.VideoCapture('traffic.mp4')
#checking fps
fps = capture.get(5)
print('Frames per second : ', fps,'FPS')
#checking total number of frames
frame_count = capture.get(7)
print('Frame count : ', frame_count)
#obtaining frame size information
frame_width = int(capture.get(3))
frame_height = int(capture.get(4))
frame_size = (frame_width,frame_height)
#writing images into a video file
'''explaination
1.to save the captured frames as a video file.First, in order to be able to write our
file,a Video Writer object must be created.
2.This object takes as input parameters the output file name. It is worth mentioning
that Fourcc (4-character code of codec) is used to compress the frames.
3.The next parmeter is fps(frame per seconds) which control the how many framethat we
want to show per second in our output video.
4.And at last the frame_size which we have alredy mentioned above'''
output = cv2.VideoWriter('LK_optical_flow_output.avi',cv2.VideoWriter_fourcc('M','J','P','G'), 30, frame_size)
#taking first frame & finding corners on it.
color = np.random.randint(0,255,(200,3))
ret,prev_frame = capture.read() #reading frame by frame
prev_gray = cv2.cvtColor(prev_frame,cv2.COLOR_BGR2GRAY) #converting to grayscale
prev_points = find_features(prev_gray)'''step 1 Finding feature by calling find_feature
function'''
#creating mask image for drawing purpose
mask_image = np.zeros_like(prev_frame)
#reading further frames
while(capture.isOpened()): #Read until video is completed
#Capture frame-by-frame
ret,frame = capture.read()
if ret == True:
current_gray =cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
current_points,status,error = LK_model(prev_gray,current_gray,prev_points)
#Select good points
if current_points is not None:
current_points = current_points[status==1]
old_points = prev_points[status ==1]
for i,(current,prev) in enumerate(zip(current_points,old_points)):
a,b =current.ravel()
c,d = prev . ravel()
mask_image = cv2.line(mask_image,(int(a),int(b)),(int(c),int(d)),color[i].tolist(),2)
frame = cv2.circle(frame,(int(a),int(b)),5,color[i].tolist(),-1)
img = cv2.add(frame,mask_image)
#img = frame
# display resulting image
cv2.imshow('frame',img)
#cv2.imshow('mask_image',mask_image)
k = cv2.waitKey(20)
if k == 27:
cv2.destroyAllWindows()
break
'''set current frame as t-1 frame and update the current points as
the previous points(t-1)'''
prev_gray = current_gray.copy()
prev_points = current_points.reshape(-1,1,2)
output.write(img) #Write the frame into the file 'LK_optical_flow_output.avi'
else:
break
capture.release() #When everything done,release the video capture object
cv2.destroyAllWindows() #Closes all the frames
#Explaination
'''1.Optical flow is the pattern of apparent motion of image objects between two
consecutive frames caused by the movement of object or camera.
2.It is 2D vector field where each vector is a displacement vector showing the
movement of the points from first frame to second frame.
3.Optical flow works on some assumptions:
1.The pixel intensities of an object do not change between consecutive frame.
I(x,y,t) = I(x + dx,y+dy,t+dt), then taken taylor series Approximation of
right - hand side and removing common terms and after dividing by dt we will
getting :fx u + fy v + ft = 0 this optical flow eqaution.
2.Neighboring pixels have similar motion.
4.fx u + fy v + ft = 0 , in it we can find fx & fy they are image gradients.
similarly, ft is the gradient along time time.but(u,v) is unknown,we cannot
solve this one equation two variables.
5.To solve this problem we are going to use Lucas kanade method.
6.The lucas-kanade method is commonly used to calculate the optical flow for sparse
feature set.
7.The main idea of the this method based on a local motion constancy assumption,
where neighboring pixel have same displacement direction.
8.This Assumption helps to get the Approximated solution for the equation with two
variables.
9.As we have seen an assumption before, that all the neighboring pixels will have
simillar motion.lucas kanade method takes a nxn patch around the point.
so all the n*n points have the same motion.we can find (fx,fy,ft) for these n*n
points.so now our problem becomes solving n*n equations with two unknown vaiables
which is over determined.for that we will apply least square approach.
10.Till now we just overlooked the mathematical intiution behind lucas kanade method.
11.Now let us implement lucas optical flow in opencv:OpeCV provides all these in
single function cv2.calcOpticalFlowPyrLK().'''
#Procedure
'''12.Procedure to calculate lucas kanade optical flow in opencv:
13.first step is to detect the features points on initial frame(prev_gray)
14.for that first we need to read out the video ,we get initial frame(prev_frame) and
then we convert this to grascale image(prev_gray)
15.After grayscale conversion, we pass this prev_gray image to Shi-Thomsi corner
detector,which gives us initial feature points(prev_points).
16.In order to find the initial features points(prev_points) we are using function
cv2.goodFeaturesToTrack().it is an implementation of Shi-Thomshi corner detector.
17.The nice property of this function is that we can specify maximal number of
feature points that we want to detect and track.
18.Here, we are detecting maximum of 200 feature points.
19.Then we have creating mask image for drawing lines.'''
#computing optical flow
'''20.After Finding feature points(prev_points) next step is to use these
prev_points,feed them to lucas kanade algorithm and track them as current_points.
21.To compute optical flow with lucas kanade algorithm we use the function called
cv2.calcOpticalFlowPyrLK().
22.cv2.calcOpticalFlowPyrLK() calculates an optical flow for sparse features set
using the iterative lucas-kanade method with the pyramid.
23.To calculate current_points we have created LK_model()function by using
cv2.calcOpticalFlowPyrLK(),it takes parameters :
1.prev_gray : first input image
2.current_gray : second input image of same size and the same type as first input
image
3.prev_points : Vector of 2D points for which the flow needs to be found.
4.winSize :The size of search window at each pyramid.
5.maxLevel:0-based maximal pyramid levelnumber;if set to 0,pyramids are not
used(single level),if set to 1,two evels are used,and so on..
6.criteria : parameter, specifying the termination criteria of the iterative
search algorithm.
24.this wll returns :current_points,status,error'''
#Updation
'''25.And the final step is to loop over the whole video and to iteratively call LK
algorithm.In addition it will be neccessary to update a set pof tracked points.'''
'''26.Updation can be done like follwing statement, here we set current frame as
prev_frame and current_points as Prev_points.
1.prev_gray = current_gray.copy()
2.prev_points = current_points.reshape(-1,1,2)'''
'''27.In this All other functions used to read,save and visvualize the output.'''
#1.cv2.VideoCapture() #used to read the video file.
'''To create video writer object in order to save the captured
frames as a video file.'''
#2.cv2.VideoWriter()
#3.cv2.imshow() # To display image
#4.cv2.line() # to draw line
#5.cv2.cicle()# to draw circle
'''28.In output video, we can see some shi-thomshi features are in motion and some are
statics.and we can see how tracked the feature points between consecutive frames
when vehicles are in motion.i.e optical flow.'''