Academia.eduAcademia.edu

Figure 9 – uploaded by Hema Koppula

See full PDFdownloadDownload figure
Fig. 9: Improved features for maneuver anticipation. We track facial landmark points using the CLNF tracker [4] which results in more consistent 2D trajectories as compared to the KLT tracker [58] used by Jain et al. [30]. Furthermore, the CLNF also gives an estimate of the driver’s 3D head pose.

Figure 9 Improved features for maneuver anticipation. We track facial landmark points using the CLNF tracker [4] which results in more consistent 2D trajectories as compared to the KLT tracker [58] used by Jain et al. [30]. Furthermore, the CLNF also gives an estimate of the driver’s 3D head pose.

Related Figures (14)

Fig. 1: Anticipating maneuvers. Our algorithm anticipates driving maneuvers performed a few seconds in the future. It uses information from multiple sources including videos, vehicle dynamics, GPS, and street maps to anticipate the probability of different future maneuvers.
Fig. 1: Anticipating maneuvers. Our algorithm anticipates driving maneuvers performed a few seconds in the future. It uses information from multiple sources including videos, vehicle dynamics, GPS, and street maps to anticipate the probability of different future maneuvers.
Fig. 2: (Left) Shows training RNN for anticipation in a sequence- to-sequence prediction manner. The network explicitly learns to map the partial context (x1,..,xz) Vé to the future event y. (Right) At test time the network’s goal is to anticipate the future event as soon as possible, i.e. by observing only a partial temporal context.
Fig. 2: (Left) Shows training RNN for anticipation in a sequence- to-sequence prediction manner. The network explicitly learns to map the partial context (x1,..,xz) Vé to the future event y. (Right) At test time the network’s goal is to anticipate the future event as soon as possible, i.e. by observing only a partial temporal context.
Fig. 3: Variations in the data set. Images from the data set [30] for a left lane change. (Left) Views from the road facing camera. (Right) Driving style of the drivers vary for the same maneuver.
Fig. 3: Variations in the data set. Images from the data set [30] for a left lane change. (Left) Views from the road facing camera. (Right) Driving style of the drivers vary for the same maneuver.
Fig. 4: System Overview. Our system anticipating a left lane change maneuver. (a) We process multi-modal data including GPS, speed, street maps, and events inside and outside of the vehicle using video cameras. (b) Vision pipeline extracts visual cues such as driver’s head movements. (c) The inside and outside driving context is processed to extract expressive features. (d,e) Using our deep learning architecture we fuse the information from outside and inside the vehicle and anticipate the probability of each maneuver.
Fig. 4: System Overview. Our system anticipating a left lane change maneuver. (a) We process multi-modal data including GPS, speed, street maps, and events inside and outside of the vehicle using video cameras. (b) Vision pipeline extracts visual cues such as driver’s head movements. (c) The inside and outside driving context is processed to extract expressive features. (d,e) Using our deep learning architecture we fuse the information from outside and inside the vehicle and anticipate the probability of each maneuver.
Fig. 5: Variable time occurrence of events. Left: The events inside the vehicle before the maneuvers. We track the driver’s face along with many facial points. Right: The trajectories generated by the horizontal motion of facial points (pixels) ‘t’ seconds before the maneuver. X-axis is the time and Y-axis is the pixels’ horizontal coordinates. Informative cues appear during the shaded time interval. Such cues occur at variable times before the maneuver, and the order in which the cues appear is also important.
Fig. 5: Variable time occurrence of events. Left: The events inside the vehicle before the maneuvers. We track the driver’s face along with many facial points. Right: The trajectories generated by the horizontal motion of facial points (pixels) ‘t’ seconds before the maneuver. X-axis is the time and Y-axis is the pixels’ horizontal coordinates. Informative cues appear during the shaded time interval. Such cues occur at variable times before the maneuver, and the order in which the cues appear is also important.
Fig. 7: Sensory fusion RNN for anticipation. (Bottom) In the Fusion-RNN each sensory stream is passed through their independent RNN. (Middle) High-level representations from RNNs are then combined through a fusion layer. (Top) In order to prevent over- fitting early in time the loss exponentially increases with time.
Fig. 7: Sensory fusion RNN for anticipation. (Bottom) In the Fusion-RNN each sensory stream is passed through their independent RNN. (Middle) High-level representations from RNNs are then combined through a fusion layer. (Top) In order to prevent over- fitting early in time the loss exponentially increases with time.
Fig. 8: Inside vehicle feature extraction. The angular histogram features extracted at three different time steps for a left turn maneuver. Bottom: Trajectories for the horizontal motion of tracked facial pixels ‘t? seconds before the maneuver. At t=5 seconds before the maneuver the driver is looking straight, at t=3 looks (left) in the direction of maneuver, and at t=2 looks (right) in opposite direction for the crossing traffic. Middle: Average motion vector of tracked facial pixels in polar coordinates. r is the average movement of pixels and arrow indicates the direction in which the face moves when looking from the camera. Top: Normalized angular histogram features.
Fig. 8: Inside vehicle feature extraction. The angular histogram features extracted at three different time steps for a left turn maneuver. Bottom: Trajectories for the horizontal motion of tracked facial pixels ‘t? seconds before the maneuver. At t=5 seconds before the maneuver the driver is looking straight, at t=3 looks (left) in the direction of maneuver, and at t=2 looks (right) in opposite direction for the crossing traffic. Middle: Average motion vector of tracked facial pixels in polar coordinates. r is the average movement of pixels and arrow indicates the direction in which the face moves when looking from the camera. Top: Normalized angular histogram features.
Fig. 10: AIO-HMM. The model has three layers: (i) Input (top): this layer represents outside vehicle features x; (ii) Hidden (middle): this layer represents driver’s latent states h; and (iii) Output (bottom): this layer represents inside vehicle features z. This layer also captures temporal dependencies of inside vehicle features. T’ represents time.
Fig. 10: AIO-HMM. The model has three layers: (i) Input (top): this layer represents outside vehicle features x; (ii) Hidden (middle): this layer represents driver’s latent states h; and (iii) Output (bottom): this layer represents inside vehicle features z. This layer also captures temporal dependencies of inside vehicle features. T’ represents time.
Fig. 11: Our data set is diverse in drivers and landscape.
Fig. 11: Our data set is diverse in drivers and landscape.
TABLE I: Maneuver Anticipation Results. Average precision, recall and time-to-maneuver are computed from 5-fold cross-validation. Standard error is also shown. Algorithms are compared on the features from Jain et al. [30]. to model the much needed long temporal dependencies for anticipation. Additionally, unlike AIO-HMM, F-RNN-EL is a discriminative model that does not make any assumptions about the generative nature of the problem. The results also highlight the importance of modeling the temporal nature in the data. Classifiers like SVM and Random Forest do not model the temporal aspects and hence performs poorly. but the driver does not perform any maneuver (i.e. driving straight); and (iv) missed predictions (mp): when it predicts driving straight but the driver performs a maneuver. We evaluate the algorithms using their precision and recall scores:
TABLE I: Maneuver Anticipation Results. Average precision, recall and time-to-maneuver are computed from 5-fold cross-validation. Standard error is also shown. Algorithms are compared on the features from Jain et al. [30]. to model the much needed long temporal dependencies for anticipation. Additionally, unlike AIO-HMM, F-RNN-EL is a discriminative model that does not make any assumptions about the generative nature of the problem. The results also highlight the importance of modeling the temporal nature in the data. Classifiers like SVM and Random Forest do not model the temporal aspects and hence performs poorly. but the driver does not perform any maneuver (i.e. driving straight); and (iv) missed predictions (mp): when it predicts driving straight but the driver performs a maneuver. We evaluate the algorithms using their precision and recall scores:
TABLE II: False positive prediction (fpp) of different algorithms. The number inside parenthesis is the standard error. a maneuver but the driver does not perform any maneuver (i.e. drives straight). Therefore low value of fpp is preferred. HMM F' performs best on this metric at 11% as it mostly assigns a high probability to driving straight. However, due to this reason, it incorrectly predicts driving straight even when maneuvers happen. This results in the low recall of HMM F at 36%, as shown in Table I. AIO-HMM’s fpp is 10% less than that of IOHMM and HMM E+ F, and F-RNN- EL is 3% less than AIO-HMM. The primary reason for false positive predictions is distracted driving. Drivers interactions with fellow passengers or their looking at the surrounding scenes are sometimes wrongly interpreted by the algorithms. Understanding driver distraction is still an open problem, and orthogonal to the objective of this work.
TABLE II: False positive prediction (fpp) of different algorithms. The number inside parenthesis is the standard error. a maneuver but the driver does not perform any maneuver (i.e. drives straight). Therefore low value of fpp is preferred. HMM F' performs best on this metric at 11% as it mostly assigns a high probability to driving straight. However, due to this reason, it incorrectly predicts driving straight even when maneuvers happen. This results in the low recall of HMM F at 36%, as shown in Table I. AIO-HMM’s fpp is 10% less than that of IOHMM and HMM E+ F, and F-RNN- EL is 3% less than AIO-HMM. The primary reason for false positive predictions is distracted driving. Drivers interactions with fellow passengers or their looking at the surrounding scenes are sometimes wrongly interpreted by the algorithms. Understanding driver distraction is still an open problem, and orthogonal to the objective of this work.
Fig. 12: Effect of prediction threshold p;;,. At test time an algorithm makes a prediction only when it is at least pz, confident in its prediction. This plot shows how Fl-score vary with change in prediction threshold.
Fig. 12: Effect of prediction threshold p;;,. At test time an algorithm makes a prediction only when it is at least pz, confident in its prediction. This plot shows how Fl-score vary with change in prediction threshold.
Fig. 13: Confusion matrix of different algorithms when jointly predicting all the maneuvers. Predictions made by algorithms are represented by rows and actual maneuvers are represented by columns. Numbers on the diagonal represent precision.
Fig. 13: Confusion matrix of different algorithms when jointly predicting all the maneuvers. Predictions made by algorithms are represented by rows and actual maneuvers are represented by columns. Numbers on the diagonal represent precision.
Related topics:
EngineeringComputer ScienceArtificial Intelligence
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved