Description: Description:       Intelligent Vehicles that Can See






Data Mining of Naturalistic Driving Video

Nowadays, many vehicles are equipped with a vehicle borne camera system for monitoring drivers’ behavior, accident investigation, road environment assessment, and vehicle safety design. Huge amount of video data is being recorded daily. Analyzing and interpreting these data in an efficient way has become a non-trivial task. As an index of video for quick browsing, this work maps the video into a temporal image of reduced dimension with as much intrinsic information as possible observed on the road. The perspective projection video is converted to a top-view temporal profile that has precise time, motion, and event information during the vehicle driving. Then, we attempt to interpret dynamic events and environment around the vehicle in such a continuous and compact temporal profile. The reduced dimension of the temporal profile allows us to browse the video intuitively and efficiently.


Mehmet Kilicarslan, Jiang Yu Zheng, Visualizing Driving Video in Temporal Profile, IEEE Intelligent Vehicle Symposim, 2014, 1-7.


Pedestrian Detection from Driving Video

Pedestrian detection has been intensively studied based on appearances for driving safety. Only a few works have explored between-frame optical flow as one of features for human classification. In this paper, however, a new point of view is taken to watch a longer period for non-smooth movement. We explore the pedestrian detection purely based on motion, which is common and intrinsic for all pedestrians regardless of their shape, color, background, etc. We found unique motion characteristics of humans different from rigid object motion caused by vehicle motion.

My Publications:

M. Kilicarslan, J. Y. Zheng, K. Raptis, Pedestrian detection from motion, 23th International Conference on Pattern Recognition, 1857-1863, 2016. (video)

Mehmet Kilicarslan, Jiang Yu Zheng, Aied Algarni, Pedestrian detection from non-smooth motion, IEEE Intelligent Vehicles Symposium (IV), 2015, pp. 487-492

Mehmet Kilicarslan, Jiang Yu Zheng, Detecting walking pedestrians from leg motion in driving video, 2014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC), pp. 2924-2929

 Bicyclists Detection in Driving Video

Monocular camera based bicyclist detection in naturalistic driving video is a very challenging problem due to the high variance of the bicyclist appearance and complex background of naturalistic driving environment. In this paper, we propose a two-stage multi-modal bicyclist detection scheme to efficiently detect bicyclists with varied poses for further behavior analysis. A new motion based region of interest (ROI) detection is first applied to the entire video to refine the region for sliding-window detection. Then an efficient integral feature based detector is applied to quickly filter out the negative windows. Finally, the remaining candidate windows are encoded and tested by three pre-learned pose-specific detectors. The experimental results on our TASI 110 car naturalistic driving dataset show the effectiveness and efficiency of the proposed method. The proposed method outperforms the traditional methods.

My Publications:

C. Liu, R. Fujishiro, L. Christopher, J. Y. Zheng, Vehicle-bicyclist dynamic position extracted from naturalistic driving videos, IEEE Transactions on Intelligent Transportation Systems, 18(4), 734-742, 2017.

Kai Yang, Chao Liu, Jiang Yu Zheng, Lauren Christopher, Yaobin Chen, Bicyclist detection in large scale naturalistic driving video, 2014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC), 2015, pp. 1638-1643.


Vehicle Collision Alert based on Visual Motion

This work models various dangerous situations that may happen to a driving vehicle on road in probability, and determines how such events are mapped to the visual field of the camera. Depending on the motion flows detected in the camera, our algorithm will identify the potential dangers and compute the time to collision for alarming. The identification of dangerous events is based on the location-specific motion information modeled in the likelihood probability distributions. The originality of the proposed approach is at the location dependent motion modeling using the knowledge of road environment. This will link the detected motion to the potential danger directly for accident avoidance. The mechanism from visual motion to the dangerous events omits the complex shape recognition so that the system can response without delay.

My Publications:

M. Kilicarslan, J. Y. Zheng, Predict Collision by TTC from Motion using single camera, IEEE Transaction on Intelligent Transportation Systems, 2018. (video)

M. Kilicarslan, J. Y. Zheng, Direct vehicle collision detection from motion in driving video, IEEE Intelligent Vehicles, 2017, 1558-1564.

M. Kilicarslan, J. Y. Zheng, Bridge motion to collision alarming using driving video, 23th International Conference on Pattern Recognition, 1870-1875, 2016.

Mehmet Kilicarslan, Jiang Yu Zheng, Modeling Potential Dangers in Car Video for Collision Alarming, IEEE International Conference on Vehicle Electronics and safety 2012, 1-6

Mehmet Kilicarslan, Jiang Yu Zheng, Towards Collision Alarming based on Visual Motion. IEEE International Conference on Intelligent Transportation Systems 2012: 1-6


Road Appearances and Edge Detection in All-weathers and Illuminations

To avoid vehicle running off road, road edge detection is a fundamental function. Current work on road edge detection has not exhaustively tackled all weather and illumination conditions. We first sort the visual appearance of roads based on physical and optical properties under various illuminations. Then, data mining approach is applied to a large driving video set that contains the full spectrum of seasons and weathers to learn the statistical distribution of road edge appearances. The obtained parameters of road environment in color on road structure are used to classify weather in video briefly, and the corresponding algorithm and features are applied for robust road edge detection. To visualize the road appearance as well as evaluate the accuracy of detected road, a compact road profile image is generated to reduce the data to a small fraction of video. Through the exhaustive examination of all weather and illuminations, our road detection methods can locate road edges in good weather, reduce errors in dark illuminations, and report road invisibility in poor illuminations.



Z. Wang, G. Cheng, J. Y. Zheng, All weather road edge identification based on driving video mining. IEEE Intelligent Transportation Systems Conference, 2017.

G. Cheng, Z. Wang, J. Y. Zheng, Big-video mining of road appearances in full spectrums of weather and illuminations, IEEE Intelligent Transportation Systems Conference, 2017.

Night Road and Illuminations

Pedestrian Automatic Emergency Braking (PAEB) for helping avoiding/mitigating pedestrian crashes has been equipped on some passenger vehicles. Since approximately 70% pedestrian crashes occur in dark conditions, one of the important components in the PAEB evaluation is the development of standard testing at night. The test facility should include representative low-illuminance environment to enable the examination of the sensing and control functions of different PAEB systems. The goal of this research is to characterize and model light source distributions and variations in the low-illuminance environment and determine possible ways to reconstruct such an environment for PAEB evaluation. This paper describes a general method to collect light sources and illuminance information by processing large amount of potential collision locations at night from naturalistic driving video data. This study was conducted in four steps. (1) Gather night driving video collected from Transportation Active Safety Institute (TASI) 110 car naturalistic driving study, particularly emphasizing locations with potential pedestrian collision. (2) Generate temporal video profile as a compact index toward large volumes of video, (3) Identify light fixtures by removing dynamic vehicle head lighting in the profile and stamp them with their Global Positioning System (GPS) coordinates. (4) Find the average distribution and intensity of illuminants by grouping lighting component information around the potential collision locations. The resulting lighting model and setting can be used for lighting reconstruction at PAEB testing site.

My Publications:

Libo Dong, Stanley Chien, Jiang-Yu Zheng, Yaobin Chen, Rini Sherony, Hiroyuki Takahashi, Modeling of Low Illuminance Road Lighting Condition Using Road Temporal Profile, SAE Technical Paper, 2016, pp. 1638-1643