Implementation of 34.7 fps Pose and Gaze Estimator for Real-Time Driver-Vehicle Interaction System

With the recent rapid development of autonomous driving, many researches on intelligent in-vehicle interaction technologies have been studied. Real-time driver behavior analysis is a key function for various in-vehicle interaction ap-plications. An important performance indicator here is real-time quality because it is directly related to safety. Therefore, in this paper, we design the convolutional neural network (CNN) architecture suited to in-vehicle driver behavior analysis using pose and gaze. First, we define the 11 key points for driver pose and gaze, and modeled a CNN architecture that can detect them quickly. The proposed architecture was re-generated and re-trained with layer reduction for high speed based on the residual CNN model. Furthermore, the hardware implementation result based on FPGA platform to verify the real-time performance are presented. In order to implement real-time interaction, an image processing speed of more than 30 fps and a latency time of less than 100 ms are generally required. We selected and implemented the FPGA platform to meet these requirements. The designed hardware architecture was implemented at the RTL level of the VCU118 FPGA, and simulation results show 34.7 fps and 75.3 ms latency. Finally, we implemented the driver pose and gaze estimator on the FPGA based hardware platform to experiment the driver-vehicle interaction system with the demo application. The detected pose and gaze results were transmitted to the GPU board in real time, reliably supporting 30 fps, and verified application to screen control and driver monitoring applications.