``` = OpenVino Driver Behaviour :idprefix: :idseparator: - :sectanchors: :sectlinks: :sectnumlevels: 6 :sectnums: :toc: macro :toclevels: 6 :toc-title: Table of Contents

https://travis-ci.org/incluit/OpenVino-Driver-Behaviour#[image:https://travis-ci.org/incluit/OpenVino-Driver-Behaviour.svg?branch=master[Build Status]] https://sonarcloud.io/dashboard?id=incluit_OpenVino-Driver-Behaviour[image:https://sonarcloud.io/api/project_badges/measure?project=incluit_OpenVino-Driver-Behaviour&metric=alert_status[Sonarcloud Status]]

toc::[]

== Foreword This is a follow-up on the OpenVino’s inference tutorials:

Version 2019 R1.0

https://github.com/intel-iot-devkit/inference-tutorials-generic/tree/openvino_toolkit_2019_r1_0

Version 2018 R5.0

https://github.com/intel-iot-devkit/inference-tutorials-generic/tree/openvino_toolkit_r5_0

Version 2018 R4.0

https://github.com/intel-iot-devkit/inference-tutorials-generic/tree/openvino_toolkit_r4_0

We will work on and extend this tutorial as a demo app for smart cities, specifically for near misses detection.

[quote] Caution!

• As of OpenVINO’s Release 2019 R1, the model’s binaries are not included in the toolkit, as they are part of the https://github.com/opencv/open_model_zoo[model zoo]. You are supposed to download them manually as described in the https://github.com/intel-iot-devkit/inference-tutorials-generic/tree/openvino_toolkit_2019_r1_0/car_detection_tutorial#downloading-the-inference-models-from-the-open-model-zoo[tutorial]. Be aware that if you choose to download them in a different path than the default, our scripts/setupenv.sh will not fully work and you will have to add the path to the models yourself when running the program.
• The API got broken since 2019 R2, if you’re using an older OpenVINO version, run git checkout OpenVINO\<\=2019R1 and work from there.

== Introduction

This project consists on showcasing the advantages of the Intel’s OpenVINO toolkit. We will develop a Driver Behaviour case scenario, where we will detect drowsiness based on blinking and yawning and gaze direction. For that, we will use the OpenVINO toolkit and OpenCV, all written in {cpp}.

As mentioned previously, we will take the https://software.intel.com/en-us/articles/OpenVINO-IE-Samples#interactive-face-detection[Interactive face detection sample] as a starting point, as it provides us with the options to run and stack different models synchronously or asynchronously. We will develop the following features based on computer vision:

. Sleep/Drowsiness Detection: .. Counting frecuency of blinking. .. Yawn detection. . Gaze detection.

=== [Optional] Simulator

To test our system with data closer to reality we added support for https://store.steampowered.com/app/227300/Euro_Truck_Simulator_2/[ETS] or https://store.steampowered.com/app/270880/American_Truck_Simulator/[ATS]. As the simulator is not free, you can opt whether to compile the project with this feature or not. The communication between the simulator and our program is done via a ROS2 client and it provides the following info:

. Engine Status (On/Off) . Trailer Status (Connected/Disconnected). . Speed. . RPM. . Acceleration. . Position (Coordinates). . Gear (-1 for Reverse, >0 the rest).

=== [Optional] AWS (In Progress)

We also plan to send the data through MQTT using AWS IoT-Core, to produce a dashboard with the trucks positions, alarms, etc. Again, using AWS may incur in a cost, so this will also be optional for you to compile with/without it.

== Bussines Logic

Using OpenVino’s model detection we can easily detect faces with great accuracy. We are currently using for testing 2 different face detection models that are included with OpenVino out-of-the-box:

. face-detection-adas-0001 . face-detection-retail-0004

=== Blink/Yawn detection

Using the image detected inside the face ROI (region of interest), we feed a facial landmarks detector to identify points of iterest. Using 6 points for each eye and 6 points for the mouth it is possible to calculate ‘Eye Aspect Ratio (EAR)’ that gives 2 values for eye/mouth open or closed (based on http://vision.fe.uni-lj.si/cvww2016/proceedings/papers/05.pdf[this paper]).

image::https://github.com/incluit/OpenVino-Driver-Behaviour/blob/master/img/blink_detection_6_landmarks.jpg[EAR]

At the moment of writing this guide, the facial landmarks detection model included with OpenVino (facial-landmarks-35-adas-0001) has not enough points to run this calculations. We are using dlib’s facial landmarks detector instead.

Once we have a positive detection for blink/yawn, we count frames of those events and trigger an alarm when they hit a threshold.

=== ‘Eyes on the road’ detection

Using the face’s ROI, we feed a head-pose detector model provided by OpenVino (head-pose-estimation-adas-0001). Analizing the output of that model we can easily detect when the face is not centered or not looking to the front.

== Prerequisites

To run the application in this tutorial, the OpenVINO™ toolkit and its dependencies must already be installed and verified using the included demos. Installation instructions may be found at: https://software.intel.com/en-us/articles/OpenVINO-Install-Linux

If to be used, any optional hardware must also be installed and verified including:

• USB camera - Standard USB Video Class (UVC) camera.

File truncated at 100 lines see the full file