GazeScroller - Using Facial Movements to perform Hands-free Gesture on the system

Overview

GazeScroller

Using Facial Movements to perform Hands-free Gesture on the system

Abstract

As our world is getting digitized on an fast rate, every person is having a device that is making life better. Also, there is a considerate amount of the society that do not have interactions as others to these devices. One such example are the quadriplegic people (people suffering from paralysis) which constitute to 5.4 million people people in the world*. Our aim here is to make them interact with the digital world. In this project, facial movements of the person's face is fed to the system on real-time and a certain list of operations can be performed on the system using these facial actions.Additionally, we will extend this system to mini-games on the internet like the Dino Game. Finally, I have evaluated the system by five people and found that they have positively to the system. These results imply that we can generalise this system to the entire world.

Approach

The project captures live stream of the video via webcam of the system. It then maps the face to 68 landmark points via the library Dlib. The movements of the points corresponding to the eye and nose are monitored continously. The functionalities covered in the project include : • Detect blink of one eye to enable/disable scrolling. • Detect the scroll movement based on the movement of the point on the nose. Using Blink to toggle scroll and head direction to scroll

Background Study

Blinking is an involuntary action of a human being.Blinks can be spontaneous, reflex and voluntary, and eye blink rate depends on various factors including environmental factors, type of activity.

In order to segregate natural blink of the eye with the intentional blink of one eye of the user for functionality 1 as discussed above, I have studied the eye width ratios of by conducting experiments study over 5 users with each subject testing for 10 times. This data analysis is used to understand to difference in the eye width ratio between both the eyes to when a user blinks one of the eye. Secondly, the intentional blink of the eye is put on a threshold for 3 frames to detect blink. These procedures helped detect the intentional one eye blink from the natural blink of the eyes. The information from the Fig 1 gives us the details of the eye ratio and the delta (difference between the eye ratios). We take the mean and use them as a reference in our code as threshold.

Technical Tools :

• Dlib - a library used to detect face per frame via webcam • Python - language to write the code • landmarksPoints.dat file - this file is used to superimpose landmarks onto the face detected. • pynput - library to invoke keyboard and mouse keys.

System Setup :

By using the tools of mentioned above, we get the face of the user per frame superimposed by landmark points. Calculations for each frame include :

rightEyeWidthRatio = height of the right eye/ width of the right eye leftEyeWidthRatio = height of the left eye/ width of the left eye delta = abs(leftEyeWidthRatio - rightEyeWidthRatio) Whenever a user blinks one eye, following cases are checked • Check 1 : if delta > threshold of delta taken from fig.1 • Check 2 : if leftEyeWidthRatio < threshold value of blink and frame count is 3. • If Check 1 and Check 2 true , trigger Blink and enable scrolling. UX Aspects : Trigger notifications in the system when scrolling is toggled.

Discussion & Future Scope:

In the present work I have not made much effort into perfectly the model and in CV. I have worked towards the thresholds and correlating to the use case I mentioned in the abstract. If substantial work is detecting the exact eye wink using ML models, the system would be much better. The false blinks being recorded is because we lack a model here. In the future scope , we can use this feature to build interactive games to the quadriplegic people to improve their psychological status too.

Conclusion :

All the subjects who have tested responded positively to the system and felt good about it. Therefore, we can say that our system is performing good to scroll pages using the nose and to capture the blink of the eye as a toggle gesture.

Hence, such a model will be beneficial to quadriplegic people and help them to interact with the digital world.Since the false blinks are low, the system is good to be used. It can be further perfected with ML models to give better accuracy to be used by the quadriplegic people.

Python scripts performing class agnostic object localization using the Object Localization Network model in ONNX.

ONNX Object Localization Network Python scripts performing class agnostic object localization using the Object Localization Network model in ONNX. Ori

Ibai Gorordo 15 Oct 14, 2022
Make a surveillance camera from your raspberry pi!

rpi-surveillance Make a surveillance camera from your Raspberry Pi 4! The surveillance is built as following: the camera records 10 seconds video and

Vladyslav 62 Feb 03, 2022
An AI Assistant More Than a Toolkit

tymon An AI Assistant More Than a Toolkit The reason for creating framework tymon is simple. making AI more like an assistant, helping us to complete

TymonXie 46 Oct 24, 2022
Custom implementation of Corrleation Module

Pytorch Correlation module this is a custom C++/Cuda implementation of Correlation module, used e.g. in FlowNetC This tutorial was used as a basis for

Clément Pinard 361 Dec 12, 2022
Neural Koopman Lyapunov Control

Neural-Koopman-Lyapunov-Control Code for our paper: Neural Koopman Lyapunov Control Requirements dReal4: v4.19.02.1 PyTorch: 1.2.0 The learning framew

Vrushabh Zinage 6 Dec 24, 2022
A script helps the user to update Linux and Mac systems through the terminal

Description This script helps the user to update Linux and Mac systems through the terminal. All the user has to install some requirements and then ru

Roxcoder 2 Jan 23, 2022
Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

Implementation for Pixel Consensus Voting (CVPR 2020). This codebase contains the essential ingredients of PCV, including various spatial discretizati

Haochen 23 Oct 25, 2022
Neural network-based build time estimation for additive manufacturing

Neural network-based build time estimation for additive manufacturing Oh, Y., Sharp, M., Sprock, T., & Kwon, S. (2021). Neural network-based build tim

Yosep 1 Nov 15, 2021
The official start-up code for paper "FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark."

FFA-IR The official start-up code for paper "FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark." The framework is inheri

Mingjie 28 Dec 16, 2022
A simple configurable bot for sending arXiv article alert by mail

arXiv-newsletter A simple configurable bot for sending arXiv article alert by mail. Prerequisites PyYAML=5.3.1 arxiv=1.4.0 Configuration All config

SXKDZ 21 Nov 09, 2022
Data and codes for ACL 2021 paper: Towards Emotional Support Dialog Systems

Emotional-Support-Conversation Copyright © 2021 CoAI Group, Tsinghua University. All rights reserved. Data and codes are for academic research use onl

126 Dec 21, 2022
Python Single Object Tracking Evaluation

pysot-toolkit The purpose of this repo is to provide evaluation API of Current Single Object Tracking Dataset, including VOT2016 VOT2018 VOT2018-LT OT

348 Dec 22, 2022
This is the code repository for the paper "Identification of the Generalized Condorcet Winner in Multi-dueling Bandits" (NeurIPS 2021).

Code Repository for the Paper "Identification of the Generalized Condorcet Winner in Multi-dueling Bandits" (To appear in: Proceedings of NeurIPS20

1 Oct 03, 2022
Official PyTorch implementation of PS-KD

Self-Knowledge Distillation with Progressive Refinement of Targets (PS-KD) Accepted at ICCV 2021, oral presentation Official PyTorch implementation of

61 Dec 28, 2022
Shared Attention for Multi-label Zero-shot Learning

Shared Attention for Multi-label Zero-shot Learning Overview This repository contains the implementation of Shared Attention for Multi-label Zero-shot

dathuynh 26 Dec 14, 2022
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

Junjie Hu 13 Dec 10, 2022
CPPE - 5 (Medical Personal Protective Equipment) is a new challenging object detection dataset

CPPE - 5 CPPE - 5 (Medical Personal Protective Equipment) is a new challenging dataset with the goal to allow the study of subordinate categorization

Rishit Dagli 53 Dec 17, 2022
This repository contains the code for our paper VDA (public in EMNLP2021 main conference)

Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models This repository contains the code for our paper VDA (publ

RUCAIBox 13 Aug 06, 2022
Code for "The Box Size Confidence Bias Harms Your Object Detector"

The Box Size Confidence Bias Harms Your Object Detector - Code Disclaimer: This repository is for research purposes only. It is designed to maintain r

Johannes G. 24 Dec 07, 2022
Implementation for the "Surface Reconstruction from 3D Line Segments" paper.

Surface Reconstruction from 3D Line Segments Surface reconstruction from 3d line segments. Langlois, P. A., Boulch, A., & Marlet, R. In 2019 Internati

85 Jan 04, 2023