This project is done with the course Ambient Intelligence at Instituto Superior Tecnico. ToolSentinel implements a smart safe that makes access and access control easy by leveraging face recognition and speech recognition. The contributors to this project are (from the left):
- Mário Santos - 99275
- Daniel Gomes - 99195
- Simen Myrrusten - 108924
ToolSentinel has the goal of having simple and secure user interaction with a safe that stores valuable tools for industrial purposes. The users are authenticated using face recognition and can interact with speech. Furthermore, the manager can track back in time who used what, and when, by checking the logs. This could also be used for checking who is currently possessing each tool.
The project is a simple prototype of a bigger system initially designed. The project is meant to show the main features, while some implementation steps were not included. In addition, a computer was used for both face and speech recognition, due to the lack of a microphone and camera, but also for simplicity.
- Arduino UNO
- Windows computer
- Python - implementing stm, face recognition, speech recognition and stm.
- C - Arduino logic
- Miniconda - For environment management. Dependencies can be found in environment.yaml
For the implemented solution, we have this deployment diagram:
Finally, we created a simplified state machine to handle the user inputs and call on the right actions based on user input. The implemented STM can be found below.
This project requires a Windows machine and an Arduino kit. It could work on other OSs, but the camera connection might be wrongly configured. The physical setup is illustrated in the image below:
To install the dependencies conda was used. For a guide to installing Conda, turn to their web pages. You can use this guide:
https://conda.io/projects/conda/en/latest/user-guide/install/windows.html After installing conda, follow these steps to install dependencies:
-
Clone this repository to your local machine:
git clone [email protected]:nemisis84/ToolSentinel.git
-
Navigate to the project directory:
cd ToolSentinel -
Create a Conda environment (optional but recommended):
conda env create -f environment.yaml
This will create a new Conda environment with the required dependencies specified in
environment.yaml. -
Activate the Conda environment:
conda activate ToolSentinelEnv
Make packages out of the folders in the project:
-
Install the project:
pip install .This will install the project along with its dependencies.
-
Firstly, you need to download Arduino IDE from the official website, according to your computer operating system. Accept all the package installations and run the software.
-
Grab the Arduino Uno board and servo, and follow the steps:
-
With the use of cables, connect the red entry of the servo to the 5V entry of the board.
-
Connect the black cable of the motor to the ground entry of the board.
-
Connect the white cable of the motor to the pin number 9 of the board.
-
Connect the board to the computer, to the USB entry.
-
-
Go to the Arduino IDE and check the board that is currently selected, if there is none, then select the Arduino uno and check the port that it is connected to. Ex: COM3. This value will be used in step 5 so keep it in mind.
-
When the lights turn on the board, copy the code on the file arduino.c and paste it into the Arduino IDE. After that, compile and send it to the board.
-
Finally, check the stm.py file and change the SERIALPORT variable to match the port that you checked in step 3. With this, Arduino is set up and ready to go.
First, you would need to register a user. To register a user run:
python register.pyInsert your name when asked. A camera on your computer will pop up. Press space to take photos, 4-6 should be sufficent. Press ESC when you are done. The program will process and train the model with the photos.
You could add more users later or overwrite your registration using the same name. Please do not register yourself under multiple names.
To run the application:
python stm.pyThis will start a state machine that will accept and handle the user interface, interact with the speech recognition, the face recognition and log the actions.
You have the option of using it with and without an Arduino, and you will be asked about this when the application starts.
You could also choose between a text-based interface or speech recognition. Due to Python's design and our choices, the response time using speech recognition will be significant. We would also recommend in this case, to be in quiet circumstances and mute your microphone after your statement. We also mainly use "one" and "two" as input, since that way it's easier for the speech recognition to recognize.
Below is a demo of the text-based version. To use the voice-based version simply enter 2 instead of one on the first question.
For a manager, it could be useful to check the log if a tool is missing or broken. To do that run:
python get_log.pyYou will get the option to enter an integer to receive the n latest entries in the log or enter a name to get all records to that person. There are three types of entries:
- A tool is specified under the tool column and has a pickup timestamp and a return timestamp != -1. This is for tools that have been used and returned.
- A tool is specified under the tool column and has a pickup timestamp and a return timestamp = -1. These are tools that are currently being used.
- Under the tool column it says register, which is an entry to when a user is registered to the facial recognition.
Below you can see examples of both possible options to get the logs.
This project is licensed under the MIT license -see the LICENSE.md file for details.
For the facial recognition we used this open source GitHub repository with some own implementation changes:
https://github.com/carolinedunn/facial_recognition
Thanks to the author Caroline Dunn for providing open source code.