Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
mark0011astra authored Jan 2, 2025
1 parent 988eda4 commit 6b2d9b6
Showing 1 changed file with 0 additions and 140 deletions.
140 changes: 0 additions & 140 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,134 +1,3 @@
# Hand Tracking Mouse Controller with Kalman Filter

## Overview

#### This project implements a system that controls the mouse cursor using real-time hand tracking with MediaPipe Hands. Kalman filter and win32api are used for high accuracy, low latency tracking and ease of operation.

## Technical and Scientific Background

### MediaPipe Hands

[MediaPipe Hands](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker) is a framework developed by Google that provides high-fidelity hand and finger tracking by employing machine learning (ML) to infer 21 3D landmarks of a hand from just a single frame. This system uses MediaPipe Hands to detect the hand position and its 21 landmarks (e.g., finger joints) in real-time from camera input.

### Kalman Filter

The Kalman filter is an algorithm that uses a series of measurements observed over time, containing statistical noise and other inaccuracies, and produces estimates of unknown variables that tend to be more accurate than those based on a single measurement alone, by estimating a joint probability distribution over the variables for each timeframe. The filter is named after Rudolf E. Kálmán, one of the primary developers of its theory.

The Kalman filter has two distinct phases: **Predict** and **Update**.

* **Predict phase:** The Kalman filter produces estimates of the current state variables, along with their uncertainties, based on the system model.
* **Update phase:** The next measurement is used to refine these predictions by a weighted average, with more weight being given to estimates with higher certainty.

In this system, the hand position is considered the state, and the landmark positions obtained from MediaPipe Hands are the observed values. A constant velocity model is assumed, and the state is estimated using the following equations:

**State Transition Matrix (F):**
content_copy
download
Use code with caution.
Markdown

```math
Q = \begin{bmatrix} q1 & 0 & 0 & 0 \\ 0 & q2 & 0 & 0 \\ 0 & 0 & q3 & 0 \\ 0 & 0 & 0 & q4 \end{bmatrix}
```
where `dt` is the time interval between frames.

**Observation Matrix (H):**
content_copy
download
Use code with caution.

H = | 1 0 0 0 |
| 0 1 0 0 |

**Process Noise Covariance Matrix (Q):**
content_copy
download
Use code with caution.

Q = | q1 0 0 0 |
| 0 q2 0 0 |
| 0 0 q3 0 |
| 0 0 0 q4 |

where `q1`, `q2`, `q3`, and `q4` are parameters representing the magnitude of process noise.

**Measurement Noise Covariance Matrix (R):**
content_copy
download
Use code with caution.

R = | r1 0 |
| 0 r2 |

where `r1` and `r2` are parameters representing the magnitude of measurement noise.

By applying the Kalman filter, the noisy hand position data obtained from MediaPipe Hands is smoothed, resulting in smoother and more accurate mouse cursor movement.

### Mouse Control

The `win32api` library is used for mouse cursor control. The `win32api.SetCursorPos()` function moves the mouse cursor to the hand position estimated by the Kalman filter.

### Sensitivity Adjustment

The sensitivity of the mouse cursor (the amount of mouse cursor movement relative to hand movement) can be adjusted using the scaling factor `scaling_factor`. Increasing this factor increases the sensitivity. Fine-tuning of the sensitivity can also be achieved by adjusting the Kalman filter parameters `Q` and `R`.

## Operating Environment

* Windows 10/11
* Python 3.7+

## Required Libraries

* mediapipe
* opencv-python
* numpy
* filterpy
* pywin32

## Installation

```bash
pip install mediapipe opencv-python numpy filterpy pywin32
```
Place your hand in front of the camera, and the mouse cursor will move according to your hand movements.

### Configuration

DEBUG (Default: False): Set to True to output debug information.

scaling_factor (Default: 2.5): Adjusts the sensitivity of the mouse cursor.

kf.Q (Default: np.eye(4) * 0.4): Process noise covariance matrix for the Kalman filter.

kf.R (Default: np.eye(2) * 0.8): Measurement noise covariance matrix for the Kalman filter.

These parameters can be configured at the beginning of main.py.

### Troubleshooting

Program does not start: Ensure that all required libraries are installed.

Hand is not detected: Verify that the camera is connected correctly and that the operating environment meets the requirements of MediaPipe Hands.

Mouse cursor does not move: Check if win32api is installed correctly. You may need to run the script with administrator privileges.

Performance is slow: Lowering the values of min_detection_confidence and min_tracking_confidence may improve performance but could reduce accuracy.

### Contribution

Contributions to this project are welcome. Whether it's reporting bugs, suggesting features, or submitting pull requests, your contributions are appreciated.

### License

This project is released under the MIT License. See the LICENSE file for details.

Acknowledgements

This project utilizes the MediaPipe Hands and filterpy libraries. We thank the developers of these excellent libraries.




# Hand Tracking Mouse Controller with Kalman Filter

## Overview
Expand Down Expand Up @@ -249,15 +118,6 @@ This project utilizes the MediaPipe Hands and filterpy libraries. We thank the d
# Hand Tracking Mouse Controller with Kalman Filter
## 概要
Expand Down

0 comments on commit 6b2d9b6

Please sign in to comment.