Óêðà¿íñüêà   Ðóññêèé
DonNTU   Masters' portal

Abstract

Content

Introduction

In today's world, technology is developing enormously. Such rapid development opens up new opportunities for humanity. The sphere of technology associated with virtual reality is developing rapidly and acquiring new significance for society. Virtual reality is a technically created world transmitted to a person through sensations such as vision, touch, hearing, etc. All this consists of two parts: software and hardware. In recent years, a breakthrough has been outlined in the hardware, that is, new devices have appeared, such as a VR helmet, gloves and whole suits of motion capture. At the moment, such devices are very common in the gaming field. But it is possible to use in more socially important areas. A good example is the training of young professionals in the medical field, especially in surgery. Using motion capture gloves can greatly help in training interns. For example, an experienced surgeon performed an operation wearing such gloves and all his hand movements were recorded. After that, the young specialist can try to repeat the operation and, if any errors are made, he will be signaled. Naturally, all this requires special software. Motion capture gloves can also be used in the field of people with disabilities. For example, a person cannot move around the house himself and, to make his life easier, he is given gloves and a special robotic device, with which a person can perform certain household chores (move, dig, clean, etc.). And the gloves themselves will serve as the governing body of this device, that is, with the help of various gestures, a person will control it.

1. Relevance of the topic

The relevance of the topic is that motion capture gloves are a very useful and progressive invention. And not less important is the task of exploring approaches to creating this device, as well as working with sensors used in the glove, processing signals from sensors and visualizing the received data and choosing a microcontroller to control the entire device.

2. The purpose and objectives of the study

The aim of the study is to create a sample of a motion capture glove with the ability to register and visualize the data obtained.

The main objectives of the study:

  1. Analysis of existing types of gloves designs.
  2. Analysis and selection of the microcontroller for the device under development.
  3. The choice of sensors for gloves.
  4. Analysis and selection of filters for the received data.
  5. Creating a 3D model for visualization.

3. Research and development review

CaptoGlove [1] (fig.1) is a glove that is able to accurately recognize a wide range of movements of both hands and fingers. All these movements are converted into control commands and can be transmitted to directly connected devices, such as PC, VR headsets, smart homes, as well as phones and tablets running on iOS / Android operating systems. This glove has a built‑in Bluetooth module, a three‑axis gyroscope, accelerometer and magnetometer, a barometer, 5 sensors that can sense the movement of fingers along the bend axis, 1 pressure sensor for the tip of the thumb, capable of sensing pressure from 100 g to 10 kg, as well as a predefined set advanced algorithms for various operating conditions.

Appearance of CaptoGlove gloves

Figure 1 – Appearance of CaptoGlove gloves.

Hi5 VR Glove [2] (fig.2) is a glove that includes 9‑DOF IMU sensors for 5 fingers and the back of the hand. There is also vibration feedback in the glove. The glove is autonomous and is powered by a battery that lasts for 7 hours of use. Data transmission is carried out using a radio module, while the delay does not exceed 5 ms. The transmission range is 5 m. The glove provides automatic channel switching to avoid radio frequency interference.

Hi5 VR Glove Appearance

Figure 2 – Hi5 VR Glove Appearance.

AcceleGlove [3] (fig.3) is a glove manufactured by AnthroTronix. For recognition of movements in this glove, accelerometers are used. Accelerometers are located just below each fingertip and on the back of your hand. When the user's hand moves, accelerometers can detect the three‑dimensional orientation of the fingers and palms with respect to Earth's gravity. Measured to within a few degrees, this information allows programs to distinguish even very slight changes in hand position. Accelerometers provide location information through light wires to the printed circuit board, which is located on the back of the hand. When the user makes a gesture, for example, squeezes his fingers together or holds his open palm outward, the board transfers data to the computer via a USB cable connected to the upper part of the wrist of the glove. The glove also receives power through the cord, avoiding the need for a bulky battery pack. AnthroTronix has created development software that allows users to adapt the glove to new goals. The user can create new functions for the glove by recording a gesture and assigning it a value; The program can store hundreds of gestures. The sensitivity with which a computer recognizes gestures can be different, so it can recognize sloppy, large gestures for an application, such as a children's educational program, or very accurate gestures for robotics. The system can also receive data from two gloves worn at the same time. All the earlier described gloves have a high cost. Therefore, there is an acute question in creating a more accessible version.

Appearance of AcceleGlove Glove

Figure 3 – Appearance of AcceleGlove Glove.

Gloves of this type are developed not only by commercial companies, but also by students and teachers of technical universities.

The glove [4] (fig.4), developed at Tallinn University of Technology, has 23 degrees of freedom. The main goal of the project was to develop a glove with inertial sensors only. The glove used 16 MPU6050 sensors, 3 on each finger and another 1 on the back of the hand. As the microcontroller used STM32F413ZH. Data transmission was carried out according to the I2C protocol with a frequency of 100 kHz. Since all MPU6050 modules have the same address, the 74HC4067 multiplexer is used. Using this multiplexer has the ability to recognize sensors. The microcontroller then transfers the data to the PC using the UART protocol. To process the data from the sensors, a complementary filter was used, which made it possible to level out certain shortcomings of the accelerometer and gyroscope. Also in this project, a 3D hand model was created in SolidWorks software, and transferred to Matlab Simulink for real‑time modeling.

Glove appearance developed at Tallinn University of Technology

Figure 4 – Glove appearance developed at Tallinn University of Technology.

4. Glove development

4.1 Sensor selection

The basis for the future information glove can be any fabric glove, to the extent of its availability and low cost. More important is the choice of sensor with which the movements of the hand and fingers will be recorded. The following options can be used as a sensor:

  1. The strain gauge;
  2. Inertial measuring unit (IMU);
  3. A sensor operating on the radiation effect;
  4. The potentiometer.

A strain gauge is a resistor whose resistance varies depending on the magnitude of its deformation. This device can be used as a bend sensor, that is, it can be used to track finger flexion. The strain gauge also has high measurement accuracy and a simple design. But to the best of its high cost, it was not used in the project.

The radiation‑effect sensor works as follows: there is a tube in which an LED is placed at one end and a photoresistor at the other. When the tube has a straight shape, the resistance is minimal. When the tube bends, the light flux decreases, as a result of which the resistance increases. This sensor was also not used in the project to the extent that it needs to be assembled independently, and the existing tubes are tight and bring discomfort when bending fingers.

An inertial measuring unit is an electronic device that measures and transmits the specific gravity of the body, the angular velocity, and sometimes the orientation of the body in space, using a combination of accelerometers, gyroscopes, and magnetometers. The project used the MPU6050 [5] and MPU9250 [6] modules. The MPU6050 is a micro‑electromechanical system (MEMS) which includes a three‑axis accelerometer and a three‑axis gyroscope. Using these devices, linear acceleration and angular velocity are determined, which in turn allows you to determine the position in space. The module also has a digital motion processor (DMP), which allows you to process the output of the accelerometer and gyroscope, thereby reducing the work performed by the microcontroller to which the sensor is connected. In general, the operation of such sensors based on the design is as follows: in a stationary state, the load that is on the suspensions remains in equilibrium, and the capacity on the capacitor plates is unchanged. Deviation along any axis leads to the displacement of the suspended load to the side, which will entail a change in the distance between the plates of the capacitor, and therefore the capacity. Further, the received data is amplified, passed through a series of filters and fed to the input of an analog‑to‑digital converter (ADC), which digitizes the data received from the capacitors. After that, the data is ready to be sent via one of the transmission interfaces for further processing. The module can transmit data using the I2C protocol. The MPU9250 is designed in the same way as the MPU6050 except that the first also has a three‑axis magnetometer. There were two options for placing sensors:

  1. 6 sensors (one on each finger and one on the back of the hand)
  2. 16 sensors (3 sensors on each finger, i.e. on each phalanx)

In the first embodiment, the sensor would be located at the tip of the finger and would have to perform additional calculations to display the movement of the entire finger. In the second case, each step of freedom would have its own sensor.

The potentiometer also found application in the project. But its use required the development of an additional design.

4.2 Construction development for potentiometer

With the help of which it would be possible to assess at what angle one phalanx of the finger has shifted relative to the other. The offset will be estimated by moving the potentiometer knob. The design (fig.5) was developed using Fusion 360 [7]software and printed on a 3D printer. The design was printed out of ABS + plastic, and Repetier Host [8] software was used for printing. The design consists of several parts: fastening to a shifting phalanx of the finger, 2 intermediate links and a fastening link to a static phalanx.The only significant drawback is that in order to evaluate the movements of all the fingers, a rather cumbersome design will be required.

The design was developed using Fusion 360 software

Figure 5 – The design was developed using Fusion 360 software.

4.3 Comparison of potentiometer and MPU6050 sensor readings

A study was also conducted with the help of which the accuracy of the readings of the MPU6050 module and the potentiometer was evaluated. The potentiometer and sensor were attached to the shaft, a servomotor that changed the angle from 0 to 90 degrees. A complementary filter was used to filter the sensor readings. Three graphs were taken for the X, Y, Z axis (fig. 6–8). Based on the graphs, a conclusion was made. That the readings of the potentiometer are more accurate, and it is necessary to use other filters for the Majwick, Mahoney or Kalman sensor, and when using the Z axis, it is necessary to use the MPU9250 module at all.

Graphs of signals from the potentiometer (orange color) and the MPU6050 sensor (blue color) along the Y axis

Figure 6 – Graph of signals from the potentiometer (orange color) and the MPU6050 sensor (blue color) along the Y axis.

Graphs of signals from the potentiometer (orange color) and the MPU6050 sensor (blue color) along the X axis

Figure 7 – Graph of signals from the potentiometer (orange color) and the MPU6050 sensor (blue color) along the X axis.

Graphs of signals from the potentiometer (orange color) and the MPU6050 sensor (blue color) along the Z axis

Figure8 – Graph of signals from the potentiometer (orange color) and the MPU6050 sensor (blue color) along the Z axis.

4.4 Microprocessor part and management program development environment

As a microcontroller that processes and transfers data to a PC, the STM32F407VE [9] is used. The choice of this microcontroller is determined by the following factors: this microcontroller has a high clock frequency, that is, the speed of the robot, has an additional processor to solve the mathematical operations that are necessary in this project, also has a large number of pins and has the ability to be programmed using the Waijung library in the Matlab package. Waijung library, is a Simulink library that can be used to easily and automatically generate C code from MATLAB / Simulink simulation models for many types of microcontrollers.

Currently, Waijung [10] has been specifically designed to support the STM32F4 family of microcontrollers (STM32F4 Target), which are STMicroelectronics Hi‑Performance & DSP MCUs.

Waijung Blockset supports 32‑bit and 64‑bit operating systems.

Waijung library has great functionality:

The following is an example of a program (fig.9) with a complementary filter for the MPU6050 sensor created using the Waijung library.

An example program with a complementary filter for the MPU6050 sensor created using the Waijung library

Figure 8 – An example program with a complementary filter for the MPU6050 sensor created using the Waijung library.

4.5 Hand movement visualization

Also in the Matlab package in the Simulink application, using the Simscape Multibody library, the future 3D model of the hand is also being developed. Which will in real time change its position in accordance with the real. The model (fig.10) consists of blocks Solid, Rigid Transform, Revolute Joint, Mechanism Configuration, World Frame. The Mechanism Configuration and World Frame blocks are essential for system health. Solid blocks contain 3D models: arm bones. The Rigid Transform block is used to transform the coordinate system. In the model, it was necessary to align the flank of the fingers. The Revolute Joint block is necessary in order to set the rotation of one phalanx of the finger relative to another. The rotation signal comes from the microcontroller, and he, in turn, receives them from sensors on the glove. Also, using this block, you can display the worked movement on the charts. The following is an example illustrating the movement of the index finger (fig.11).

Index finger model created using the Simscape Multibody library

Figure 10 – Index finger model created using the Simscape Multibody library.

3D model of the index finger (animation: 11 frames, 7 repeat cycles, 259 kilobytes)

Figure 11 – 3D model of the index finger
(animation: 11 frames, 7 repeat cycles, 259 kilobytes)

Conclusion

It is necessary to analyze the quality of filtering the output of the sensor using the Majwik, Mahoney, Kalman filter. And also conduct research on the quality of data processing by a digital motion processor (DMP).

References

  1. CaptoGlove [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://www.captoglove.com/.
  2. Hi5 VR Glove [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://hi5vrglove.com/.
  3. Ðó÷íîé èíòåðôåéñ: Ïåð÷àòêè AcceleGlove [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://www.popmech.ru/gadgets/9285-ruchnoy-interfeys-perchatki-acceleglove/.
  4. Motion tracking glove for augmented reality and virtual reality [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://www.degruyter.com/view/j/pjbr.2019.10.issue-1/pjbr-2019-0012/pjbr-2019-0012.xml.
  5. MPU6050 [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://www.invensense.com/products/motion-tracking/6-axis/mpu-6050/.
  6. MPU9250 [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://www.invensense.com/products/motion-tracking/9-axis/mpu-9250/.
  7. Fusion 360 [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://www.autodesk.com/products/fusion-360/students-teachers-educators.
  8. Repetier Host [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://www.repetier.com/download-now/.
  9. STM32F407VE [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: https://www.st.com/en/microcontrollers-microprocessors/stm32f407ve.html.
  10. Waijung [Ýëåêòðîííûé ðåñóðñ]. – Ðåæèì äîñòóïà: http://waijung.aimagin.com/.