Ievgen Sobolev

Faculty of computer science and technology (CST)

Department of applied mathematics and informatics

Speciality Software engineering

Means of real and virtual spaces interaction based on three-dimensional scanning of objects using platform Microsoft Kinect 

Scientific adviser : PhD Victor Babkov

• CV
• Abstract

Abstract

Content

• Introduction
• 1. Relevance of the topic
• 2. The purpose and objectives of the study, expected results
• 3. Augmented reality systems
• 4. Three-dimensional scanning methods
• 5. Using Microsoft Kinect
• 6. Interim results
• Conclusion
• References

Introduction

The rapid development of computer science and technology provides a person an opportunity to take an alternate world that is different from reality. The term virtual reality refers to the world created by technical means, transmitted to man through his senses: sight, hearing, smell, touch, and others. Virtual reality simulates the impact and reactions to the impact. To create a convincing sense of reality is a complex computer synthesis reactions and properties of virtual reality is in real time. Objects of virtual reality is usually behave close to the behavior of similar objects of physical reality [1].

Augmented reality - a technology to complement the images of real objects with another objects of computer graphics, as well as to combine images from different sources. According to Ronald Azuma augmented reality must possess a number of features: combining real and virtual worlds, interactive, three-dimensional representation of objects[2].

The task of the three-dimensional scanning of real objects is becoming urgent at the junction of augmented and virtual reality.

Three-dimensional scanning - a systematic process for determining the coordinates of points belonging to the surfaces of physical objects with a view of obtaining their spatial mathematical models. The devices with which the scanning of objects is held, called the three-dimensional scanners. These devices not only simplify the process of creating three-dimensional models, but also allow us to solve this problem with the maximum degree of certainty with respect to the original object.

1. Relevance of the topic

It goes without saying that virtual and augmented reality technologies more firmly integrated into modern life. Their applicability is enormous: from games to education and medicine. To represent objects in the virtual and augmented reality it is used three-dimensional scanning, which is the most advanced method for obtaining an image of the object of the real world.

Thus, the master's work is devoted to actual scientific task of developing a method of combining virtual and augmented reality on the basis of three-dimensional scanning, aimed to be implemented on parallel computer systems. Microsoft Kinect is used as a three-dimensional scanner controller, and the tools of research are the MS Visual Studio 2010, MS Kinect for Windows SDK, platform .NET and programming language C #.

2. The purpose and objectives of the study, expected results

The aim of this study is to develop a method of combining objects of virtual and augmented reality, based on a three-dimensional scanning.

The main objectives of the study:

1. Analyze virtual and augmented reality building systems approaches on the basis of three-dimensional scanning.
2. Propose the method of integrated virtual and augmented reality based on a three-dimensional scanning technology, and focused on the implementation on parallel computer systems.
3. Show the proposed method on the parallel architecture of computer systems and architecture platform Microsoft Kinect.
4. Develop interactive subsystem in the training complex, using the proposed method.

The object of study: the process of combining virtual and augmented realities.

Subject of study: methods, algorithms and architecture for the implementation of the virtual and augmented realities merge, based on a three-dimensional scanning.

As part of the master's work it is planned to obtain the actual scientific results in the following areas:

1. Develop a virtual and augmented realities merge method based on the three-dimensional scanning.
2. Display the proposed method on the parallel architecture of computer systems and architecture platform Microsoft Kinect.
3. Implementation of this method as part of an interactive, non-contact system simulator.

3. Augmented reality systems

Layar project

Layar service allows you to use smartphones based on Android, iOS or OVI to receive real-time access to information about the world through his camera. It may be information about cafes, restaurants, hotels and other social places [4].

ARTag project

The project is dedicated to inserting digital models in a video stream from the camera [5].

AlterGeo project

Looking through the camera of your mobile device, users can now see a pop-up messages on the field and friends.[6]

Google Goggles

The project idea is in that to find information about the object in front of you, just take a picture of it. Goggles identifies products, famous landmarks, the facades of shops, art and popular images from the Internet [3].

Google’s "Project Glass" Augmented Reality Glasses

Development of Google which is under testing. Google’s glasses allow the user to receive additional information via speech [7].

Figure 1 - Google Project Glass - augmented reality example

4. Three-dimensional scanning methods

There are many methods to obtain projection data for modeling three-dimensional objects. A common feature of these methods - issue the search results in a point clouds describing the surface of an object or a system of interconnected objects. The cloud of points - a set of vertices in a three-dimensional coordinate system. These vertices are usually defined by the coordinates X, Y and Z, and, as a rule, intended to represent the outer surface of the object. An example of the object as a "cloud" of points shown in fig. 2.

Figure 2 - Point cloud representation of a car

Methods for obtaining point clouds can be divided into two categories: non-destructive methods of investigation of the internal structure of objects and methods of scanning the external surface of objects.

One of the most famous non-destructive methods of research - is tomography in its various manifestations. The principle of tomography is based on the irradiation of radiation research facility with a certain wavelength, and obtain the projection data with the help of sensors.

Functional representation of multidimensional geometric objects - is represented as a set of points in a multidimensional space. The major methods are polynomial interpolation using splines [12], methods based on triangulation, the method of surfaces, and others. The most common modern approach to solving this problem is the use of the RBF (radial basis function)[13].

5. Using Microsoft Kinect

Kinect is a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs. Based around a webcam-style add-on peripheral for the Xbox 360 console, it enables users to control and interact with the Xbox 360 without the need to touch a game controller, through a natural user interface using gestures and spoken commands.(рис. 3)

Figure 3 - Microsoft Kinect

Device capabilities with Microsoft Kinect for Windows SDK [16] [17]:
• detection and tracking of one or two people moving in the field of view of the sensor, using the tracking of parts of the skeleton (Fig. 4);
• Determining the distance from the sensor to the object using the XYZ-depth camera, which has access to the flow of colors and the flow of depth;
• Capture audio with noise and echo cancellation;
• Finding a location audio source using a beam search;
• Speech Recognition.

Figure 4 - Human recognition via Kinect

6. Interim results

At this stage, a prototype interactive subsystem of user interaction with virtual and augmented realities was developed. The main objective was to provide a native, intuitive and easy to control method for the movements of avatar in the virtual space. The main problem when designing the subsystem was converting the data obtained with the Kinect in high-level commands to control the avatar in the virtual world. The structure of the subsystem is shown in Figure 5.

Figure 5 - The structure of the subsystem
Animation. Size - 40 KB, duration - 8 sec, number of frames - 8

For any human move Kinect generates a new frame, indicating the current position of the body being tracked. The developed subsystem, based on these data, calculates the angles of the arms, legs, torso, and then comparing the current values with the values on the previous frame, or generates an event of the specific movements. Also, it should be noted that the thresholds are configurable that make the system comfortable for any user.

The program creates a listener that handles the generated events: depending on the type of event the system receives the appropriate commands (keystrokes, mouse movements) using the SendInput() method. This approach is very flexible and versatile as you can send commands to any active application. [18]

Conclusion

Master's study is devoted to actual scientific problem of combining virtual and augmented reality on the basis of three-dimensional scanning. In the trials carried out:

1. Analyzed virtual and augmented reality building systems approaches on the basis of three-dimensional scanning.
2. Based on the analysis of the literature highlights the main algorithms that can be used in the proposed approach to the combination of virtual and augmented realities.
3. Reviewed the possibilities of interactive systems using platforms Microsoft Kinect. Evaluated the software requirements.
4. Microsft Kinect for Windows SDK library was learned for programming using the Kinect.
5. A number of experiments on the use of the sensor Microsoft Kinect as a 3D scanner were held.
6. A prototype of an interactive subsystem user interaction with virtual and augmented realities was realized.

Further studies focused on the following aspects:

1. Improvement of the proposed method.
2. Display of the proposed method on the parallel architecture computer systems.
3. Analysis of the characteristics, performance and quality of the proposed method.
4. Development an application in the simulator training system using the proposed method.

Important notice

Master's work is not yet complete at the time of writing this essay. Estimated date of completion: December 2012. Full text of the paper, as well as materials on the subject may be obtained from the author or his scientific advisor after that date.

References

1. Wikipedia - виртуальная реальность [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://ru.wikipedia.org/wiki/Виртуальная_реальность
2. Ronald T. Azuma A Survey of Augmented Reality // In Presence: Teleoperators and Virtual Environments. – 1997. – № 4. – P. 355–385.
3. Google Goggles [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://www.google.com/mobile/goggles
4. Layar [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://www.layar.com/
5. ArTag [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://www.artag.net/
6. AlterGeo [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://altergeo.ru/
7. Google Project Glass [Electronic resourse] / Интернет-ресурс. - Режим доступа: https://plus.google.com/111626127367496192147/posts
8. AR-Door [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://ar-door.com/
9. Redmadrobot [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://redmadrobot.com/projects/
10. Дополненная реальность: состояние, проблемы и пути решения - Бойченко И.В., Лежанкин А.В. [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://www.tusur.ru/filearchive/reports-magazine/2010-1-2/161-165.pdf
11. Marching cubes [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://ru.wikipedia.org/wiki/Marching_cubes
12. Интерполяция сплайнами [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://alglib.sources.ru/interpolation/polynomial.php
13. RBF (Radial basis function) [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://en.wikipedia.org/wiki/Radial_basis_function
14. Kinect for Windows [Electronic resourse] / Интернет-ресурс. - Режим доступа:http://www.microsoft.com/en-us/kinectforwindows/
15. Ten Million Kinects [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://www.xbox.com/en-US/Press/archive/2011/0308-Ten-Million-Kinects
16. Kinect for Windows SDK [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://www.microsoft.com/en-us/kinectforwindows/develop/overview.aspx
17. Kinect for Windows Programming Guide [Electronic resourse] / Интернет-ресурс. - Режим доступа: http://msdn.microsoft.com/en-us/library/hh855348.aspx
18. Бабков В.С., Соболев Е.Г., Разработка подсистемы интерактивного взаимодействия в составе тренажерной системы с использованием платформы Microsoft Kinect, ИУС и КМ 2012 Материалы III Всеукраинской научно-технической конференции студентов, аспирантов и молодых ученых — Донецк, ДонНТУ — 2012, с. 685-689.