CartooNect: Sensory motor playing system using cartoon actions

FUJIMURA Wataru1, IWDATE Shoto1, SHIRAI Akihiko1

1Kanagawa Institute of Techonology, Atsugi Japan

cartoonect at shirai.la

Abstract—CartooNect is a VR game system that used KINECT by the motion of whole body based on child's imagination by manga actions. In our plan of experiment, children or adult draw a picture by hand drawing on paper that background and properties like the world in manga and/or picture book. When a player stand in front of the system, the player's whole body is displayed to player in a real time with the drawn picture as a background and they can move free to play as an actor in virtual world. The outline of the system is that the effect happens on the screen according to the operation recognized because KINECT connected with the computer does the operation decided beforehand while controlled by OpenNI. When player make a pose, the visual and the sound effect can be reproduced to decorate their imagination. This project tells a message to children to show and share their kiddy imagination using whole body expressions. Through the embodiment of their own hero and the heroine imaginations using a latest VR techniques, visitors can enjoy to create the world interactively, collaboratively, freely, then they may understand new function of virtual reality motivations is not only in the computer but also human imaginations.

Keywords-KINECT; Natural Input, OpenNI, Manga, Interaction, drawing, sensory-motor play

I. Introduction

This project is focusing to find a new VR game system and its contents using current emerging video game technologies. Microsoft KINECT [1] (formerly Natal) is a most advanced device in this moment to detect player dynamic motions by a depth camera and its software.

As we did before at Nintendo Wii [2] and WiiRemote, a lot of hackers had tried to reveal and release the technology. However, it was rapidly shared via OpenNI[3] framework by the key technology suppliers, PrimeSence, Willow Garage and SideKick by free.

Then this project had thought again about what is a state of innovation then it took an orientation to realize “sensory motor play” in children imagination.

II. Sensory motor play in children immagination

“Sensory motor play” is the first stage of Piaget's cognitive development [4]. It can be observed at 0 to 24 months period basically, but it constructs most important function of intellectual development through their playing naturally. It can be explained as obtaining the external world to make interact with something from the brain. Then this stage is not limited in a baby, childhoods to teenagers are also studying slowly when they face to a new behavior in the external world.

Piaget also defined “Symbolic Play” and “Rule Play” stages after the sensory motor stage in development. They are very important to develop human linguistic skill and logical thinking and relations but they are also important to work out a game play in current video games design.

In a classic video games design, it always based on rule plays to define winners or losers. Logical human interfaces like buttons and sticks were functioned as logical input to measure players' rapidness, preciseness and toughness in a virtual with some scenario, logics and renditions. It may affect to children’s sensory motors if their parents do not limit the playing time in their development stage. Because of obtaining the sensory motor, it can give an amusing sense breezy when children just run in a field. It means the gaming system should focus to move players than showing complex things. This hypothesis can be applied not only for children but also adults in Virtual Reality entertainment system because they can be just a beginner like a baby in a virtual world in development.

<Japanese description and updates below> 日本語解説がページの右側になります

If you need academic papers, here is link of PDFs. 関連論文は以下のダウンロードページにあります．

http://www.shirai.la/downloads

（左の英語ページとは直接関係なく）

CartooNectから始まる，白井研のKINECT研究を紹介します．

この動画は投稿時．モーションの認識エンジンをOpenNIで開発し，Unity3DとXNAの両方でプロトタイピングしています．

その後，震災があり，コンテンツは「放射能汚染された大地にヒマワリを植える」というシリアスゲームに変更されました．

「溜めモーション」を実現したい，というアイディアはこのあたりで既に実装されています．

その後，ここで開発された技術はGAMICのアルゴリズムに生かされています．

なお，後期のCartooNectでは，現在のゲームで十字操作などでよく使われるデジタル入力をKinectの入力でエミュレーションする技術についても開発しています．これらの技術は，「AccuMotion」として一般化し，画像フィードバックを必要としない，より一般的な家電操作方法として特許出願中です．

III. Playing using cartoon actions

Cartoons like Manga, Bande-Dessinee (B.D.) and picture books give a great part of imaginations to children. Now we have a various contents from these cartoons to media like cinema, video games and TV animations. However, these media are mainly based on consuming models and most media are non-interactive exclude video games. It sounds a normal sense but it might limit human imagination and innovation into a consuming model. Because children can draw their original worlds and personas by themselves thanks emulating some creations of cartoons.

This project focuses to apply cartoon actions in virtual world to enhance such a creativity and motivation. The aim is not to develop a character merchandising game. It should be shared and be motivated the human creations which include drawing, motion and reactions in a virtual world.

「GAMIC: Game Action Motion Interaction Controller」ゲームアクション・モーションインタラクション・コントローラー．

KINECT世代のゲームアクションにおいて，高い一体感を演出する「溜めモーション」を実現するためのアルゴリズムと制御UI．

IV. Technology

“CartooNect” is a code name of supposed system. It is a coined word of Cartoon and KINECT. “KINECT” is also a coined word of kinematics and connect. Then it might be a best word to explain the technical characteristic “Connecting into cartoon world using Kinect” for the public.

A. Installation

白井研・Laval Virtualでの新作「PARAOKE：聴衆指向のカラオケエンタテイメントシステムのためのパラレル拡張現実」動画．

現在，Laval Virtual ReVolution2012での発表にむけ，開発中進行中です．

「AccuMotion」

現在発表準備中．

Figure 1. Concept sketch of project

CartooNect is very smart system. It is configured by KINECT, PC and a projector in 3x3m space. Through our experiments, it should keep 2.5m distance from KINECT to a player and each player should keep about 1m diameter or more. In our testing space, we had used an ultra-short range projector (NEC NP-U310WJD) with an aspherical mirror. It can be replaced by rear projection in an exhibition.

B. Software and Algorythms

OpenNI is an open and emerging framework to create “Natural Input”. It contains communication drivers to KINECT OS “XIRON” via USB connection for several platforms. It also has useful APIs to handle human kinematics like skeleton structure, transformation functions from depth map to real coordinates and/or projection coordinates with actual sample code. However it glows day by day then it should be refer the latest information at the website (www.openni.org) to obtain the technical details.

In the alpha development, we had tried to develop our own algorithms to recognize human motions to detect cartoon hero style actions. It had designed as Figure 3.

Figure 2. Example of a rule based motion detections

Figure 3 describes a series of motion detection to realize homage of a manga “DragonBall” fighting game. Through the development and testing, we had found it limits the player’s actions which just replay “pre-recorded actions” in a rule play using matching of motion. It is normal sense if we will apply well prepared model in a character merchandizing product but it is completely opposite with our concept. Additionally, the matching was also not worked well because it needs very precise motion in 3D and some player who has different shape could not control it well.

Figure 3. Example of a figure caption. (figure caption)

Figure 4 describes our final implementation, “part kinematics based detection” in a same context. In the detection of “Guard” and “Crouch”, it compares a part of series of bones with a target motion in 2D coordinates using simple dot products. An evaluation function for a target motion multiples Cosines of angles between target bone matrices and current player’s bone matrices. If each bone has very close value of angles, the output becomes to 1.0. This model can detect combination motion like “Crouching + Guard” by multistep values.

Figure 4. Natrual condition in prototype application

Figure 5. “Crouching” motion detected (face and background)

Figure 6. “Guard” motion detected. (a picture overrayed)

Figure 7. A combination detection and effect of Guard and Crouching.

Figure 4 – 7 are screenshots from our prototype application. To describe, they shows observing kinematics as lines on depth shadow of player. When it detects “Crouching” motion by legs, it changes the background and picture on the face. And when it detects “Guard” by arms, it shows a large picture on the player. Figure 7 is a result to play to make a combination effect of Guard and Crouching.

V. Contents and Experience

We are improving the system to create a valuable experience through “CartooNect”, a code name of proposing content. It is a coined word of Cartoon and KINECT. (“KINECT” is also a coined word of kinematics and connect.) Then it might be a best word to describe the technical characteristic “Connecting into cartoon world using Kinect” for the public.

Figure 8. Realized experience of “CartooNect”

Figure 9. Top view of floor plan in Laval Virtual ReVolution 2011

Figure 10. Scanning process to generate the scene.

Figure 11. A scene of CartooNect (in development)

In the experience, the visitors can choose a world or draw it by themselves. Background, props and cartoon calligraphies can be put by hand drawing and scanning. This design has a role to make immerse players by commitment to generated world.

All target motions can be saved as effects with drawn textures, calligraphies with particle effects. This is very simple implementation but it allows creating new motion interaction between players.

Figure 12. Players had invented their own harmonized action

Detected kinematics could be applied to a realistic 3D model but it looked not funny like a combat figure then we will develop a special character dynamic animation engine to improve CartooNect.

VI. Conclusion

CartooNect had been developed in a very short time (approximately 2 weeks in training period). It will be tested in more actual playful conditions to add new functions and experiences. Through this project, we are touching a new generation which is sensory-motor oriented computer entertainment. This point of view is more important than just developing an application of KINECT content. We would like to continue to share our knowledge and methodologies into actual virtual reality industry and next generation of video games.

Acknowledgment

The project thanks to Taisuke YAMASHITA, Tomohiro MIYAGAWA, Yukua KOIDE, Haruka OGUMA and Motofumi HATTORI for their creation, management and encouragement.

References

[1] Microsoft KINECT_TM, http://www.xbox.com/kinect

[2] Nintendo Wii®, http://www.nintendo.co.jp/wii/

[3] OpenNI, http://www.openni.org/

[4] Tracy S. Kendler, Levels of cognitive development. Routledge, 1995

[5] TORIYAMA Akira, Bird Studio, “Dragon Ball”, Shueisha.

[6] ARAKI Hirohiko, LUCKY LAND COMMUNICATIONS, “JoJo’s Bizzare adventure”, SHUEISHA, 1987-2005.

タイトル

・リアルタイムモーションの「溜め」を認識できるゲームの開発

【科学のひろば(2011/5/21)での展示】

概要

以前のモーションキャプチャーを用いたコンテンツやシステムは、専用の設備が無ければ開発することができなかった。

現在では、マイクロソフト社より販売されているKinectを用いることで、容易に開発を行うことができるようになった。

制作したゲームでは、全身を使った屈伸運動によって、キャラクターのジャンプアニメーションを操作する部分にKinectを用いることにした。

実際に制作してみると、Kinectのモーションキャプチャーデータから、プレイヤーが「溜め」をしてるか判断が難しいという問題が上がった。

私達は、プレイヤーのリアルタイムモーションキャプチャーデータと、事前に外部ファイルに用意しておいた数種類のモーションデータとの一致率を、関数によって評価値を求め、評価値がモーションごとに設定された閾値を満たしていた場合、そのモーションだと検知することができるアルゴリズムを作成した。

これにより、複数のモーションの検知ができるようになった。さらにモーション同士を関連付けすることで、決まったモーションをした後にのみ検知させることができるようにした。これによって「溜め」の状態を検知することができるようになった。

制作したゲームでは、しゃがんだ状態に伸びをした状態を関連付けてある。しゃがみ状態が検知されており、伸びを検知していない場合、プレイヤーは「溜め」を行っていると検知できる。

溜めを検知を出来たことで、プレイヤーのリアルタイムモーションとCGキャラクターのアニメーションを同調させることができ、プレイヤーがより一体感を得ることができる。