Dunhuang on Fingers was a project collaboratively done by 6 Tsinghua student and Tencent AI lab. I was mainly responsible for designing UI interface of the game and data collection and annotation for gesture recognition.

Dunhuang ( an ancient city in China ) on Fingers was a collectable card game on Wechat developed by 6 Tsinghua students with technological assistance from Tencent. In the game, we invited users to imitate the gestures in Buddha images to collect digital cards of Dunhuang frescoes.

Since Dunhuang frescoes were all dicovered in dark caves and were too fragile to be taken photos, we hope our work may add commemorative significance to tourists, as well as promoting traditional culture.

On-the-spot Investigation

The Mogao Caves, also known as the Thousand Buddha Grottoes, housed more than 50000 square meters of gorgeous murals drawn more than 1000 years ago.

Sketches of gestures

We made an on-the-spot investigation during Summer Holiday, trying to find breakthrough points for presenting ancient artistic works in a more digital, lively and accessible way. We found that although there were thousands of murals, almost none of them shared a same gesture. Craftsmen vividly depicted gestures of people and Buddhas playing instruments, holding objects and sitting in meditation.

We finally selected 2o kinds of gestures and recorded them on by sketching, as is shown in the above picture.

User Experience Design

To make the gesture interaction game a light-weight one which tourists can easily handle during their tour. We designed a cave with 10 kinds of Chinese tradition instruments and tools with flash of light around it. Once a user click one of them, the camera of the mobile phone will be waken.

Users then imitates the given gesture with their hands. They’ll be awarded with a ‘mural card’ once their imitations are close enough with the given one.

The following graph briefly demonstrated the interaction logic of our design.

Interface Design

We collected pictures of murals and referred to elements such as Buddhas and Geisha to design the title page of the game to make it more mural-style.

Gesture Recognition Algorithm

We collected 1000 pictures to train a model to solve the pattern recognition problem. We are mainly responsible for data collection and annotation. The model was mainly trained by workers in Tencent. The final algorithm was able to return a degree of confidence given an input picture.

Results show

The final version of the game has been removed after exhibition. However, a trial version is still available. Use Wechat to scan the following QR code and experience the gesture interaction game.