S-TEL : A Sign Language Telephone Using Virtual Reality Technologies

Tomohiro Kuroda
Graduate School of Information Science
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara, 630-01, JAPAN
TEL: +81(7437)2-5274
FAX: +81(7437)2-5279
Internet: tomohi-k@is.aist-nara.ac.jp

Kosuke Sato
Graduate School of Information Science
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara, 630-01, JAPAN
TEL: +81(7437)2-5271
FAX: +81(7437)2-5279
Internet: sato@is.aist-nara.ac.jp

Kunihiro Chihara
Graduate School of Information Science
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara, 630-01, JAPAN
TEL: +81(7437)2-5270
FAX: +81(7437)2-5279
Internet: chihara@is.aist-nara.ac.jp

Web Posted on: December 12, 1997

1. Introduction

Although modern telecommunication systems have changed our social communication styles so drastically, audibly challenged people cannot take benefits of them based on phonetic media. Facsimiles and video phones might assist audibly challenged instead of telephone. However, the previous studies showed that they have some serious problems for daily practical use that is poor resolution and frame rate, and a privacy infringement. To solve the above problems, we developed S-TEL that is a new telecommunication system for sign language.

In this paper, the features of Japanese sign language are briefly explained in section 2 and some previous studies on computer aid for signers are mentioned in section 3. In section 4, the conceptual design of S- TEL is introduced. In section 5, the prototype of S-TEL is experimented.

2. Japanese Sign Language

Japanese Sign Language (JSL) is the mother tongue of the audibly challenged for a hundred years in Japan. As JSL is a visual language, there are some features in comparison with phonetic languages.

The meanings of the signs are defined by hands' feature, position, movement, and direction.
The emotional expressions are given by the modulation of the signs and the facial expressions. In some signs, the facial expression even varies the meanings of them.
Persons and their ranks are shown by the position to indicate the signs. Three-dimensional position of the signs denotes the situation of the current communication.

3. Sign Telecommunication

There are many works on computer aid for signers. Most of them focus on the communication between signer and non-signer. Only few attempts have so far been made at the communication among signer.

Some experiments using video phone for conversation in sign language were performed. These experiments cleared the following problems to be solved.

The received images do not have enough resolution, view, and frame rate to read signs and facial expressions.
User's eyes may break into the other's private space through taken images. The other's privacy may be trespassed.

Some telecommunication systems for signers have been examined. Most systems developed in United States are based on video phone. In these systems, researchers' main interest is to reduce transmitted data without degrading the communication in American sign language (ASL). In the other hand, most systems developed in Japan interpret input signs into symbolic codes and transmit them.

Although these systems can transmit the meanings of signs, the emotional expressions must be ignored unfortunately. Signers cannot make natural conversations through these systems.

4. Design of S-TEL

4.1 Design concepts and policies

To solve the above problems of the previous telecommunication systems, we introduce S-TEL that transmits the signs without losing emotional expressions and avoids invasions of privacy. For realizing these specifications, the following policies should be respected.

S-TEL treats the motions of hands and arms as geometric data from glove type sensors and magnetic positioning sensors.
S-TEL treats the face image data not to degrade emotional expressions.
S-TEL displays signs using real-time three dimensional CG which is used in virtual reality applications.
S-TEL functions via even low bit-rate data channel.

These above policies promise that S-TEL transmits whole emotional expressions and meanings of signs, and enables users to speak in signs naturally and to read signs easily without invasion of others' privacy.

4.2 Design overview

According to the above policies, we designed S-TEL as follows. S-TEL consists of following two components.

S-TEL sender : which obtains signs as geometric data and sends it. It measures hands, arms and upper body motions with three sensor components; two gloves and an intercom, each of them has a magnetic positioning sensor. Signs are treated as the bending angles of fingers and wrists and the positions and orientations of wrists and top of head. Facial expressions are also taken by CCD camera attached on the intercom. Those data sends to communication channel.
S-TEL receiver : which receives parametric data of signs and displays a virtual avatar according to reader's view point. It obtains the postures of hands and the position and orientation of wrists and top of head that are transmitted from the sender. The whole upper body actions of the avatar are estimated properly from these parameters by easy geometrical calculation. It obtains the head position of the reader from the magnetic positioning sensor attached on the intercom used in S-TEL sender also. The avatar is displayed using real-time three-dimensional CG same as virtual reality applications from reader's view point.

5. Prototype and experiments

A prototype of S-TEL along the design discussed in section 4 is developed. This prototype is a subset of S-TEL that does not treat facial expressions.

A developed prototype of S-TEL was experimented. Testee tries to talk in sign language through S-TEL and Internet video phone between Nara City and Kumamoto City in Japan. The distance between two cities is about 700 Km.

The platform used in the experiment is as follows.

S-TEL sender composed of Pentium 166MHz PC with windows95, two CyberGloves of Virtual Technologies Inc. and a Fastrak of Polhemus Inc.
Communication channel is Internet UDP/IP connection through JCSAT-1 of Japan Satellite Systems Inc. S-TEL uses only 19Kbps from 4.0Mbps bi-directional communication channel given by the satellite.
S-TEL receiver composed of Intergraph TD-5Z workstation ( Pentium 100MHz PC with OpenGL acceleration ) with Windows NT 3.51 and a Fastrak.
All software components are built on World Tool Kit Ver. 2.1 for Windows NT of Sense 8 Corp. and Visual C++ Ver 2.0 of Microsoft.
NV, an Internet video phone software, is used as a touchstone. The video stream is sent through 190 Kbps UDP/IP channel.
VAT, an Internet phone software, is used for voice communication.

Experimental result clears the following.

(1) The realized frame rate of S-TEL was 26.1 frame per sec.
(2) It is almost impossible to talk in Internet video phone in 190Kbps connection because of its poor frame rate and resolution.
(3) Users can talk in sign language on S-TEL. The CG generated by S-TEL has enough quality to read signs.

6. Summarize

In this paper, the conceptual design of S-TEL is presented. S-TEL acquires the signs as sensors' geometric data, transmits them as they are, and produces the signs by 3D CG from the data. Thus, S-TEL allows for talking and reading signs naturally, transmits the emotional expressions effectively, and protects users' privacy.

The experiments to talk in signs through prototype of S-TEL were performed. This clears the superiority of S-TEL over video phone as a telecommunication media for sign.

When S-TEL gets popular among audibly challenged people widely, their isolated community would associate together. S-TEL would increase the quality of their daily lives. Authors are integrating the transmission of facial expressions of signers into S-TEL in progress.

7. Notice

This study is cooperated with Mrs. Kumagai and Mr. Matsuda of Kyoto City Sign Language circle "Mimizuku", Kyoto City Hearing Disabilities Association, Wide Project and Japan Satellite Systems Corp.

"S-TEL: A Telecommunication System for Sign Language", (1996), Conference Companion of First Asia Pacific Computer Human Interaction, pp.83--91

"Reconstruction of Signer's Actions in a VR Telecommunication System for Sign Language", (1996), Proceedings of International Conference on Virtual Systems and Multimedia VSMM'96 in Gifu, pp.429--432

Go to the top of this page. | Go to the upper category.