SMILE II: Automated sign language recognition at sentence level

Category Project

Ausgangslage und Ziele

The project builds on the groundwork of the SNSF Sinergia project SMILE I, in which an automated assessment system of manual components for DSGS at word level was developed. SMILE II extends this technology to sign language recognition and assessment of DSGS at sentence level production, including both manual and non-manual components.

Project Management

Tobias Haug Title Prof. Dr.

Position

Professor für Gebärdensprache und Partizipation bei Hörbehinderung / Leiter Bachelor Gebärdensprachdolmetschen

Facts

  • Duration
    01.2021
    12.2024
  • Project number
    4_45

Project Team

Financial support

Challenges

Some of the challenges in the project are: (a) lack of DSGS resources (e.g. language corpora); (b) continuous sign language contains both manual information (information on the hands/arms) and non-manual information (information on the torso, head, and face), and almost all work on recognition to date has focused on manual information only; (c) lack of standardized instruments for DSGS assessment; and (d) lack of automatic methods to assess continuous sign language productions at both the manual and non-manual level.

Collaborations and methods

To achieve the ambitious goals of the project, researchers from different fields will collaborate across four different institutions:

  • Idiap Research Institute, Martigny (automated language recognition)
  • University of Teacher Education in Special Needs (HfH) (sign language assessment)
  • University of Zurich (sign language linguistics)
  • University of Surrey (UK) (sign language technology)

HfH staff, with their expertise in sign language assessment, will employ the following methods:

  • Test development of a DSGS yes/no vocabulary test, a DSGS sentence repetition test with corresponding rating scale, a DSGS translation test, and a DSGS interview with corresponding rating scale
  • Statistical analysis of test results (Classical Test Theory and Rasch analyses)
  • Rater training, studies on rater reliability, comparisons between automated and human ratings
  • Development and analysis of feedback instruments for user studies (questionnaires, focus groups)

SMILE-II in sign language

You can see a project summary in Swiss German Sign Language in the following video. You can also activate German subtitles in the video.

Publications