PopSignAI: recognising sign language on the phone in 7 milliseconds

Context

In PopSignAI, a player aims at a bubble and must perform the matching sign. The camera watches the player's hands, and the bubble is shot only when the sign is recognised. Research shows this generative practice teaches better than passive watching.

The dataset behind it, PopSign ASL v1.0, was the largest public isolated sign dataset of its time and became the foundation of Google's Isolated Sign Language Recognition competition on Kaggle.

The challenge

1. 01

   Real time on ordinary phones

   Recognition had to keep up with gameplay at 30 frames per second, on older devices, without a network connection.

2. 02

   Accuracy for people the model never saw

   Players are not the signers in the training data, so the model must generalise.

3. 03

   Data that is expensive to collect

   Every video needs native Deaf signers and careful annotation.

4. 04

   Reusable for other languages

   The pipeline had to be replicable, starting with Albanian Sign Language.

Our approach

1. 1

   Data at scale

   More than 210,000 labelled videos of 250 signs, recorded by 47 native Deaf signers. Videos were deliberately captured on Pixel 4a phones at medium quality, which lowered collection cost and produced the best recognition results.

2. 2

   Landmarks before pixels

   A custom MediaPipe plugin for Unity extracts 21 hand keypoints per frame at 30 fps. Keypoints are normalised for handedness and windowed to 60 frames, which keeps the model small and robust to lighting and background.

3. 3

   Model selection by latency

   LSTM and 2D CNN models were trained and compared; the lightweight 2D CNN was chosen for gameplay because it delivered the lowest latency at around 2.5 MB.

4. 4

   Validated on real devices

   Tested on a device matrix including iPhone 11, Pixel 4a, Pixel 6a and Samsung Galaxy S23 Ultra.

5. 5

   Ablation studies for new languages

   We established how much data a new sign language really needs: as few as 8 signers are adequate for gameplay, 20 to 30 signers give the best balance of generality and cost, with 20+ examples per sign.

Architecture

Everything runs on the device: no server, no network, no video leaves the phone. The pipeline, annotation tools and data collection software were released to support replication.