Real-Time Sign Language Translation System Using Computer Vision & NLP

Abstract

This thesis presents a real-time, bidirectional sign language communication system that combines a lightweight Transformer recognizer, a Large Language Model (LLM), and a full NLP pipeline to enable natural two-way conversation between deaf and hearing users on standard hardware. Operating as a translation system, it classifies a 100-sign ASL vocabulary through its main SignLanguageNet classifier while integrating the 26-letter fingerspelled alphabet via a dedicated Alphabet Mode subsystem. In the deaf-to-hearing direction, a standard webcam captures movements, Google MediaPipe Holistic extracts 3D skeletal landmarks, and a hybrid Causal Convolutional and Transformer architecture classifies signs using a model trained on the Google ISLR dataset. Recognized gloss sequences are translated into coherent English sentences by an LLM and spoken via a Text-to-Speech engine. In the hearing-to-deaf direction, a Speech-to-Text engine transcribes spoken responses on-screen, completing the loop. Trained using Adversarial Weight Perturbation, mixed-precision, and OneCycleLR scheduling, the model achieves 91.4% top-1 validation accuracy. This system transforms isolated sign recognition into a practical communication assistant. Future work will extend this to continuous signing and additional sign languages.