Implementasi Whisper Fine-Tuning sebagai Model Closed Caption pada Browser Extension untuk Aksesibilitas Tunarungu
Article Sidebar
Main Article Content
Abstract
Pengguna tunarungu menghadapi hambatan signifikan dalam mengakses konten audio digital seperti video conference, podcast, dan siaran langsung, karena minimnya closed caption otomatis berbahasa Indonesia yang dapat diandalkan. Kondisi ini diperparah oleh jumlah penyandang disabilitas di Indonesia yang mencapai lebih dari 22 juta jiwa, dengan kelompok tunarungu sebagai salah satu bagian terbesar yang membutuhkan solusi aksesibilitas digital yang konkret. Penelitian ini mengimplementasikan model Whisper hasil fine-tuning sebagai model utama untuk closed caption Bahasa Indonesia pada browser extension bernama CaptionAI, dengan model Whisper baseline (zero-shot) dan Fast Conformer sebagai model pembanding untuk mengevaluasi sejauh mana fine-tuning meningkatkan performa model ASR berskala kecil yang dapat di-hosting mandiri. Model Whisper varian small di-fine-tune penuh menggunakan dataset gabungan FLEURS, Common Voice 17, dan Common Voice 19 Bahasa Indonesia (26.400 sampel), dengan konfigurasi learning rate 5e-6, batch size efektif 32, dan max_steps 1.000 langkah menggunakan HuggingFace Seq2SeqTrainer. Checkpoint terbaik dipilih otomatis melalui EarlyStoppingCallback berdasarkan Word Error Rate (WER) validasi terendah, lalu diekspor ke format ONNX untuk deployment. Evaluasi pada 800 sampel test yang belum pernah dilihat model selama pelatihan menunjukkan model Whisper hasil fine-tuning mencapai WER 15,84% dan Character Error Rate (CER) 5,35%, unggul signifikan dibanding Whisper baseline zero-shot (WER 30,83%, CER 9,29%) maupun Fast Conformer (WER 34,72%, CER 15,40%), dengan latensi rata-rata 0,474 detik yang kompetitif untuk live captioning meski belum memenuhi standar real-time ketat di bawah 400 milidetik. Sistem berhasil diimplementasikan sebagai browser extension dengan server backend FastAPI, menerapkan pola inferensi paralel antara Whisper ONNX lokal dan Groq API sebagai jalur cloud. Evaluasi usability melalui System Usability Scale (SUS) terhadap 25 responden menghasilkan skor rata-rata 67,1 dari skala 100 (kategori "ok"), sementara pengujian black-box terhadap 12 skenario fungsional menunjukkan sistem berjalan sesuai harapan. Hasil ini menegaskan bahwa fine-tuning memberikan peningkatan performa signifikan terhadap kemampuan dasar model ASR pre-trained untuk konteks Bahasa Indonesia, serta layak diterapkan sebagai solusi aksesibilitas real-world bagi pengguna tunarungu.
Downloads
Article Details
Section
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
[1] WHO, “WORLD REPORT O N HEARING,” Mar. 2021. Accessed: Apr. 10, 2026. [Online]. Available: https://www.who.int/publications/i/item/9789240020481
[2] Kemenko PMK, “Pemerintah Penuhi Hak Penyandang Disabilitas di Indonesia | Kementerian Koordinator Bidang Pembangunan Manusia dan Kebudayaan,” https://www.kemenkopmk.go.id/pemerintah-penuhi-hak-penyandang-disabilitas-di-indonesia. Accessed: Apr. 10, 2026. [Online]. Available: https://www.kemenkopmk.go.id/pemerintah-penuhi-hak-penyandang-disabilitas-di-indonesia
[3] N. Fathallah, M. Bhole, and S. Staab, “Empowering the Deaf and Hard of Hearing Community: Improving Video Captions with Large Language Models,” in ACM International Conference Proceeding Series, Association for Computing Machinery, Jul. 2025, pp. 345–353. doi: 10.1145/3696593.3696617.
[4] G. Kalaiselvi, N. Badri, and C. Karnan, “SignSpeak – Sign language translation system for hearing impaired,” International Journal of Science and Research Archive, vol. 15, no. 2, pp. 921–930, May 2025, doi: 10.30574/ijsra.2025.15.2.1523.
[5] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” Dec. 2023, [Online]. Available: http://arxiv.org/abs/2212.04356
[6] E. William and A. Zahra, “SPEECH RECOGNITION DENGAN WHISPER DALAM BAHASA INDONESIA,” Action Research Literate, vol. 9, no. 2, 2025.
[7] J. Jafar, M. A. W. Tb, F. Aziz, R. Iriany, and N. Nasir, “Fine-Tuning Whisper Model for Mandar Speech Recognition: Approach and Performance Evaluation,” Journal of Applied Engineering and Technological Science (JAETS), vol. 7, no. 1, pp. 261–273, Dec. 2025, doi: 10.37385/jaets.v7i1.7170.
[8] R. S. A. Pratama and A. Amrullah, “ANALYSIS OF WHISPER AUTOMATIC SPEECH RECOGNITION PERFORMANCE ON LOW RESOURCE LANGUAGE,” Jurnal Pilar Nusa Mandiri, vol. 20, no. 1, pp. 1–8, Mar. 2024, doi: 10.33480/pilar.v20i1.4633.
[9] N. Lubis, Mhd. Z. Siambaton, and R. Aulia, “Implementasi Algoritma Deep Learning pada Aplikasi Speech to Text Online dengan Metode Recurrent Neural Network (RNN),” sudo Jurnal Teknik Informatika, vol. 3, no. 3, pp. 113–126, Sep. 2024, doi: 10.56211/sudo.v3i3.583.
[10] D. Rekesh et al., “Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition,” Sep. 2023, [Online]. Available: http://arxiv.org/abs/2305.05084
[11] A. Gulati et al., “Conformer: Convolution-augmented Transformer for Speech Recognition,” May 2020, [Online]. Available: http://arxiv.org/abs/2005.08100
[12] N. Giovanni and W. Nurhayat, “The Effectiveness Of Automatic Speech Recognition In The Capcut Application For Developing Inclusive Learning Media,” JKTP: Jurnal Kajian Teknologi Pendidikan, vol. 8, no. 2, pp. 157–169, May 2025, doi: 10.17977/um038v8i22025p157.
[13] A. Barcovschi, R. Jain, and P. Corcoran, “A comparative analysis between Conformer-Transducer, Whisper, and wav2vec2 for improving the child speech recognition,” Nov. 2023, [Online]. Available: http://arxiv.org/abs/2311.04936
[14] A. A. Attia, D. Demszky, T. Ogunremi, J. Liu, and C. Espy-Wilson, “Continued Pretraining for Domain Adaptation of Wav2vec2.0 in Automatic Speech Recognition for Elementary Math Classroom Settings,” May 2024, [Online]. Available: http://arxiv.org/abs/2405.13018
[15] I. Degtyarenko, O. Radyvonenko, N. Tkach, V. Sielikhov, K. Seliuk, and O. Ivanov, “Real-Time Multilingual Video Subtitle Spotting on Resource-Constrained Devices,” IEEE Access, vol. 13, pp. 211701–211714, 2025, doi: 10.1109/ACCESS.2025.3643512.
[16] P. Romero-Fresco and Y. Van Gauwbergen, “Fit for What Purpose? NER Certification of Automatic Captions in English and Spanish,” Applied Sciences (Switzerland), vol. 15, no. 3, Feb. 2025, doi: 10.3390/app15031387.
[17] S. Di Leo, L. De Cicco, and S. Mascolo, “Real-Time Speech-to-Text on Edge: A Prototype System for Ultra-Low Latency Communication with AI-Powered NLP,” Information (Switzerland), vol. 16, no. 8, Aug. 2025, doi: 10.3390/info16080685.
[18] R. Kannojia, A. K. Singh, I. Sharma, and S. Gupta, “Gen AI driven multilingual audio dubbing and synthesis system for cross-language video platforms,” Results in Engineering, vol. 27, Sep. 2025, doi: 10.1016/j.rineng.2025.106241.
[19] Mozilla Foundation, “Browser extensions,” https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions. Accessed: May 01, 2026. [Online]. Available: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions
[20] R. Anrahvi, N. Pratama, and S. Stevani, “Penerapan Metode System Usability Scale (SUS) dalam Mengukur Kepuasan Mahasiswa terhadap Website Direktori Akademik,” Indonesian Journal of Business Economics and Management, vol. 3, no. 2, pp. 74–80, Jun. 2024, doi: 10.57152/ijbem.v3i2.2020.
[21] I. J. Ratul, Y. Zhou, and K. Yang, “Accelerating Deep Learning Inference: A Comparative Analysis of Modern Acceleration Frameworks,” Electronics (Switzerland), vol. 14, no. 15, Aug. 2025, doi: 10.3390/electronics14152977.
[22] S. Papi, M. Gaido, A. Karakanta, M. Cettolo, M. Negri, and M. Turchi, “Direct Speech Translation for Automatic Subtitling,” Jul. 2023, [Online]. Available: http://arxiv.org/abs/2209.13192