Reducing latency of neural automatic piano transcription models

Dasol Lee; Dasaem Jeong

doi:10.7776/ASK.2023.42.2.102

All Issue

2023 Vol.42, Issue 2 Preview Page Next Page

Research Article

Reducing latency of neural automatic piano transcription models 인공신경망 기반 저지연 피아노 채보 모델

31 March 2023. pp. 102-111

PDF XML

Abstract

Automatic Music Transcription (AMT) is a task that detects and recognizes musical note events from a given audio recording. In this paper, we focus on reducing the latency of real-time AMT systems on piano music. Although neural AMT models have been adapted for real-time piano transcription, they suffer from high latency, which hinders their usefulness in interactive scenarios. To tackle this issue, we explore several techniques for reducing the intrinsic latency of a neural network for piano transcription, including reducing window and hop sizes of Fast Fourier Transformation (FFT), modifying convolutional layer’s kernel size, and shifting the label in the time-axis to train the model to predict onset earlier. Our experiments demonstrate that combining these approaches can lower latency while maintaining high transcription accuracy. Specifically, our modified model achieved note F1 scores of 92.67 % and 90.51 % with latencies of 96 ms and 64 ms, respectively, compared to the baseline model’s note F1 score of 93.43 % with a latency of 160 ms. This methodology has potential for training AMT models for various interactive scenarios, including providing real-time feedback for piano education.

Keywords

Automatic music transcription

Piano transcription

Low-latency

Convolutional neural network

자동 음악 채보는 주어진 오디오에서 음표 정보를 추출하는 태스크로, 이 연구에서는 피아노 음악의 자동 음악 채보 모델에서 지연 시간을 줄이는 방법을 소개한다. 신경망 기반 채보 모델이 피아노 채보에도 적용되어 높은 정확도를 기록하였고 이를 이용한 실시간 구현도 소개된 바 있지만, 채보를 위한 지연 시간이 길어 인터랙티브 시나리오에서 활용하기에 한계가 있었다. 이 문제를 해결하기 위해 본 연구는 Fast Fourier Transformation(FFT)에서 윈도우 크기와 홉 크기를 줄이거나 합성곱 레이어의 커널 크기를 수정하고 시간 축에서 레이블을 이동하여 모델이 시작을 더 일찍 예측하도록 훈련하는 등 피아노 전사를 위한 신경망의 내재적 지연 시간을 줄이는 몇 가지 기술을 제안한다. 실험 결과, 이러한 접근 방식을 결합하면 높은 전사 정확도를 유지하면서 지연 시간을 줄일 수 있음을 알 수 있었다. 기존 모델은 160 ms의 지연 시간을 가지고 음표 F1 점수는 93.43 %였으나 제안한 방법을 적용하면 96 ms와 64 ms의 지연 시간 동안 각각 92.67 %와 90.51 %의 F1 점수를 달성할 수 있었다. 이러한 결과는 향후 피아노 교육을 위한 실시간 피드백 제공 등 다양한 인터랙티브 시나리오를 위한 자동 채보 모델에 활용될 수 있을 것이다.

키워드

자동 채보

피아노 채보

저지연

합성곱 신경망

References

K. O'Hanlon and M. D. Plumbley, "Polyphonic piano transcription using non-negative matrix factorisation with group sparsity," Proc. IEEE ICASSP, 3112-3116 (2014). 10.1109/ICASSP.2014.6854173

C. Raphael, "Automatic transcription of piano music," Proc. 3^rd ISMIR, (2002).

V. Emiya, R. Badeau, and B. David "Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle," IEEE Trans. on Audio, Speech, and Lang. Process. 18, 1643-1654 (2010). 10.1109/TASL.2009.2038819

L. Su and Y-H. Yang, "Combining spectral and temporal representations for multipitch estimation of polyphonic music," IEEE/ACM Trans. on Audio, Speech, and Lang. Process. 23, 1600-1612 (2015). 10.1109/TASLP.2015.2442411

S. Sigtia, E. Benetos, and S. Dixon, "An end-to-end neural network for polyphonic piano music transcription," IEEE/ACM Trans. on Audio, Speech, and Lang. Process. 24, 927-939 (2016). 10.1109/TASLP.2016.2533858

R. Kelz, M. Dorfer, F. Korzeniowski, S. Böck, A. Arzt, and G. Widmer, "On the potential of simple framewise approaches to piano transcription," Proc. 17^th ISMIR, 475-481 (2016).

C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck, "Onsets and frames: Dual-objective piano transcription," Proc. 19^th ISMIR, 50-57 (2018).

C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C-Z A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, "Enabling factorized piano music modeling and generation with the MAESTRO dataset," Proc. 7^th ICLR, 1-12 (2019).

Q. Kong, K. Choi, and Y. Wang, "Large-scale MIDI- based composer classification," arXiv preprint arXiv: 2010.14805 (2020).

H. Zhang, J. Tang, S. R. Rafee, S. Dixon, G. Fazekas, and G. A. Wiggins, "ATEPP: A dataset of automatically transcribed expressive piano performance," Proc. 23^st ISMIR, 446-453 (2022).

Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, "High resolution piano transcription with pedals by regressing onset and offset times," IEEE/ACM Trans. on Audio, Speech, and Lang. Process. 29, 3707-3717 (2021). 10.1109/TASLP.2021.3121991

T. Kwon, D. Jeong, and J. Nam, "Polyphonic piano transcription using autoregressive multi-state note model," Proc. the 21^st ISMIR, 454-461 (2020).

D. Jeong, "Real-time automatic piano music transcription system," Proc. Late Breaking/Demo of the 21^st ISMIR, 4-6 (2020).

A. A. Sawchuk, E. Chew, R. Zimmermann, C. Papadopoulos, and C. Kyriakakis, "From remote media immersion to distributed immersive performance," Proc. ACM SIGMM workshop on Experiential Telepresence, 110-120 (2003). 10.1145/982484.982506

J. W. Kim and J. P. Bello, "Adversarial learning for improved onsets and frames music transcription," Proc. 20^th ISMIR, 670-677 (2019).

M. Akbari and H. Cheng, "Real-time piano music transcription based on computer vision," IEEE Trans. Multimedia, 17, 2113-2121 (2015). 10.1109/TMM.2015.2473702

A. Dessein, A. Cont, and G. Lemaitre, "Real-time polyphonic music transcription with non-negative matrix factorization and beta-divergence," Proc. 11^th ISMIR, 489-494 (2010).

C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. Ellis, "Mir_eval: A transparent implementation of common mir metrics," Proc. 15^th ISMIR, 367-372 (2014).

Information

Publisher :The Acoustical Society of Korea
Publisher(Ko) :한국음향학회
Journal Title :The Journal of the Acoustical Society of Korea
Journal Title(Ko) :한국음향학회지
Volume : 42
No :2
Pages :102-111
Received Date : 2023-03-03
Revised Date : 2023-03-22
Accepted Date : 2023-03-26
DOI :https://doi.org/10.7776/ASK.2023.42.2.102

The Journal of the Acoustical Society of KoreaISSN:1225-4428(Print) 2287-3775(Online)한국음향학회

All Issue