Integrated modeling of zero shot audio effects based on dual conditioning

Sanghoo Shin; Jihyun Park; Yoseop Hwang; Kyu-Chil Park

doi:10.7776/ASK.2025.44.6.640

All Issue

2025 Vol.44, Issue 6 Preview Page Next Page

Research Article

Integrated modeling of zero shot audio effects based on dual conditioning 듀얼 컨디셔닝에 기반한 제로샷 오디오 이펙트 통합 모델링

30 November 2025. pp. 640-646

PDF XML

Abstract

In this paper, we propose a dual-conditioning approach that simultaneously utilizes label embedding and tone embedding to model other amplifiers and audio effects in an integrated manner. The proposed method is designed to generalize the tone characteristics of the device, which are not learned through the structure of the device, to zero-shot. As the generator structure, we adopted Field Programable Gate Array (FPGA)-based Temporal Convolutional Network (TCN) and WaveNet to compare the lightweight model that requires real-time performance with the large-scale model that aims for high-quality synthesis, and the dual-conditioning model showed consistent improvement over the conventional label embedding single condition. In FiLM-TCN, MAE and RMSE were reduced by 41.5 % and 52.0 %, respectively, and in WaveNet, they were reduced by 54.7 % and 61.9 %, respectively. In addition, MS-STFT errors were also reduced by more than 10 % in both structures. As a result, we confirmed that the dual-conditioning approach is more effective than the single-label embedding condition.

Keywords

Audio effect modeling

Dual conditioning condition injection

Temporal Convolutional Network (TCN)

Deep learning

본 연구에서는 기타앰프 및 오디오 이펙트를 통합적으로 모델링하기 위해 라벨 임베딩과 톤 임베딩을 동시에 활용하는 듀얼 컨디셔닝 접근법을 제안한다. 제안 방식은 디바이스의 구조를 통해 학습되지 않은 장치의 음색 특성을 제로샷으로 일반화 할 수 있도록 설계하였다. 생성기 구조로는 Field Programable Gate Array(FPGA) 기반 시계열 합성곱 네트워크(Temporal Convolutional Network, TCN)과 WaveNet을 채택하여 실시간성이 요구되는 경량 모델과 고품질 합성을 지향하는 대규모 모델을 비교한 실험 결과, 듀얼 컨디셔닝 모델은 기존 라벨 임베딩 단일 조건대비 일관된 개선을 보였다. FiLM-TCN에서 MAE와 RMSE가 각각 41.5 %, 52.0 % 감소했으며, WaveNet에서 각각 54.7 %, 61.9 % 감소하였다. 또한 MS-STFT 오차도 두 구조에서 10 % 이상 감소하였다. 결과적으로 듀얼 컨디셔닝 접근법이 단일 라벨 임베딩 조건보다 효과적임을 확인하였다.

키워드

오디오 이펙트 모델링

듀얼 컨디셔닝 조건 주입

시계열 합성곱 네트워크

딥러닝

References

J. Pakarinen and D. T. Yeh, “A review of digital techniques for modeling vacuum-tube guitar amplifiers,” Comput. Music J. 33, 85-100 (2009).

10.1162/comj.2009.33.2.85

M. A. M. Ramírez and J. D. Reiss, “Modeling nonlinear audio effects with end-to-end deep neural networks,” Proc. IEEE ICASSP, 331-335 (2019).

A. Wright, E.-P. Damskägg, L. Juvela, and V. Välimäki, “Real-time guitar amplifier emulation with deep learning,” Appl. Sci. 10, 766 (2020).

10.3390/app10030766

J. Engel, K. K. Agrawal, S. Chen, I. Gulrajani, C. Donahue, and A. Roberts, “GANSynth: Adversarial neural audio synthesis,” arXiv:1902.08710 (2019).

Y. H. Chen, Y. T. Yeh, Y. C. Cheng, J. T. Wu, Y. H. Ho, J. S. R. Jang, and Y. H. Yang, “Towards zero-shot amplifier modeling: One-to-many amplifier modeling via tone embedding control,” arXiv:2407.10646 (2024).

A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499 (2016).

R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi- resolution spectrogram,” Proc. IEEE ICASSP, 6199-6203 (2020).

10.1109/ICASSP40776.2020.9053795

Y.-H. Chen, W.-Y. Hsiao, T.-K. Hsieh, J.-S. R. Jang, and Y.-H. Yang, “Towards automatic transcription of polyphonic electric guitar music: A new dataset and a multi-loss transformer model,” Proc. IEEE ICASSP, 786-790 (2022).

10.1109/ICASSP43922.2022.9747697

H. Pedroza, G. Meza, and I. R. Roman, “Egfxset: Electric guitar tones processed through real effects of distortion, modulation, delay and reverb,” Proc. ISMIR Late Breaking Demo, 1-3 (2022).

F. Roche, T. Hueber, S. Limier, and L. Girin, “Autoencoders for music sound modeling: A comparison of linear, shallow, deep, recurrent and variational models,” arXiv:1806.04096 (2018).

Information

Publisher :The Acoustical Society of Korea
Publisher(Ko) :한국음향학회
Journal Title :The Journal of the Acoustical Society of Korea
Journal Title(Ko) :한국음향학회지
Volume : 44
No :6
Pages :640-646
Received Date : 2025-09-04
Revised Date : 2025-10-10
Accepted Date : 2025-10-13
DOI :https://doi.org/10.7776/ASK.2025.44.6.640

The Journal of the Acoustical Society of KoreaISSN:1225-4428(Print) 2287-3775(Online)한국음향학회

All Issue