International Audio Laboratories Erlangen

Score-Informed Audio Parameterization of Music Signals (SIAMUS)

Logo_DFG Teaser_SIAMUS Logo_FAU

In the SIAMUS project, we developed robust and practical methods for the automated parameterization and decompositon of music signals. The project was funded by the German Research Foundation. On this website, we summarize the project's main objectives and provide links to project-related resources (data, demonstrators, websites) and publications.

Project Description

Teaser_SIAMUS

Score-Informed Audio Parameterization of Music Signals

In this project, we developed robust and practical methods for the automated parameterization of music signals. In particular, the goal was to identify and reconstruct signal components that correspond to individual notes events or entire melodic and instrumental voices—a task being closely related to what is commonly known as source separation in general audio signal processing. In the case of music, a key challenge is to allow complex superpositions of closely related musical sources including singing/instrumental voices and accompaniments in music such as piano songs or operas. Without additional knowledge, a decomposition of a one- or two-channel audio recording into such voices is hardly possible. Therefore, in this project, we followed an informed approach, where additional score information is exploited in the audio parameterization process. On the basis of automatically computed score–audio alignments, the score information was used to specify and guide the parameterization process as well as to support the signal analysis and reconstruction steps. In addition to fundamental research on signal modeling and parameter optimization, this project was also concerned with the development of novel applications demonstrating the practical relevance and ensuring the sustainability of the project.

Projektbeschreibung

Notentext-Informierte Audioparametrisierung von Musiksignalen

Teaser_SIAMUS

Dieses Projekt widmete sich der Erforschung und Umsetzung von robusten und praktikablen Verfahren zur automatisierten Parametrisierung von Musiksignalen. Hierbei ging es insbesondere um die Erfassung und Rekonstruktion von Signalkomponenten, die einzelnen Notenereignissen oder ganzen Melodie- und Instrumentalstimmen entsprechen – eine Aufgabenstellung, die im übergeordneten Bereich der Audiosignalverarbeitung einen engen Bezug zum Quellentrennungsproblem hat. Im Fall von Musik besteht eine wesentliche Herausforderung darin, komplexe Überlagerungen von in enger Beziehung stehenden musikalischen Stimmen zu erlauben (z. B. Gesangsstimmen, Instrumentalstimmen und Begleitstimmen in Klavierliedern oder Opern). Ohne Zusatzwissen ist eine Zerlegung einer ein- oder zweikanaligen Audioaufnahme in solche Stimmen kaum lösbar. Daher wurde in diesem Projekt ein informierter Ansatz verfolgt, bei dem zusätzlich ein Notentext für die Audioparametrisierung herangezogen wurde. Auf Basis von automatisiert berechneten Verküpfungen zwischen Notentext und Signal kam dabei die Notentextinformation zur Spezifikation und Steuerung des Parametrisierungsprozesses sowie zur Unterstützung der Signalanalyse und -rekonstruktion zum Einsatz. Neben grundsätzlichen Fragestellungen der Signalmodellierung und Parameteroptimierung, wurden auch neuartige Anwendungen zur Audioeditierung realisiert, um somit den Praxisbezug und die Nachhaltigkeit des Projekts sicherzustellen.

Projected-Related Resources and Demonstrators

The following list provides an overview of the most important publicly accessible sources created in the SIAMUS project:

Projected-Related Publications

The following publications reflect the main scientific contributions of the work carried out in the SIAMUS project.

  1. Stefan Balke, Christian Dittmar, Jakob Abeßer, and Meinard Müller
    Data-Driven Solo Voice Enhancement for Jazz Music Retrieval
    In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 196–200, 2017. PDF Demo Presentation 
    @inproceedings{BalkeDAM17_SoloVoiceEnhancement_ICASSP,
author = {Stefan Balke and Christian Dittmar and Jakob Abe{\ss}er and Meinard M{\"u}ller},
title = {Data-Driven Solo Voice Enhancement for Jazz Music Retrieval},
booktitle = {Proceedings of the {IEEE} International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})},
pages = {196--200},
location = {New Orleans, USA},
year = {2017},
url-demo={https://www.audiolabs-erlangen.de/resources/MIR/2017-ICASSP-SoloVoiceEnhancement},
url-pdf = {2017_BalkeDAM_SoloVoiceEnhancement_ICASSP.pdf},
url-presentation = {2017_BalkeDAM_SoloVoiceEnhancement_ICASSP_presentation.pdf}
}
  2. Christian Dittmar, Jonathan Driedger, and Meinard Müller
    A Separate and Restore Approach to Score-Informed Music Decomposition
    In Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2015. PDF Demo 
    @inproceedings{DittmarDM15_MusicDecomposition_WASPAA,
author    = {Christian Dittmar and Jonathan Driedger and Meinard M{\"u}ller},
title     = {A Separate and Restore Approach to Score-Informed Music Decomposition},
booktitle = {Proceedings of the {IEEE} Workshop on Applications of Signal Processing to Audio and Acoustics ({WASPAA})},
address   = {New Paltz, New York, USA},
year      = {2015},
pages     = {},
url-pdf   = {2015_DittmarDM_SeparateRestore_WASPAA.pdf},
url-demo  = {https://www.audiolabs-erlangen.de/resources/MIR/2015-WASPAA-SeparateAndRestore}
}
  3. Jonathan Driedger, Stefan Balke, Sebastian Ewert, and Meinard Müller
    Template-Based Vibrato Analysis of Music Signals
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 239–245, 2016. PDF Demo 
    @inproceedings{DriedgerBEM16_Vibrato_ISMIR,
author    = {Jonathan Driedger and Stefan Balke and Sebastian Ewert and Meinard M{\"u}ller},
title     = {Template-Based Vibrato Analysis of Music Signals},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference  ({ISMIR})},
address   = {New York, USA},
year      = {2016},
pages     = {239--245},
url-pdf   = {2016_DriedgerBEM_VibratoDetection_ISMIR_ePrint.pdf},
url-demo = {https://www.audiolabs-erlangen.de/resources/MIR/2016-ISMIR-Vibrato}
}
  4. Jonathan Driedger and Meinard Müller
    A Review on Time-Scale Modification of Music Signals
    Applied Sciences, 6(2): 57–82, 2016. PDF Demo 
    @article{DriedgerMueller16_ReviewTSM_AppliedSciences,
author  = {Jonathan Driedger and Meinard M{\"u}ller},
journal = {Applied Sciences},
title   = {A Review on Time-Scale Modification of Music Signals},
year    = {2016},
month   = {February},
volume  = {6},
number  = {2},
pages   = {57--82},
url-pdf   = {2016_DriedgerMueller_TSMOverview_AppliedSciences_ePrint.pdf},
url-demo = {https://www.audiolabs-erlangen.de/resources/MIR/TSMtoolbox}
}
  5. Jonathan Driedger and Meinard Müller
    Extracting Singing Voice from Music Recordings by Cascading Audio Decomposition Techniques
    In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 126–130, 2015. PDF Demo 
    @inproceedings{DriedgerM15_SingingVoice_ICASSP,
author    = {Jonathan Driedger and Meinard M{\"u}ller},
title     = {Extracting Singing Voice from Music Recordings by Cascading Audio Decomposition Techniques},
booktitle  = {Proceedings of the {IEEE} International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})},
address  = {Brisbane, Australia},
year       = {2015},
pages    = {126--130},
url-pdf    = {2015_DriedgerMueller_SVECascadedDecomposition_ICASSP.pdf},
url-demo   = {https://www.audiolabs-erlangen.de/resources/MIR/2015-ICASSP-SVECD/}
}
  6. Jonathan Driedger and Meinard Müller
    Verfahren zur Schätzung der Grundfrequenzverläufe von Melodiestimmen in mehrstimmigen Musikaufnahmen
    In Wolfgang Auhagen and Claudia Bullerjahn and Richard von Georgi (ed.): Musikpsychologie — Anwendungsorientierte Forschung, Hogrefe-Verlag, 25: 55–71, 2015. PDF 
    @incollection{DriedgerM15_F0Estimation_DGM,
author    = {Jonathan Driedger and Meinard M{\"u}ller},
title   = {{V}erfahren zur {S}ch{\"a}tzung der {G}rundfrequenzverl{\"a}ufe von {M}elodiestimmen in mehrstimmigen {M}usikaufnahmen},
booktitle     = {Musikpsychologie -- Anwendungsorientierte Forschung},
editor    = {Wolfgang Auhagen and Claudia Bullerjahn and Richard von Georgi},
year      = {2015},
volume    = {25},
pages     = {55--71},
publisher = {Hogrefe-Verlag},
series    = {Jahrbuch Musikpsychologie},
url-pdf   = {2015_DriedgerMueller_F0Estimation_DGM.pdf}
}
  7. Jonathan Driedger, Meinard Müller, and Sebastian Ewert
    Improving Time-Scale Modification of Music Signals using Harmonic-Percussive Separation
    IEEE Signal Processing Letters, 21(1): 105–109, 2014. PDF Demo 
    @article{Driedger14_SourceSeparation_IEEE-LettersSP,
author    = {Jonathan Driedger and Meinard M{\"u}ller and Sebastian Ewert},
title     = {Improving Time-Scale Modification of Music Signals using Harmonic-Percussive Separation},
journal   = {{IEEE} Signal Processing Letters},
volume    = {21},
number    = {1},
year      = {2014},
pages     = {105--109},
url-demo = {https://www.audiolabs-erlangen.de/resources/2014-SPL-HPTSM/},
url-pdf   = {https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6678724}
}
  8. Jonathan Driedger, Meinard Müller, and Sascha Disch
    Extending Harmonic-Percussive Separation of Audio Signals
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 611–616, 2014. PDF Demo 
    @inproceedings{DriedgerMD14_SeparationHP_ISMIR,
author    = {Jonathan Driedger and Meinard M{\"u}ller and Sascha Disch},
title     = {Extending Harmonic-Percussive Separation of Audio Signals},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference  ({ISMIR})},
address   = {Taipei, Taiwan},
year      = {2014},
pages     = {611--616},
url-pdf   = {2014_DriedgerMuellerDisch_ExtensionsHPSeparation_ISMIR.pdf},
url-demo = {https://www.audiolabs-erlangen.de/resources/2014-ISMIR-ExtHPSep/}
}
  9. Jonathan Driedger, Thomas Prätzlich, and Meinard Müller
    Let It Bee — Towards NMF-Inspired Audio Mosaicing
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 350–356, 2015. PDF Demo 
    @inproceedings{DriedgerPM15_AudioMosaicingNMF_ISMIR,
author    = {Jonathan Driedger and Thomas Pr{\"a}tzlich and Meinard M{\"u}ller},
title     = {{L}et {I}t {B}ee -- Towards {NMF}-Inspired Audio Mosaicing},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference  ({ISMIR})},
address   = {M{\'a}laga, Spain},
year      = {2015},
pages     = {350--356},
url-demo  = {https://www.audiolabs-erlangen.de/resources/MIR/2015-ISMIR-LetItBee},
url-pdf   = {2015_DriedgerPM_AudioMosaicingNMF_ISMIR.pdf}
}

Projected-Related Ph.D. Thesis

Jonathan Driedger won the Promotionspreis 2017 of the Staedtler Stiftung for his outstanding dissertation.

  1. Jonathan Driedger
    Processing Music Signals Using Audio Decomposition Techniques
    PhD Thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg, 2016. PDF 
    @phdthesis{Driedger16_AudioDecompostion_PhD,
author      = {Jonathan Driedger},
year        = {2016},
title       = {Processing Music Signals Using Audio Decomposition Techniques},
school      = {Friedrich-Alexander-Universit{\"a}t Erlangen-N{\"u}rnberg},
url-pdf = {2016_Driedger_AudioDecomposition_PhD-Thesis.pdf}
}