International Audio Laboratories Erlangen

Automated Methods and Tools for Analyzing and Structuring Choral Music (AnChor)

Logo_DFG Teaser_AnChor Logo_FAU

In the AnChor project, we investigated in collaboration with the Carus publishing house efficient search, navigation, and analsis methods with a specific focus on singing and choir recordings. This transfer project was funded by the German Research Foundation. On this website, we summarize the project's main objectives and provide links to project-related resources (data, demonstrators, websites) and publications.

Project Description

Automated Methods and Tools for Analyzing and Structuring Choral Music

Teaser_AnChor_interface

The increasing digitization results in extensive music collections, which, in addition to audio and video recordings, also contain symbolically or graphically encoded sheet music. One such example is the multimedia music catalog offered by the Carus publishing house, a leading international music publisher for religious and secular choral music. In addition to its extensive catalog of carefully edited sheet music comprising more than 30,000 choir pieces, Carus also produces reference recordings and video material for teaching and practice purposes. Generally speaking, the main goal of Music Information Retrieval (MIR) is to develop efficient search and navigation systems that allow users to analyze and search through complex music collections concerning different music-related aspects. In this DFG-funded transfer project, we adapted and improved MIR methods while testing them in practice.

  • As a first central contribution, we developed web-based user interfaces for interactive and synchronous access to different music representations and analysis results. Not only are these interfaces of interest for possible commercial applications, but they also serve to communicate research results across different disciplines.

  • One of the project's important research activities was to develop and implement a content-based retrieval system, where a user can make a search request in the form of a short audio excerpt or a YouTube link (e.g., a choir recording sung by an amateur choir). The goal is then to automatically identify, based purely on the acoustic waveform, all other performances or versions of the piece of music in a given collection (e.g., a high-quality recording by a professional choir in the Carus catalog). For this task, we could achieve considerable improvements in terms of the retrieval system's runtime and storage requirements by combining embedding techniques based on neural networks with graph-based indexing techniques.

  • Another research focus was on automated methods for measuring, analyzing, and adjusting intonation fluctuations in unaccompanied, polyphonic vocal music.

  • Finally, based on the experience gained with our project partner, we developed music datasets along with annotations, which we made freely available for research purposes. These datasets are suitable for developing and evaluating algorithmic approaches for various MIR tasks, including music transcription, music synchronization, chord recognition, fundamental frequency estimation, and intonation analysis.

  • Through the collaboration with the Carus publishing house, it was also possible to establish cross-connections to music education and musicology and initiate interdisciplinary collaborations.

Projektbeschreibung

Automatisierte Methoden und Werkzeuge zur Analyse und Strukturierung von Chormusik

Teaser_AnChor_interface

Durch die zunehmende Digitalisierung entstehen große Musikkollektionen, die neben Audio- und Videoaufnahmen auch symbolisch oder grafisch kodierte Notentexte enthalten. Ein Beispiel hierfür ist das multimediale Musikangebot des Carus-Verlag, einem im Bereich der geistlichen und weltlichen Chormusik international führenden Musikverlag, der neben seinem Angebot von sorgfältig edierten Notenausgaben für mehr als 30,000 Chorwerke auch Referenzeinspielungen produziert und Videos für Unterrichts- und Übezwecke entwickelt. Allgemein gesprochen ist das Hauptziel des Music Information Retrieval (MIR) die Entwicklung effzienter Such- und Navigationssysteme, die es dem Benutzer erlauben, komplexe Musikdatenbestände bezüglich unterschiedlicher musikrelevanter Aspekte zu durchsuchen. Bei diesem von der DFG geförderten Transferprojekt wurden unterschiedliche MIR-Verfahren erforscht und in der Praxis erprobt.

  • Als ein erster zentraler Beitrag wurden webbasierte Benutzerschnittstellen entwickelt, die einen interaktiven und synchronen Zugriff auf unterschiedliche Musikdarstellungen und Analyseergebnisse erlauben. Diese Schnittstellen sind nicht nur für mögliche kommerzielle Weiterentwicklungen von Interesse, sondern dienen auch der Kommunikation von Forschungsergebnissen über unterschiedliche Disziplinen hinweg.

  • Ein wichtiger Forschungsschwerpunkt bestand in der Entwicklung inhaltsbasierter Suchverfahren, bei denen ein Benutzer eine Anfrage in Form eines kurzen Audioausschnitts oder eines YouTube-Links (z. B. eine Choraufnahme gesungen von einem Laienchor) stellt. Die Aufgabe besteht dann darin, automatisch, rein auf Basis der akustischen Wellenform, alle anderen Interpretationen oder Versionen des Musikstücks in einer vorgegebenen Kollektion zu identifizieren (z. B. eine hochqualitative Aufnahme eines professionellen Chors im Carus-Katalog). Durch den Einsatz von auf neuronalen Netzwerken basierender Einbettungstechniken in Kombination mit graphenbasierten Indexierungstechniken konnten im $\PN$-Projekt erhebliche Verbesserungen hinsichtlich der benötigten Laufzeit und des Speicherplatzbedarfs solcher Suchverfahren erzielt werden.

  • In einem weiteren Forschungsschwerpunkt wurden automatisierte Methoden zur Messung, Analyse und Anpassung von Intonationsschwankungen in unbegleiteter, mehrstimmiger Vokalmusik erforscht.

  • Schließlich wurden, auf Basis der mit dem Projektpartner gesammelten Erfahrungen, Musikdatensätze entwickelt, annotiert und für Forschungszwecke frei verfügbar gemacht, die für die Erforschung und Auswertung algorithmischer Ansätze für ganz unterschiedliche MIR Aufgaben (z. B. Musiktranskription, Musiksynchronisation, Akkorderkennung, Fundamentalfrequenzschätzung und Intonationsanalyse) von großem wissenschaftlichen Wert sind.

  • Durch die Zusammenarbeit mit dem Carus-Verlag konnten darüber hinaus Querverbindungen zur Musikpädagogik und zu den Musikwissenschaften hergestellt und weitere interdisziplinäre Kooperationen initiiert beziehungsweise unterstützt werden.

Projected-Related Resources and Demonstrators

The following list provides an overview of the most important publicly accessible sources created in the AnChor project:

Projected-Related Master Projects and Theses

Besides the actual project work, the qualifications of students was another central goal of the Anchor project. The following list provides an overview of student work (research internships and theses) that were directly related to the AnChor project.

  • Maria Alejandra Acuna Rojas: Informed Music Processing Techniques for Singing Voice Analysis. Master Thesis, FAU, 2019.
  • Leo Brütting: Hierarchical Tonal Analysis of Music Signals. Bachelor Thesis, FAU, 2019.
  • Judith Bauer: Deep-Learning Approaches for Fundamental Frequency Estimation of Music Recordings. Master Thesis, FAU, 2019.
  • Angel Villar-Corrales: Deep Learning Techniques for Cross-Version Retrieval of Music Recordings. Major Research Project, FAU, 2019.
  • Julian Brandner: Efficient Cross-Version Music Retrieval Using Dimensionality Reduction and Indexing Techniques. Master Thesis, FAU, 2019.
  • Lu Cheng: Version Identification of Music Recordings for Choir Music. Research Internship, FAU, 2020.
  • Lukas Dietz: Interactive Interface for Choir Singer Performance Analysis. Research Internship, TH Nürnberg, 2020.
  • Florian Schuberth: Methods for Global Tuning Estimation. Research Internship, FAU, 2020.
  • Simon Schwär: Analysis of Intonation in Unaccompanied Vocal Music. Major Research Project, FAU, 2020.
  • Benjamin Brunner: Evaluating Tuning Estimation Algorithms for Music Processing Applications. Master Thesis, FAU, 2021.

Projected-Related Publications

The following publications reflect the main scientific contributions of the work carried out in the AnChor project.

  1. Stefan Balke, Christian Dittmar, Jakob Abeßer, Klaus Frieler, Martin Pfleiderer, and Meinard Müller
    Bridging the Gap: Enriching YouTube Videos with Jazz Music Annotations
    Frontiers in Digital Humanities, 2018. PDF Details Demo DOI 
    @article{BalkeDAFPM18_JazzYoutube_Frontiers,
author = {Stefan Balke and Christian Dittmar and Jakob Abe{\ss}er and Klaus Frieler and Martin Pfleiderer and Meinard M{\"u}ller},
title = {Bridging the Gap: {E}nriching {Y}ou{T}ube Videos with Jazz Music Annotations},
journal = {Frontiers in Digital Humanities},
volume = {},
number = {},
pages = {},
doi = {doi.org/10.3389/fdigh.2018.00001},
year = {2018},
url-details={https://www.frontiersin.org/articles/10.3389/fdigh.2018.00001/full},
url-demo={http://mir.audiolabs.uni-erlangen.de/jazztube/},
url-pdf={2018_BalkeDAFPM_JazzYouTube_Frontiers.pdf},
}
  2. Sebastian Rosenzweig, Helena Cuesta, Christof Weiß, Frank Scherbaum, Emilia Gómez, and Meinard Müller
    Dagstuhl ChoirSet: A Multitrack Dataset for MIR Research on Choral Singing
    Transactions of the International Society for Music Information Retrieval (TISMIR), 3(1): 98–110, 2020. PDF Demo DOI 
    @article{RosenzweigCWSGM20_DCS_TISMIR,
author    = {Sebastian Rosenzweig and Helena Cuesta and Christof Wei{\ss} and Frank Scherbaum and Emilia G{\'o}mez and Meinard M{\"u}ller},
title     = {{D}agstuhl {ChoirSet}: {A} Multitrack Dataset for {MIR} Research on Choral Singing},
journal   = {Transactions of the International Society for Music Information Retrieval ({TISMIR})},
volume    = {3},
number    = {1},
year      = {2020},
pages     = {98--110},
publisher = {Ubiquity Press},
doi       = {10.5334/tismir.48},
url       = {http://doi.org/10.5334/tismir.48},
url-pdf   = {2020_RosenzweigCWSGM_DagstuhlChoirSet_TISMIR_ePrint.pdf},
url-demo  = {https://www.audiolabs-erlangen.de/resources/MIR/2020-DagstuhlChoirSet}
}
  3. Sebastian Rosenzweig, Lukas Dietz, Johannes Graulich, and Meinard Müller
    TuneIn: A Web-Based Interface for Practicing Choral Parts
    In Demos and Late Breaking News of the International Society for Music Information Retrieval Conference (ISMIR), 2020. PDF Demo 
    @inproceedings{RosenzweigDGM20_TuneIn_ISMIR-LBD,
author      = {Sebastian Rosenzweig and Lukas Dietz and Johannes Graulich and Meinard M{\"u}ller},
title       = {{TuneIn}: {A} Web-Based Interface for Practicing Choral Parts},
booktitle   = {Demos and Late Breaking News of the International Society for Music Information Retrieval Conference ({ISMIR})},
address     = {Montreal, Canada},
year        = {2020},
url-demo = {https://www.audiolabs-erlangen.de/resources/MIR/TuneIn},
url-pdf     = {2020_RosenzweigDGM_Carus_ISMIR-LBD.pdf}
}
  4. Sebastian Rosenzweig, Simon Schwär, Jonathan Driedger, and Meinard Müller
    Adaptive Pitch-Shifting with Applications to Intonation Adjustment in A Capella Recordings
    In Proceedings of the International Conference on Digital Audio Effects (DAFx), 2021. PDF Details Demo 
    @inproceedings{RosenzweigSDM_PitchShifting_DAFx,
author    = {Sebastian Rosenzweig and Simon Schw{\"a}r and Jonathan Driedger and Meinard M{\"u}ller},
title     = {Adaptive Pitch-Shifting with Applications to Intonation Adjustment in A Capella Recordings},
booktitle = {Proceedings of the International Conference on Digital Audio Effects ({DAFx})},
address   = {Vienna, Austria},
year      = {2021},
url-pdf   = {2021_RosenzweigSDM_ChoirPitchShift_DAFX.pdf},
url-details  = {https://www.audiolabs-erlangen.de/resources/MIR/2021-DAFX-AdaptivePitchShifting},
url-demo  = {https://github.com/meinardmueller/libtsm}
}
  5. Christof Weiß, Sebastian J. Schlecht, Sebastian Rosenzweig, and Meinard Müller
    Towards Measuring Intonation Quality of Choir Recordings: A Case Study on Bruckner's Locus Iste
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 276–283, 2019. PDF DOI 
    @inproceedings{WeissSRM19_ChoirIntonation_ISMIR,
author    = {Christof Wei{\ss} and Sebastian J. Schlecht and Sebastian Rosenzweig and Meinard M{\"u}ller},
title     = {Towards Measuring Intonation Quality of Choir Recordings: {A} Case Study on {B}ruckner's {L}ocus {I}ste},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
address   = {Delft, The Netherlands},
pages     = {276--283},
year      = {2019},
doi       = {10.5281/zenodo.3527798},
url-pdf   = {2019_WeissSRM_ChoirIntonation_ISMIR_PrintedVersion.pdf}
}
  6. Christof Weiß, Frank Zalkow, Vlora Arifi-Müller, Meinard Müller, Hendrik Vincent Koops, Anja Volk, and Harald Grohganz
    Schubert Winterreise Dataset: A Multimodal Scenario for Music Analysis
    ACM Journal on Computing and Cultural Heritage (JOCCH), 15(2): 1–18, 2021. Demo DOI 
    @article{WeissZAMKVG21_WinterreiseDataset_ACM-JOCCH,
author   = {Christof Wei{\ss} and Frank Zalkow and Vlora Arifi-M{\"u}ller and Meinard M{\"u}ller and Hendrik Vincent Koops and Anja Volk and Harald Grohganz},
title    = {{S}chubert {W}interreise Dataset: {A} Multimodal Scenario for Music Analysis},
journal  = {{ACM} Journal on Computing and Cultural Heritage ({JOCCH})},
volume   = {15},
number   = {2},
pages    = {25:1--18},
year     = {2021},
doi      = {10.1145/3429743},
url-demo = {https://doi.org/10.5281/zenodo.4122060}
}
  7. Frank Zalkow, Julian Brandner, and Meinard Müller
    Efficient Retrieval of Music Recordings Using Graph-Based Index Structures
    Signals, 2(2): 336–352, 2021. PDF DOI 
    @article{ZalkowBM21_IndexGraphs_Signals,
author    = {Frank Zalkow and Julian Brandner and Meinard M{\"u}ller},
title     = {Efficient Retrieval of Music Recordings Using Graph-Based Index Structures},
journal   = {Signals},
volume    = {2},
number    = {2},
pages     = {336--352},
year      = {2021},
doi       = {10.3390/signals2020021},
url-pdf   = {2021_ZalkowBM21_MusicRetrievalGraphs_Signals.pdf}
}
  8. Frank Zalkow and Meinard Müller
    Learning Low-Dimensional Embeddings of Audio Shingles for Cross-Version Retrieval of Classical Music
    Applied Sciences, 10(1), 2020. PDF DOI 
    @article{ZalkowMueller20_Shingles_AppliedSciences,
author      = {Frank Zalkow and Meinard M{\"u}ller},
title       = {Learning Low-Dimensional Embeddings of Audio Shingles for Cross-Version Retrieval of Classical Music},
journal     = {Applied Sciences},
volume      = {10},
number      = {1},
year        = {2020},
doi         = {10.3390/app10010019},
url-pdf     = {2020_ZalkowM_AudioShingle_AppliedSciences_ePrint.pdf}
}
  9. Frank Zalkow and Meinard Müller
    CTC-Based Learning of Chroma Features for Score-Audio Music Retrieval
    IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 2957–2971, 2021. PDF Details DOI 
    @article{ZalkowMueller21_ChromaCTC_TASLP,
author      = {Frank Zalkow and Meinard M{\"u}ller},
title       = {{CTC}-Based Learning of Chroma Features for Score-Audio Music Retrieval},
journal     = {{IEEE}/{ACM} Transactions on Audio, Speech, and Language Processing},
volume      = {29},
pages       = {2957--2971},
year        = {2021},
doi         = {10.1109/TASLP.2021.3110137},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/2021_TASLP-ctc-chroma},
url-pdf = {https://ieeexplore.ieee.org/document/9531521},
}
  10. Frank Zalkow, Sebastian Rosenzweig, Johannes Graulich, Lukas Dietz, El Mehdi Lemnaouar, and Meinard Müller
    A Web-Based Interface for Score Following and Track Switching in Choral Music
    In Late-Breaking and Demo Session of the International Conference on Music Information Retrieval (ISMIR), 2018. PDF Demo 
    @inproceedings{ZalkowRGDLM18_InterfaceChoralMusic_ISMIR-LBD,
author = {Frank Zalkow and Sebastian Rosenzweig and Johannes Graulich and Lukas Dietz and El Mehdi Lemnaouar and Meinard M{\"u}ller},
title = {A Web-Based Interface for Score Following and Track Switching in Choral Music},
booktitle = {Late-Breaking and Demo Session of the International Conference on Music Information Retrieval ({ISMIR})},
address = {Paris, France},
year = {2018},
url-pdf = {2018_ZalkowRGDMM_Carus_ISMIR-LBD.pdf},
url-demo={https://www.audiolabs-erlangen.de/resources/MIR/2018-ISMIR-LBD-Carus}
}
  11. Frank Zalkow, Angel Villar Corrales, TJ Tsai, Vlora Arifi-Müller, and Meinard Müller
    Tools for Semi-Automatic Bounding Box Annotation of Musical Measures in Sheet Music
    In Demos and Late Breaking News of the International Society for Music Information Retrieval Conference (ISMIR), 2019. PDF Demo 
    @inproceedings{ZalkowVTAM19_MeasureAnnotation_ISMIR-LBD,
author      = {Frank Zalkow and Angel Villar Corrales and TJ Tsai and Vlora Arifi-M{\"u}ller and Meinard M{\"u}ller},
title       = {Tools for Semi-Automatic Bounding Box Annotation of Musical Measures in Sheet Music},
booktitle   = {Demos and Late Breaking News of the International Society for Music Information Retrieval Conference ({ISMIR})},
address     = {Delft, The Netherlands},
year        = {2019},
url-demo    = {https://www.audiolabs-erlangen.de/resources/MIR/2019-ISMIR-LBD-Measures},
url-pdf   = {2019_ZalkowVTAM_BoundingBox_ISMIR-LBD.pdf}
}

Projected-Related Ph.D. Thesis

  1. Frank Zalkow
    Learning Audio Representations for Cross-Version Retrieval of Western Classical Music
    PhD Thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg, 2021. PDF Details 
    @phdthesis{Zalkow21_CrossVersionRetrieval_PhD,
author      = {Frank Zalkow},
year        = {2021},
title       = {Learning Audio Representations for Cross-Version Retrieval of Western Classical Music},
school      = {Friedrich-Alexander-Universit{\"a}t Erlangen-N{\"u}rnberg},
url-details = {https://opus4.kobv.de/opus4-fau/frontdoor/index/index/docId/16777},
url-pdf     = {2021_Zalkow_AudioRepRetrieval_ThesisPhD.pdf}
}
  2. Sebastian Rosenzweig
    Interactive Signal Processing Tools for Analyzing Multitrack Singing Voice Recordings
    PhD Thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg, 2022. PDF Details 
    @phdthesis{Rosenzweig22_Singing_PhD,
author      = {Sebastian Rosenzweig},
year        = {2022},
title       = {Interactive Signal Processing Tools for Analyzing Multitrack Singing Voice Recordings},
school      = {Friedrich-Alexander-Universit{\"a}t Erlangen-N{\"u}rnberg},
url-details = {https://opus4.kobv.de/opus4-fau/frontdoor/index/index/docId/19750},
url-pdf     = {2022_Rosenzweig_Singing_ThesisPhD.pdf}
}

Further Links