Applications and Demos
A Real-Time Beat Tracking System with Zero Latency and Enhanced Controllability (TISMIR 2024)
Peter Meier, Ching-Yu Chiu, Meinard Müller
This Accompanying Website is related to the following publication:
- Peter Meier, Ching-Yu Chiu, and Meinard Müller
A Real-Time Beat Tracking System with Zero Latency and Enhanced Controllability
Transactions of the International Society for Music Information Retrieval (TISMIR), 2024. Demo DOI@article{MeierCM24_RealTimePLP_TISMIR, author = {Peter Meier and Ching-Yu Chiu and Meinard M{\"u}ller}, title = {{A} Real-Time Beat Tracking System with Zero Latency and Enhanced Controllability}, journal = {Transactions of the International Society for Music Information Retrieval ({TISMIR})}, year = {2024}, doi = {10.5334/tismir.189}, url-demo = {https://audiolabs-erlangen.de/resources/MIR/2024-TISMIR-RealTimePLP} }
Abstract
Identifying beat positions in music recordings, a central task in Music Information Retrieval (MIR), is commonly referred to as beat tracking. Typically, this involves computing an activation function to reveal beat information and then conducting post-processing to derive final beat positions. Existing methods often operate offline, requiring access to the entire music track for processing. In this article, we introduce a real-time beat tracking system based on the predominant local pulse (PLP) concept, originally designed for offline use. Our main contribution is the successful transformation of the PLP-based algorithm into a real-time procedure. Unlike traditional offline methods providing static beat positions, our real-time approach dynamically captures changes in local pulse characteristics with each frame of an audio stream. This yields additional insights, including beat context, beat stability, and beat lookahead for predicting beats in advance. In this way, our system not only demonstrates high controllability for real-time applications but also can operate at zero latency. Additionally, we present experiments comparing our real-time beat tracking system with other models and evaluating the accuracy of our lookahead feature. Finally, we showcase two real-world applications for interactive music making and educational music gaming that creatively leverage our system's output. In summary, our real-time beat tracking system offers a lightweight algorithm that is particularly well-suited for interactive music software development.
A Real-Time Approach for Estimating Pulse Tracking Parameters for Beat-Synchronous Audio Effects (DAFx 2024)
Peter Meier, Simon Schwär, Meinard Müller
This Accompanying Website is related to the following publication:
- Peter Meier, Simon Schwär, and Meinard Müller
A Real-Time Approach for Estimating Pulse Tracking Parameters for Beat-Synchronous Audio Effects
In Proceedings of the International Conference on Digital Audio Effects (DAFx), 2024. Demo@inproceedings{MeierSM24_RealTimePulseParameters_DAFX, title = {{A} Real-Time Approach for Estimating Pulse Tracking Parameters for Beat-Synchronous Audio Effects}, author = {Peter Meier and Simon Schw{\"a}r and Meinard M{\"u}ller}, booktitle = {Proceedings of the International Conference on Digital Audio Effects ({DAFx})}, year = {2024}, address = {Guildford, Surrey, UK}, url-demo = {https://audiolabs-erlangen.de/resources/MIR/2024-DAFx-RealTimePLP} }
Abstract
Predominant Local Pulse (PLP) estimation, an established method for extracting beat positions and other periodic pulse information from audio signals, has recently been extended with an online variant tailored for real-time applications. In this paper, we introduce a novel approach to generating various real-time control signals from the original online PLP output. While the PLP activation function encodes both predominant pulse information and pulse stability, we propose several normalization procedures to discern local pulse oscillation from stability, utilizing the PLP activation envelope. Through this, we generate pulse-synchronous Low Frequency Oscillators (LFOs) and supplementary confidence-based control signals, enabling dynamic control over audio effect parameters in real-time. Additionally, our approach enables beat position prediction, providing a look-ahead capability, for example, to compensate for system latency. To showcase the effectiveness of our control signals, we introduce an audio plugin prototype designed for integration within a Digital Audio Workstation (DAW), facilitating real-time applications of beat-synchronous effects during live mixing and performances. Moreover, this plugin serves as an educational tool, providing insights into PLP principles and the tempo structure of analyzed music signals.
Audio Examples for Real-Time PLP Control Signals
Peter Meier, Simon Schwär, Meinard Müller
This Demo Website is related to the following publication:
- Peter Meier, Simon Schwär, and Meinard Müller
A Real-Time Approach for Estimating Pulse Tracking Parameters for Beat-Synchronous Audio Effects
In Proceedings of the International Conference on Digital Audio Effects (DAFx), 2024. Demo@inproceedings{MeierSM24_RealTimePulseParameters_DAFX, title = {{A} Real-Time Approach for Estimating Pulse Tracking Parameters for Beat-Synchronous Audio Effects}, author = {Peter Meier and Simon Schw{\"a}r and Meinard M{\"u}ller}, booktitle = {Proceedings of the International Conference on Digital Audio Effects ({DAFx})}, year = {2024}, address = {Guildford, Surrey, UK}, url-demo = {https://audiolabs-erlangen.de/resources/MIR/2024-DAFx-RealTimePLP} }
Example
In this example, we use a real-time PLP system to modulate the cutoff frequency of a low-pass filter, automatically shaping the sound of instruments such as bass or piano in a rhythmically interesting way. In another example, we utilize our system to modulate the position of a wavetable synthesizer in real-time.
An Interactive Music Game for Singing Education and Data Collection
Peter Meier, Simon Schwär, Meinard Müller
This Accompanying Website describes a music gaming experiment conducted at the Dagstuhl Seminar 24302: Learning with Music Signals.
Abstract
In previous work [1], we developed a prototype for an interactive music game called "Sing Your Way." The prototype resembles a jump-and-run game and uses a gaming controller as the input device. Additionally, it incorporates the player's singing voice to interact with the game world. For this, we estimate the pitch of a microphone signal in real time and use it to control parts of the game world. For example, by singing a note, a pitch line appears in the game that a player can jump on to add stair-like elements, allowing them to overcome obstacles and reach the end of a level. The game is deliberately kept simple, with a limited set of rules: players can move left, move right, jump, and sing notes, where pitch determines the vertical position in the game world. Within this simple gaming environment, we consider two specific use cases: singing education and data collection for MIR research. First, the prototype serves as a tool for singing education, offering a fun and motivating environment for practicing singing techniques. Players are challenged with different gaming levels that include various voice-related tasks, such as singing the correct notes, intervals, chords, scales, or melodies. Combining gamification with educational objectives helps players train their ear and improve their vocal control in an engaging and playful manner. Second, the game can also serve as a valuable tool for collecting data on vocal performance and singing style. By capturing detailed information about players' pitch accuracy, timing, and vocal range, educators and researchers can analyze trends and patterns in singing proficiency. This data can contribute to research in music education and assist in developing new, personalized methods for interactive vocal training and assessment. At the Dagstuhl Seminar, we presented a demonstrator of our game and invited feedback and discussions on the following topics: user experience, educational impact, technical improvements, potential collaborations, and data privacy. By actively playing our music game prototype, the Dagstuhl Seminar participants provided diverse perspectives and ideas on game level design for music education and MIR research.
References
- Peter Meier, Simon Schwär, Gerhard Krump, and Meinard Müller
Evaluating Real-Time Pitch Estimation Algorithms for Creative Music Game Interaction
In: INFORMATIK 2023 — Designing Futures: Zukünfte gestalten, Gesellschaft für Informatik e.V.: 873–882, 2023. DOI@incollection{MeierSKM23_EvaluatingPitchGame_GI, author = {Peter Meier and Simon Schw{\"a}r and Gerhard Krump and Meinard M{\"u}ller}, title = {Evaluating Real-Time Pitch Estimation Algorithms for Creative Music Game Interaction}, booktitle = {INFORMATIK 2023 -- Designing Futures: Zuk{\"u}nfte gestalten}, publisher = {Gesellschaft f{\"u}r Informatik e.V.}, address = {Bonn, Germany}, year = {2023}, pages = {873--882}, doi = {10.18420/inf2023_97}, }
Automated Real-Time Beat Tracking: Response Time and Confidence Analysis (Bachelor Thesis)
This Demo Website is related to the following Bachelor Thesis:
- Rico Rosenbusch
Automated Real-Time Beat Tracking: Response Time and Confidence Analysis
Bachelor Thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 2024. Demo@misc{Rosenbusch24_ARTBeaT_Bachelor_Thesis, title = {Automated Real-Time Beat Tracking: Response Time and Confidence Analysis}, author = {Rico Rosenbusch}, year = {2024}, note = {Bachelor Thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)}, url-demo = {https://audiolabs-erlangen.de/resources/MIR/2024-ARTBeaT} }
Real-Time Beat Tracking for Creative Music Production (Bachelor Thesis)
This Demo Website is related to the following Bachelor Thesis:
- Ole Frederik Müermann
Real-Time Beat Tracking for Creative Music Production
Bachelor Thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 2024. Demo@misc{Müermann24_RealTimeBeatMusicProduction_Bachelor_Thesis, title = {Real-Time Beat Tracking for Creative Music Production}, author = {Ole Frederik Müermann}, year = {2024}, note = {Bachelor Thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)}, url-demo = {https://audiolabs-erlangen.de/resources/MIR/2024-RealTimeBeat-MusicProduction} }