An Interactive Music Game for Singing Education and Data Collection
Dagstuhl Music Game Experiment
Peter Meier, Simon Schwär, Meinard Müller
Abstract
In previous work [1], we developed a prototype for an interactive music game called "Sing Your Way." The prototype resembles a jump-and-run game and uses a gaming controller as the input device. Additionally, it incorporates the player's singing voice to interact with the game world. For this, we estimate the pitch of a microphone signal in real time and use it to control parts of the game world. For example, by singing a note, a pitch line appears in the game that a player can jump on to add stair-like elements, allowing them to overcome obstacles and reach the end of a level. The game is deliberately kept simple, with a limited set of rules: players can move left, move right, jump, and sing notes, where pitch determines the vertical position in the game world. Within this simple gaming environment, we consider two specific use cases: singing education and data collection for MIR research. First, the prototype serves as a tool for singing education, offering a fun and motivating environment for practicing singing techniques. Players are challenged with different gaming levels that include various voice-related tasks, such as singing the correct notes, intervals, chords, scales, or melodies. Combining gamification with educational objectives helps players train their ear and improve their vocal control in an engaging and playful manner. Second, the game can also serve as a valuable tool for collecting data on vocal performance and singing style. By capturing detailed information about players' pitch accuracy, timing, and vocal range, educators and researchers can analyze trends and patterns in singing proficiency. This data can contribute to research in music education and assist in developing new, personalized methods for interactive vocal training and assessment. At the Dagstuhl Seminar, we presented a demonstrator of our game and invited feedback and discussions on the following topics: user experience, educational impact, technical improvements, potential collaborations, and data privacy. By actively playing our music game prototype, the Dagstuhl Seminar participants provided diverse perspectives and ideas on game level design for music education and MIR research.
Game Levels
For the Dagstuhl seminar, we have designed different levels prototypes (0-5) with specific goals and challenges, which are described in the following sections.
Level 0: Introduction to Game Mechanics
In Level 0, the player is introduced to the general game mechanics. The game window displays a jump-and-run style game where the character can be controlled by the player using a traditional gaming controller. The player can move left, right, and jump to overcome obstacles; however, the jump height is limited. To reach higher terrain, the player must utilize an additional game control mechanism. By using their singing voice to sing a note, a pitch line appears in the game. The vertical position of this pitch line represents the fundamental frequency (F0) of the singing, with varying frequencies for higher or lower pitches.
Note Blocks
The player can jump onto this pitch line to create a "note block." This note block stores the pitch at the moment when the player lands on it from above. Jumping on the note block again reproduces the stored pitch, allowing players to make sounds within the game and potentially play notes or even entire melodies if multiple note blocks are involved. Furthermore, these note blocks serve as steps to help reach higher terrains and ultimately complete each level.
Note blocks can be deleted by pressing the dedicated delete button while jumping on the blocks. This is useful for removing or correcting a misplaced note block to achieve better scores (more on that later) or to clear the path in a game level. Furthermore, this delete feature enables the player to creatively use note blocks to build something, such as a staircase or bridge to other platforms.
Another feature of note blocks is the ability to mute them by pressing the dedicated mute button while jumping on them. With this feature, the player can add soundless blocks into the game. This can be useful for challenging the player with certain tasks, such as creating a specific interval with note blocks and playing the notes back one after another. For such a task, the player would need to climb up or down a staircase without making a sound.
Collectables
When the player reaches the high plateau in Level 0, fruits can be collected simply by moving into them. Like note blocks, collecting fruits also produces a sound by playing a short note that corresponds to their vertical position in the game. The difference with note blocks is that fruits disappear once they are collected. In this way, fruits not only serve as collectibles (potentially granting points) but also guide players to hear or sing certain pitches for a single time. This opens up interesting possibilities for different game level designs to challenge the player, which we will discuss in subsequent examples.
Root Note
When starting the game, a root note can be set that matches the vertical position of the ground terrain in the game. This is useful for adjusting the pitch line to a comfortable singing height for each player, as there might be significant differences between male and female voices.
Score
The score displayed in the upper left corner of the game level is calculated based on the stored pitch values (in cents) of all note blocks in the game. The closer a note block lands to a fixed half-note grid (in cents) based on the root note, the higher the score. For example, when a player creates a C3 note with 0 cents deviation, they are granted full points of 100. However, if the player is 20 cents off the grid, they receive only 80 points. In this way, players can train to hit pitches more accurately. The maximum possible score for the current note blocks in the game is also given as a value separated by a slash behind the actual score, e.g., like 80/100. This way, players know they have earned 80 out of 100 possible points.
Users Playing Level 0
Level 1: Listening and Singing Notes
In Level 1, the player is challenged to select each fruit and replace it with note blocks that have corresponding pitches. This allows the player to train their musical skills by listening to a given pitch (by collecting the fruit) and reproducing this pitch with their own singing voice (by singing a note, holding it, and jumping on it). The fruits are arranged so that the player ends up singing the first three notes of a major scale, with each note being a whole step interval apart. To help the player reach higher terrains, there are also fixed grass blocks that the player can use to stand on.
Users Playing Level 1
Level 2: Mastering Simple Intervals
Similar to Level 1, in Level 2, the player has to collect fruits and replace them with note blocks to reach the end of the level. This time, there are not only whole step intervals but also half step intervals, as the fruits are given in such a way that they represent the first five notes of a major scale. However, there are no aiding grass blocks anymore, so the player must climb up using only the note blocks they created themselves.
Users Playing Level 2
Level 3: Major Scale Challenge
With Level 3, we extend the challenge from Level 2 to encompass an entire major scale from bottom to top. In doing so, we end up with half-tone steps between the 3rd and 4th, as well as the 7th and 8th positions of the scale.
Singing eight notes in succession while jumping on note blocks without falling to the ground requires a high level of precision in controlling the player character with the game controller. This adds another layer of complexity to the game and becomes more difficult the longer the scale is to sing.
Users Playing Level 3
Level 4: Sing a Song
In Level 4, we introduce a new game level idea by presenting the player with an empty level and one task: sing the first few lines of the children's song "Are You Sleeping, Brother John" into the game.
Are you sleeping? Are you sleeping?
Brother John, Brother John.
The note blocks must all fit within the initial game window. The order of the blocks is not important, but it should be possible to play back the melody by jumping on the blocks one by one.
As there are no reference tones in the form of a fruit or note block in the game, players can use a new guidance method: the drone tone. This tone represents the root note, which is set individually for each player at the startup of the game and can be activated or deactivated by pressing a dedicated button on the controller.
This gives players great flexibility to creatively solve this task, but it also introduces interesting new challenges, such as solving it very quickly (speed run), achieving high accuracy in pitch, or completing the level with or without using the drone tone.
Users Playing Level 4
Level 5: Flappy (Bird) Beethoven
Level 5 introduces an exciting new game mechanic: flying mode. By pressing and holding a dedicated button on the gaming controller, the character enters flying mode, and its vertical position is set to the pitch line. In this mode, players can fly through narrow pipes or gaps in the level, similar to games like Flappy Bird, but this time it is controlled by the player's singing voice.
The flying mode also offers another type of vocal skill that can be developed. Instead of singing stable, steady pitches one by one, players can smoothly slide between different pitches to change their vertical position in the game. This continuous change of pitch provides a fun and interesting way to train the voice. For example, the player could (a) try to navigate through the pipe course in one take with just one breath and (b) aim to avoid touching the pipes to reach the end of the level.
While solving Level 5, the player might notice the famous four-note "short-short-short-long" motif from Ludwig van Beethoven's Symphony No. 5 in C minor, Op. 67.
Users Playing Level 5
User Experience Evaluation
To evaluate the user experience, we conducted usability testing by observing users as they interacted with the product to identify any issues or challenges they faced. Furthermore, we conducted one-on-one discussions with users to gain deeper insights into their experiences and preferences.
While playing and testing the game, participants offered valuable insights and feedback on both the positive and negative aspects of the game prototype. They also provided suggestions for improvement. The most notable points are summarized in the following section, where we have also noted our overall impression of the gaming experiment.
Positive Aspects
- Innovative Mechanics: Participants appreciated the incorporation of musical elements into the gameplay. For instance, the ability to jump on note blocks and hear the associated tones was described as an engaging experience.
- Interactive Audio Elements: The idea of having a drone tone and being able to hear a root note for guidance received positive feedback. The possibility of deactivating or playing back this note offered flexibility that users appreciated.
- Creativity in Gameplay: The idea of integrating music and gameplay, such as singing to match notes after collecting a fruit, was appreciated. The educational aspect of recognizing and replicating notes seemed intriguing.
- Flight Mode Feature: The ability to enter flight mode by using a specific button was also positively received, indicating that this mechanic allows more freedom in gameplay, which participants found engaging.
Negative Aspects
- Difficulty Learning Curve: There were challenges related to understanding certain mechanics and gameplay objectives. Initial confusion about how to interact effectively with the game indicates that the instructions might not be fully clear or intuitive.
- Complex Controls: Participants found the controls confusing, particularly the need to coordinate using their singing voice and controller buttons for movement and jumping simultaneously. Without experience in playing jump-and-run games, the controls seemed too complex for beginners.
- Lack of Feedback: The game currently lacks auditory feedback, like "sparkling sounds," to signify when actions are performed correctly. This absence made it difficult for participants to have a satisfying interaction.
- Sound Design Issues: Participants noted that the drone tones produced by the in-game mechanics were too aggressive. The suggestion was to modify these tones to be less aggressive, potentially by adjusting the wave shape to something between pure sine tones and a sawtooth.
Suggestions for Improvement
- Clearer Instructions: Enhancing the clarity of initial instructions, especially regarding controls and musical elements, would improve the user experience.
- Guided Learning: Implementing a tutorial that clearly explains how to use singing to navigate the game could help new players better understand and enjoy the mechanics.
- More Feedback Mechanisms: Providing immediate feedback or visual aids to show how close a user is to the correct pitch could enhance understanding and enjoyment. Additionally, including feedback or rewards for when players successfully complete levels can enhance their sense of accomplishment.
- Adjust Sensitivity: Refining pitch detection to be more forgiving or accurate could help players with varying singing experience feel more in control of the game.
Overall Impression
The game concept is innovative and has the potential to be both engaging and educational. While participants enjoyed the novel mechanics, there were technical and clarity challenges that, if addressed, could greatly enhance the overall player experience. The combination of gameplay and music presents unique opportunities for creativity and learning, making the game appealing to a broad audience, particularly children.
References
- Peter Meier, Simon Schwär, Gerhard Krump, and Meinard Müller
Evaluating Real-Time Pitch Estimation Algorithms for Creative Music Game Interaction
In: INFORMATIK 2023 — Designing Futures: Zukünfte gestalten, Gesellschaft für Informatik e.V.: 873–882, 2023. DOI@incollection{MeierSKM23_EvaluatingPitchGame_GI, author = {Peter Meier and Simon Schw{\"a}r and Gerhard Krump and Meinard M{\"u}ller}, title = {Evaluating Real-Time Pitch Estimation Algorithms for Creative Music Game Interaction}, booktitle = {INFORMATIK 2023 -- Designing Futures: Zuk{\"u}nfte gestalten}, publisher = {Gesellschaft f{\"u}r Informatik e.V.}, address = {Bonn, Germany}, year = {2023}, pages = {873--882}, doi = {10.18420/inf2023_97}, }