Mini Minstrels
A teeny choir that sings original songs, written for you in real time.
Making Music
For years, I've been interested in getting a computer to compose music that sounds good. Back in 2022, when I was applying to PhD programs, I even mentioned it in a couple of interviews as a potential research area. Growing up, I did piano and cello lessons and sang in various choirs. I even joined BYU's barbershop chorus when I was an undergrad. I love music, and I love making music. I also love computer science, and teaching computers to do interesting things.
In 2023, I trained a small transformer model on 23,000 transcribed songs to write piano music. It did okay - the music sounded kind of like someone who half-remembered how to play the piano. Here's an example of the kind of music it could write:
At least it wasn't discordant trash, but it wasn't great either. The next year, I created a very small Markov model from about 400 of Bach's chorales. It made music that actually sounded like a Bach chorale, at least for a few seconds. However, it only remembered the last 2 notes it wrote, so there was no long-term structure. It could also only write songs that sounded like Bach chorales.
It's been a few years since my last attempt, and I'm a much better machine learning engineer than I was back then. I decided that perhaps the third time would be the charm, and that I should try again.
AI Music
This might be hard to believe based on the last few paragraphs, but I don't really like AI music. You might have encountered some of this music online - it's often used as background audio in short-form videos. Since anyone can make it, it's everywhere. I generally don't like all the AI slop music that can be found online.
I have a similarly complex relationship with many aspects of generative AI. It can be extremely useful, but also runs a risk of usurping human creativity, or being an easy path that leads us into complacency. There's a time and a place for everything. Sometimes, judicious AI usage is good though often it is not.
Even though LLMs like ChatGPT are good at writing, I never use them to write any of my blog posts. I do, however, use them for proofreading. I sometimes use generative AI to create thumbnails for my blog, but I make sure to disclose this in the alt text. I've never used a commercial music generator to generate music, though I've used AI to help me navigate a new DAW or work on an effects chain while recording music. It's a balancing act.
Many of the common criticisms of AI music resonate with me. Frontier music generation models were trained on commercial music without consent from the artists and without compensating them. There is also an "AI quality" to many of these songs. Just like a practiced eye can usually distinguish AI photos or videos, AI music has its own unique sound, which I personally find off-putting.
AI music also has the potential to impact artists. High-quality AI music could replace humans without the AI ever demanding royalties or other considerations. Music generation is also beholden to the environmental concerns associated with generating any AI content.
While creating the Mini Minstrels, I was very thoughtful about these concerns, and I think I avoided basically all of these issues.
Teaching Music to the Minstrels
How does top-of-the-line AI music generation work? While companies like Suno and Udio keep their systems proprietary, we have a pretty good idea of what's going on beneath the hood. These models are usually transformers, like the LLMs you may have used as chatbots. They are trained on massive amounts of audio data and learn to generate more audio like what they 'heard' during training. This is known as raw audio generation. Sometimes, these models also have a direct language model component that creates the lyrics, while another component creates the audio.
The minstrels, like my last two music-generation projects, perform what is known as symbolic music generation. In this way, you could think of the model as reading a bunch of sheet music, learning from it, and then writing sheet music.
Before I could teach the minstrels, I needed a bunch of data. I didn't want to train on any data that wasn't allowed, so every song the minstrels learned from is in the public domain in the US or Canada.
Even if I had all the sheet music in the world, this would be exponentially less data than most language models are trained on. There's a lot more text on the internet than sheet music. Since my last attempt to create a model that would write music, I've learned a lot about training under data-constrained regimes (even participating in an academic challenge about it). I learned from the TinyStories paper that constraining to a well-defined domain could allow a small model to perform very well. The Phi models (a series of small and impressive LLMs from Microsoft) showed that high-quality data could compensate for quantity.
So I knew I needed to pick a smaller domain. In my past singing with choirs and playing in orchestras, I developed an affinity for polyphonic and multi-part music. I decided that would be a good type of music to try to teach the model.
Getting the data was a huge hassle. I started by gathering sheet music from any source I possibly could: the Cyber Hymnal, OpenScore, the Werner Icking Music Archive, the Petrucci Music Library, PDMX, Open Hymnal, Bach/Praetorius chorales, Humdrum, DCML, Mutopia, music21, MuseData, the Josquin Research Project, and more. One of the largest datasets, PDMX (Public Domain MusicXML), had many songs in it that are under copyright and were incorrectly tagged. I applied a filter to weed out anything that might not be in the public domain.
Then, I had to deduplicate it. Famous works could be found in several of these collections, and we wanted to train on each song once. The music was in many different formats: MIDI, MusicXML, Kern, etc., and all needed to be converted into REMI, the same text tokenization I used for the piano transformer. There were many bugs in conversion that needed to be solved, and some pieces were discarded for being low-quality. I tagged each song by genre as well. After all that work, I had a very large collection of music from composers you might know:
| Genre | Pieces | Share of Music | Biggest Names |
|---|---|---|---|
| Hymn | 16,414 | 7% | Bach (chorales), Charles H. Gabriel, W. J. Kirkpatrick |
| Polyphony | 10,075 | 13% | Palestrina, Josquin, Lassus |
| Piano | 9,481 | 19% | Bach, D. Scarlatti, Chopin |
| Chamber | 9,235 | 15% | Vivaldi, Praetorius, Bach |
| Choral | 7,180 | 10% | Moniuszko, Bach, Purcell |
| Art song | 5,153 | 9% | John Rogers Thomas, Stephen Foster, Brahms |
| Organ | 4,638 | 7% | Lasceux, Lebègue, Beauvarlet-Charpentier |
| Other | 1,735 | 2% | 18th-century German chorale books |
| Choral-orchestral | 1,650 | 7% | Volckmer, Raszek, Stefani |
| Orchestral | 1,267 | 7% | Vivaldi, Bach, Stefani |
| String quartet | 1,105 | 4% | Haydn, Mozart, Beethoven |
You may have noticed that there are more hymns than any other genre, but hymns don't contribute the largest percentage of music. Why is that? Well, it's because hymns tend to be pretty short, so 16,000 hymns lined up is the same amount of music as 4,600 organ pieces.
Pretty much every 'classical' (we're including Baroque, Renaissance, etc. music) song I like is in the minstrels' training corpus.
The Minstrels' Brain
Of course, data by itself isn't enough, we needed a model to train. I decided to create a relatively small (95 million parameters) autoregressive transformer, the same kind of model used for LLMs.
Autoregressive language models work by predicting the next token (word) in a series of tokens. So when you ask ChatGPT a question, it guesses the next word (the first word of its response) and repeats the process, guessign new tokens until the next token it predicts means "stop talking." Transformers are based around a mechanism called attention - they care about whatever they said in the past as they decide the next token to generate. This has implications for how we train and structure our data.
Transformers and Inductive Bias
When you create a machine learning model, you impart something called inductive bias.
Decisions made about model architecture have consequences for how that model will learn.
Because the minstrels use an autoregressive model, any important information about the song needs to be visible from the beginning.
To take advantage of this, each song is prefaced with special control tokens.
We start with a BOS token, which means "starting a new song."
Then, we have a series of tokens describing the genre, key, era, and tempo before starting the song.
It looks something like this:
BOS Genre_Hymn Key_G_major Era_Romantic Pace_3 Syl_8 Syl_7 Syl_8 Syl_7 ... EOS
This way, when the model sees a training example, it sees Baroque music prefixed with a Baroque token. It sees a G major token before writing a song in G. After seeing that hundreds of thousands of times, the model learns what the key of G is, how Classical music sounds, or the difference between a hymn and chamber music. We never explicitly teach any of this, just show lots of examples and hope the model learns.
This is a bit of a double-edged sword, though. Models can then learn spurious correlations. For example, imagine that all Baroque songs in the training set were in the key of D, purely by coincidence. Then, the model might accidentally learn a rule like, "All Baroque music is in D." To avoid that, we perform data augmentation. We randomly transpose our pieces up or down some number of half steps before training, so the model sees each piece in multiple keys. (We do have to do this carefully so the transposed piece doesn't play notes that the instrument can't in real life.) We also re-mix the proportions of each genre. There aren't very many string quartets, so the model sees those more often during training. Over a training run, the model sees about 1.2 billion tokens of music.
We also made a few tweaks to the out-of-the-box model. Starting with a Llama-type decoder-only transformer, we followed other research on small models and made the transformer deeper and thinner than we would for larger-scale text generation, with a much smaller vocabulary.
Training the Minstrels
Training the model came with its own share of difficulties. First, I wasn't sure how big of a model we should train, so I trained four different models from 15 million to 180 million parameters. The 95-million-parameter model was as good as the one that was twice as big, and was faster to train and easier to deploy.
Autoregressive transformers have something called a context window. This is the number of tokens the model can attend to at one given time. You could think of it as the maximum amount a model can hold in its short-term memory while coming up with the next token. Modern LLMs have tons of parameters (trillions) and huge context windows (a million tokens). Our model is at least a thousand times smaller than something like Claude Opus, so our context window is only 1,024 tokens long.
This has an impact on how our model learns. Because many songs in our training data are longer than one context window, the model can't see the beginning of the song from the end. This means that for some genres, the model never learns how long the songs should be. If we let the model keep writing, it would go for a really long time, lose track of the beginning of the song, and maybe never end.
The solution here was something called countdown tokens. When we get a certain number of bars away from the end of the song, we insert a token that's meant to teach the model, "We're 16 bars from the end" or "We're 8 bars from the end." Then, when we generate a new song, we can essentially tell the model, "start winding down" by inserting our own countdown token before we want to end. In practice, these countdown tokens make the model stop at the requested length over 98% of the time, which is great. What's not so great is that the model doesn't always write a graceful ending. Sometimes, it stops kind of abruptly.
We only put these countdown tokens in half of the songs so the model wouldn't get confused whether it saw them or not. This held for several of the special control tokens - not putting them in all the songs meant the model didn't need to rely on them in the future.
Training the model took 1-3 hours, depending on size and data, and fit on a single A100 GPU on the BYU supercomputer. I ended up having to train 10 versions of the model, but by the end we had a model that wrote much better music than BachByte or the Piano Transformer. The average model took about the same amount of electricity to train as it takes to bake a batch of cookies in your home oven.
Results
The trained model was better at composing in some genres than others. It was really good at hymns, and not very good at other genres like organ. This shouldn't surprise us. The model saw over 15,000 hymns while training, and these hymns tend to have a pretty straight, simple rhythm and pattern. The training mixture also benefits hymns - most songs are multi-part harmony, so the model saw mostly examples of that, and relatively fewer examples of organ solos or counterpoint. Also - a full hymn easily fits in a context window, so the model had ample opportunity to learn the full structure of each hymn.
Here are two hymns. One is a real hymn from our dataset, but from the evaluation data, so the model never trained on it. The other one was written by the Mini Minstrels. Listen to both and see if you can guess which is which. Both of them are played on a synth organ.
Hymn A
Written by the Mini Minstrels
Hymn B
Emilie, John Wesley Baume (1880)
Our different control tokens work pretty well. Testing hymns, we see that about 97% of hymns follow the requested key signature. It gets meter correct more than half of the time, so you can choose from those meters in the player. Genre works well, and the period seems right most of the time, but that's not something you can measure objectively. There's not really math for whether a song is Baroque or Romantic.
At this point, I had a model that could write music, but nobody to perform it. I knew from my previous projects that something that spat out music was cool, but not very fun.
Creating the Minstrels
The biggest problem with my piano transformer project is that people couldn't try it out on their own. A model that generates music is fine, but not that interesting if all you can do is play samples. I wanted the model to be able to be run client-side. That means that when you use the Mini Minstrels, your computer or phone generates the music. The first thing I did was quantize and export the model.
The parameters inside a machine learning model are stored as numbers. Each number takes a certain amount of space on your computer. More space = more precision. You can think of it sort of like how more decimal places after a number take up more space on a page, and are also more exact. Quantization is a process where you take a trained model and lower the precision on the parameters. You could think of this like rounding a number to reduce the number of decimal places. Ideally, this saves you a lot of space while only hurting performance a little bit.
Our trained model had 95 million parameters, each of which was saved in 32-bit precision. This meant the model took up 365 MB, which is a lot to download to a user's browser. We were able to quantize it down to 4 bits per parameter, which meant the actual model only takes up about 65 MB. That's about the same as opening Google Earth or downloading a podcast episode. And the performance dip from quantization was almost nothing.
I tested the model, and it can write music on phones and laptops. That worked, but it wasn't fun. But I wanted little guys to actually perform the music.
So I decided to design some. I was inspired by Mr. Men/Little Miss books and Bob Books, both of which feature simple, geometric-shaped characters. I created my designs and implemented them as vector images drawn on an HTML canvas. They stand in a church when singing sacred music, and on a stage while singing secular pieces.
It was important that they feel alive, so I made them blink, breathe, follow the cursor with their eyes, move up and down while singing (in a way that corresponds to the pitch they sing), and do little things like wave when they stand still for too long.
Then, I needed to give them voices. Here, I was inspired by the voices from Animal Crossing, The Sims, and Illumination's Minions. I wanted them to seem like they were talking, but clearly not in a real language. To do this, I synthesized voices. Each minstrel has a different voice with a different timbre. Each minstrel also says different consonants as it sings. When singing together, they sync up on vowels, and their mouths match the vowel.
Additional Features
I added several features. If you like the piece the minstrels composed, you can download it as a MIDI file, a MusicXML file to import into music software, or a PDF of sheet music. You can change the tempo, or select one or more minstrels as star performers.
At this point, you're probably itching to try it out, and I don't blame you. Every time you compose a song, the minstrels sing something completely new that nobody on Earth has heard before. Composing a song takes about a minute, depending on your hardware.
Sometimes, the minstrels don't compose that good of a song. That's okay - you can try again! Sometimes, they sing too fast or too slow. In that case, you can help them by changing the tempo slider.
Ethics
Overall, I think this is a good use of AI. All of the training data is in the public domain, so I'm not taking anybody's creative work unjustly. I really don't think the Mini Minstrels are replacing any human creatives. The energy use is pretty negligible. Training the model used less energy than a normal commute, and when you generate new songs, the only electricity used is the charge in your phone or computer.
Try It
The Mini Minstrels are great! I'm super proud of this project, and I feel like I've finally taught a computer to make good music. You should try them out. What are you waiting for? Click Here!