Anybody can be a composer

Ed Newton-Rex is a pioneer in the field of AI music composition. When people find out that he’s been doing this for ten years, they often say: "That’s interesting, what does AI music sound like?” His answer is invariably: "It doesn't sound like anything" – not in the sense that it sounds horrible, but that you shouldn't expect this technology to produce a specific sound of its own. If you listen to synthesizer music from the early eighties – say Human League or the Eurythmics – it had its own instantly recognizable character, and was a document of its time. AI is different, as the entries to the AI Song Contest show. When we speak to Newton-Rex, a member of the contest jury, he’s just finished listening to all the entries. What strikes him most is their enormous diversity. Although the teams used the same tools, no two songs sound the same, and to him this confirms that AI music is not a style or genre.

AI is a new music-making technique that you can use in many creative ways. Vincent Koops leads the contest panel. He is an AI researcher and composer, with degrees in Sound Design and Music Composition from the Utrecht School of the Arts, and in Artificial Intelligence from Utrecht University. He has a PhD in music information retrieval and works on AI multimedia projects at the Dutch broadcaster RTL. What Koops noticed when listening to the songs was the many different ways in which the teams used AI techniques. Each team also provided a process document describing how their song was created. “Many did not necessarily have an academic musical background, but consisted of data scientists who participated because they thought it was cool to experiment,” Koops says. “And they sometimes came up with amazing solutions. For example, in OSX’s Terminal there's an internal speech computer that some people used to generate vocals for their songs, and then put it through autotune. That’s super creative, and I would never have come up with something like that myself. Another team tried to turn music into images, to make music out of them. You can't just feed an AI machine music, you have to convert it into some representation. Most people use MIDI, but this particular team used images. Not the most obvious solution, but apparently a technique they were familiar with.”

Ed Newton-Rex founded the music composition company Jukedeck ten years ago, and it was acquired by ByteDance in 2019. It provides AI music for video and other media, and has created more than a million original pieces of music. So you might say he's seen a lot in the field of AI music creation. And yet he was amazed to see how one team combined Bach's music with death metal to train their AI. “If I composed a song myself without AI, as a human being, it’s quite possible that I’d have been influenced consciously or unconsciously by multiple genres, like metal or Bach. In that sense, AI music composition fits in with how we humans are creative. Still, those genres would influence the composition, but you would hear the references back less directly and clearly, as with this special AI music composition. It delivers something unique.” This team was also creative in generating their lyrics: they trained their AI in German and English. Newton-Rex: 'In doing so, they took advantage of the limitation of AI: it doesn't know how to choose between German or English, or that it has to translate, because it has no general intelligence. Something very strange and surprising emerges which  a person would never make up.”

All the teams used AI to generate different components of their songs, such as lyrics and melody. But the entries showed the panel that having AI produce a coherent song at the push of a button is a bridge too far. Firstly, not all the output generated by the AI machine turned out to be usable, panel member Anna Huang noticed. She is a leading AI researcher in the field of music, who previously worked on the Magenta project at Google Brain - an open-source tool for generating AI music that many teams have used. She is also the creator of the machine learning model behind Google’s first AI-powered Doodle, the Bach Doodle, which in two days harmonized more than 55 million user created melodies around the world. Her compositions have won several awards, including first place in the San Francisco Choral Artists New Voices project. 

She found it remarkable to see how some teams played ping-pong with their AI. “When the AI didn't necessarily provide immediately usable output, I saw teams using it as inspiration, interpreting the output, re-feeding their AI and so on," she says. “I noticed that people with AI dare to take more risks and are more inclined to experiment. Logically, it doesn't take much time and effort to follow a crazy melody line and see what the AI does with it. You can easily try something out, and if it doesn't work out, that's okay: you don't throw away hours of work.”

Furthermore, slotting the AI-generated components together always involves a human hand, and the musician becomes more of an editor and producer. “You collect and curate,” Huang says. “That can be done manually, but I also saw teams that created an algorithm to help with the curation, to make the AI output focused so that you're not overwhelmed by thousands of possibilities.” Newton-Rex was also struck by the way in which another team tried to apply randomness in choosing which AI parts to use. "The team members rolled the dice to determine which elements would be used for the song. I found that an interesting and fun way to add even more surprise to your entry.”

The teams' AI machines have been working overtime. Melody generated by an AI: check. Don't write the text yourself but have the AI write one: check. But generating those elements simultaneously so that everything fits together is very difficult for an AI, Vincent Koops says. “In many cases, the teams have used the hamburger method, building a song using different layers. This is not radically different from how existing pop musicians do it. And orchestral pieces are not linearly composed either, all the instruments at the same time. What people are good at is improvising: playing a melody on a guitar and improvising lyrics at the same time. Computers may be able to improvise a bit here and there, but not all things at the same time in a coherent way.”

Does this mean we can breathe a collective sigh of relief, and that the rise of AI does not herald a new dystopia? Are humans still better than computers when it comes to creating music? Anna Huang thinks the AI Song Contest shows how every step in the process involves human labour: choosing the data to input, training the AI, curating the output and putting it all together. All AI art, including music, is art for people, and that means that human elements have to be added to it, Vincent Koops explains. “A system that has never seen human art cannot generate art that’s interesting to a human being. Melodies adhere to certain forms and you have to train an AI. That means you have to point the AI machine in a direction, you have to steer it. If you start generating music from noise you get a sound cloud, random clutter. Not a song.”

But Ed Newton-Rex says that for some forms of music, computers can take over from people just fine. Music that functions as wallpaper, for example in the background in shops or video games –  there's no need for people to be pressing buttons. But he hastens to add that AI will not deliver the new David Bowie. “In that case, you're talking about artistically made music, and that's a different métier. An AI can't create something with a purpose. Because it has no idea what it’s doing. Artists do.” 

“Musicians also respond to the spirit of the age”, Anna Huang adds. “How we feel about ourselves at a certain point: a lot of music during this coronavirus pandemic will probably be about loneliness and feeling isolated, but also new ways of supporting and connecting with each other. People can reflect on the zeitgeist, computers can't.”

The panel has found that apart from inspiration, the main thing AI offers the music world is democratisation. Many team members in the AI Song Contest have no musical background, and admit that without AI they would never compose music. Vincent Koops observes: “The teams that do have a musical and technical background used state-of-the-art tools and did everything in a neat academic way. Those that didn't have that background were looking for fun solutions. This sometimes resulted in very interesting harmonies, even though the production didn't always sound very slick. With AI, these people’s creative worlds have expanded. They thought they weren't musical because they couldn't play an instrument, but your ability to express your ideas in music is more than just playing a guitar. The great thing is that with AI, anybody can be a composer now.” But doesn't that level down the craft of music making enormously? Koops thinks it's not that bad. “I just think it raises the bar. It challenges other artists and composers to push new boundaries and invent things that people with AI can't come up with.”