The way people create music has largely been influenced by advances in technology. For instance, the past century witnessed the prevalence of electronic instruments and synthesizers in music making, the use of software for composition, and accordingly the development of new genres such as rock, metal and electronic dance music. Computer technology plays significant roles in these changes. Over the past few decades, the music industry has rapidly digitalized, and people are now used to listen to music and watch videos online. This paradigm shift calls for more efficient ways in music composition, production, and their combination with other multimedia modalities such as videos.

In light of this demand, we have been working on the “GenMusic” project since 2016, aiming to develop algorithms for AI-based music generaiton/composition. Our goal is not to build products but instead to address challenging research issues for this topic. And, to promote research in this field, this is meant to be an open source project and we share with the publich the code for most of our models.

So far, we have been working on the following sub-projects that are related to AI Composer (generating music in the symbolic domain).

• Melody generation (ISMIR'17) (demo)
• Melody harmonization (demo)
• Lead sheet generation and interpolation (demo)
• Drum pattern generation (demo)
• Multitrack music generation (AAAI'18) (demo)
• Lead sheet arrangement (ICMLA'18) (demo)

And the following sub-project that is related to AI Performer (rendering score into expressive music)

• CNN-based score-to-audio generation (AAAI'19) (demo)

Our research have the following features:

• We work on multitrack pianorolls a lot. We have also built a Python library called pypianoroll (link) to process pianorolls.
• We use convolutional layers quite often, which are easier to train than recurrent laeyrs often used in other music generation models.
• The MidiNet and MuseGAN models we proposed represent respectively the first attempt to use generative adversarial networks (GAN) to generate music in the form of melodies and pianorolls. The two paeprs (ISMIR'17 and AAAI'18) have been cited collectively by more than 100 times. The MuseGAN github repo (link) has over 240 stars by far.
• We propose a new task called "lead sheet arrangment" (link) and propose to use chord to connect the data space of lead sheets and multitrack pianorolls.

We are currently working on the following topics:

• Emotion-based music generaiton
• Generating music with longer structure
• Generating the instrumentation of music
• Benchmarking melody harmonization and chord-based melody generation