The Future of Creative AI

For over a decade, Mick Grierson has been leading research into creative applications of AI. As a co-founder of the Creative Computing Institute, he has helped to drive inclusive approaches to a range of fields: from creative coding to machine learning, and from sensing to software. In that time, Luba Elliott has become an essential curator in the field of creative AI, a term which Grierson first attributes to Elliott. At a moment when artists are rewiring the digital systems that envelop society, RCS invited them to discuss AI’s trajectory as a creative tool.‍

‍Luba Elliott: Since I started working in creative AI in 2016, I’ve seen the space explode in popularity, beginning with DeepDream, followed by the controversial Christie’s auction of the blurry GAN-generated portrait by Obvious, and now finally the rise of prompt-based art and its increasingly close relationship to the field to NFTs. It feels to me that the latest generation of artists is much more focused on producing images as opposed to using AI as a tool within the broader creative process. As someone who has been in the field of AI for many years, how have you seen the space change in its concerns, terminology, and artistic output? ‍

Mick Grierson: There has been huge change, much of it in a positive direction, but now we’re in a new place and there are new questions at play. 

For example, far fewer people take seriously the idea that AI is going to wake up and take over humanity, and this is largely positive because it means that people can see more easily the genuine opportunities and risks we’re facing — for example, AI and ML being leveraged by political actors and corporations to marginalize communities. 

It’s been interesting to experience this change. The idea that AI could become aware and/or be an independent threat is still very much a live issue in some areas due to the way artificial general intelligence (AGI) has been used as a carrot to entice investment. I am contacted quite regularly by researchers in industry who are tasked with creating AGI. They sometimes want my advice on how to program systems that can simulate human creativity. They also have huge, speculative, philosophical theories about human creativity which make no sense whatsoever to anyone with experience in creative fields, yet somehow they think it’s possible to simulate human processes such as creative action without knowing anything about them — purely by modeling the output data (e.g. paintings). I take seriously what they’re trying to do because they will certainly produce something, even though the way they have been asked to achieve it borders on the insane. 

This kind of work needs to be led by those who know what creative work is at an expert and practical level.

Artists such as Parag Mital, Terry Broad, and Memo Akten have been exploring the affordances of AI and ML (machine learning) among other things, from the perspective of creators. I encouraged them to develop new kinds of core ML tech based on what was becoming available, driven by how people go about making things rather than by how computers might simulate creativity. This is mainly because I care about how new tech enables new affordances and, potentially, new artistic voices in computational arts. That is how historic figures like Daphne Oram, John Whitney Sr., and Vibeke Sorensen developed new kinds of technology through practice.

What is amazing about the past decade is that there has been as much innovation going on at the technical level as there has been at the creative level, which has produced exciting art. At the same time, there’s been a decline in areas such as computational creativity, which foregrounded the idea that AI and ML might be a way to simulate human “creativity.” This led to a form of strong AI where creativity was considered a proxy for awareness, as in works like Marcus Du Sautoy’s book, The Creativity Code (2019). 

It’s now much more widely understood that people with a strong understanding of creative practice who also have a strong technical background can successfully drive technology development.

I’m also deeply impressed by the work of people such as Katherine Crowson, who have helped to catalyze a punk scene in Creative AI, and made AI and ML art-making so much more accessible. While this is leading to a new set of questions around ownership, authorship, novelty, transparency, safety, and so forth, nobody is arguing that such systems are “alive,” which is definitely progress. ‍‍

LE: Do you consider AI to be different from other types of emerging technology? Some artists might argue that their close creative partnership with AI renders it a very different type of technology to quantum computing or blockchain. Perhaps it’s in the eye of the creator. ‍

MG: AI definitely has different capabilities and affordances when compared to other technologies, but this is normal for technology. As an academic, I need to start from a rational position, and there’s much more evidence that AI meets existing definitions of technology rather than representing a fundamental shift in what technology is. I also think it’s fair to describe AI and ML specifically as a form of media technology.

One of the most powerful things that machine learning can do is automate the creation of representations and their relations. You could argue that this feature separates it from all prior forms of technology, but I’m not convinced about that because you could also do this before, it just took a lot more time. However, ML is potentially a powerful tool for those considered “on the fringe.” 

The perspective I’ve tried to reinforce through the Creative Computing Institute (CCI) and at Goldsmiths previously, is that practically everything humanity creates — including technology — is driven by human need. Sometimes this need is a desire to experience and to express, indeed one can argue that the experience of expression is at the very root of the need for some technologies. 

Some forms of expression, such as dancing, don’t require any technology at all. However, a drum, which is an early form of technology, can also transform dancing by providing a new framework within which that kind of expression can occur. The way that AI and ML is representing human culture mirrors its use as a form of expression.  I guess this might be why OpenAI named their text-guided diffusion approach after Salvador Dalí.

The Dadaist cut-up technique is part of the process by which text-guided diffusion systems orient visual compositions. ‍

‍LE: What interests me most is how artists are working with AI technologies that aren’t specifically designed for artistic practice, such as facial recognition or reinforcement learning. Rather than waiting for the next version of DALL-E or ChatGPT, I’m more excited to see what new types of algorithms the AI research labs can develop. How do you see artists working with AI in the future? Do you think AI companies are developing the right tools for artists? ‍

MG: I totally agree — artists can transform technology — often as a form of critique (I’m thinking of Nam June Paik, for example). It’s important that we keep doing so, and part of that is making sure AI technology can be customized. There’s an upside and a downside to the enormous uptake of AI tools by people in all areas, professions, and walks of life, which is driven by the creation and interpretation of media. 

The upside is the incredible potential for increased diversification of creative voices through AI, which is tremendously exciting. 

A downside is that customization isn’t always easy. But it’s important because the idea that those with different perspectives can automate processes the way they wish, producing images they wish, writing text that communicates their ideas, thoughts, feelings, and perspectives, all in a much more accessible way is amazing for people who are otherwise excluded from these activities. 

‍Heart n Soul is an arts charity that we’ve been working with that is focused on the power and talents of people with learning disabilities, physical disabilities, and autism. Working together with a lot of amazing artists, we’ve been working to create new kinds of AI-based communication tools by fine-tuning open source multimodal large language models (LLMs) and generative AI systems. This involves co-design methods, where games that the community has made using paper prototyping have then been turned into interfaces with LLM backends interpreting the inputs. These inputs might be sounds, images (using image adapters), or otherwise physical interfaces comprising two-button devices that create binary classification systems for fine-tuning and customization. 

I think it’s important that everyone feels empowered to create their own AI and ML systems from scratch, which work in precisely the way they want. That is what great artists often do, knowing that it will engineer a new kind of voice, and new kinds of representation. 

Whatever great AI tools end up in the marketplace, they will be no substitute for those tools that artists create for their own purposes.  ‍

‍LE: You’ve worked with sound, music, and visual art. How much does artistic engagement with AI depend on the medium? What particular challenges and affordances might AI music-based systems offer to artists? Initially, I considered music a more restrictive field as it is sound-based (whereas contemporary art can be anything), but the focus on music performance has also produced a number of interactive systems, such as Rebecca Fiebrink’s Wekinator or Shimon the marimba robot.‍

MG: It’s quite easy to innovate quickly in the audio domain because these tools require less memory, storage, and bandwidth. The issue is that sometimes it’s easier to get a high-quality output using images rather than sounds. Getting an audio process to work well is very different to getting it to work well at reasonable quality. This is because the frame rate and resolution is tough for models to train on given their periodic nature. There are loads of audio models but, most often, they sound terrible because the noise in the higher frequency components is problematic to model. It’s sometimes easier for people to notice random variations in high-frequency audio than it is for them to notice similar kinds of variations in an image. 

You can compress an image quite aggressively and most people would think it’s fine. But if you compress an audio fragment in a similar way, more people would notice that it is noisy in the upper frequency bands. Lots of current AI and ML audio tools suffer from this problem. 

Lots of people now use the approach we proposed with our models back in 2017, where we turned the audio into a 2D spectrogram and worked as hard as possible to reduce the blurriness. That allowed us to reconstruct the signal from the spectrogram output, which is the approach we took for the Massive Attack Mezzanine remix (Mezzanine vs. MAGNet), which worked very well.  ‍

‍LE: Having consulted on the development of more than 40 prototypes, what would be your advice to creators who want to turn their artwork or research project into a commercial tool? Personally, I would tell them to partner with someone from the industry they’re trying to disrupt so that they can figure out the exact needs of that field. It would also be sensible to lower the expectations of their initial clients, since AI systems don’t always work as well as people imagine.‍

MG: You need to hit a sweet spot between what the technology affords, and what people want to do. That is really tough. There are also different kinds of founders, and I tend to focus on those at either end of the spectrum: those who have built a prototype themselves, and those who haven’t. Those who have built a prototype often know a great deal about what a given technology affords but have no idea what anybody else might want to do with it. While those who haven’t built a prototype might have a great idea for something they imagine meets or creates a need, which inspires them and a bunch of investors, but have no way of knowing whether it is even plausible. I come across both situations all the time. 

My main advice is: make sure you understand the technology you intend to innovate before you make any design decisions, and make sure you understand what people in the relevant domain might actually want to do with it. 

If you can’t do this, you are going to need a greater amount of luck or a greater amount of money, and you won’t be as successful as you would if you had nailed this stuff first. ‍

‍LE: Looking beyond art, what role can AI play for social good? You have received funding to transform heritage collections and augment immersive learning and well-being. How might AI be applied in those areas? For me, it’s been exciting to observe how museums are using AI to engage audiences and build new links between items in their collections. There is a lot of scope to apply AI in many such domains and I hope we can use these systems to make researchers and educators more effective.‍

MG: Over half of the funding we’ve received in the last five years has supported work on the ways AI and ML can play a role for social good. In all cases, the overarching philosophy of this work has been to put humans at the center of the machine learning and AI creation process, and not only its use. We are trying to understand how to create technology that people can use to make, edit, improve, and customize AI and ML systems. That is very different from creating AI that is somehow “aligned” with society, which seems like a slightly weird way of trying to make a positive impact. 

It’s clear that what technology companies who have invested in AI want is a good social outcome, which is great. One possible problem is that they don’t seem to care how they get it, which has led to some hilarious and messy economic consequences for companies like Google. It’s particularly ironic because they also fired staff members for attempting to warn them about precisely these kinds of issues. In general, and although I accept that alignment has its place, my view is that AI and ML customization is paramount if these technologies are to have a positive impact on society. 

Ordinary people need to be able to automate whatever they want without limitation. That’s what computers are supposed to make possible. People really want this kind of technology and AI and ML could do it very quickly. What is missing is the investment to make it happen.