Creative Artificial Intelligence: Painting and Video Games
A Series on Computational Creativity. Part Three.
This article is part III of my guide through the field of computational creativity for practitioners and executives in the creative industries. Here are parts I and II:
Today, I’ll provide an overview of the current state of computational creativity in two creative domains: the visual arts/painting and in game development. As I did in part two, I had to deviate from the intended format again (let’s call it a tradition!). I cut the part on advertising because the piece would have gotten way too long. I’ll post it tomorrow instead. So, we now officially have a six-part-series!
Visual Arts & Painting
Computers have long been used to create visual art and paintings. The latter is an interesting extra hurdle. Most creative domains I covered so far produce mainly digital output. Creating a painting, in contrast, involves applying color to a physical medium, usually a canvas. Thus, if a computer is asked to paint, it needs a way to interact with the physical world. It takes robotics.
The best-known painting program is likely AARON which was created by the late Harold Cohen. Cohen started developing it in 1973 and continuously worked on (and with) it for the next decades. In the process, he often altered AARON’s output methods which ranged from plotting to inkjet printing. As one of the discipline’s pioneers, Cohen can give some valuable insight into the early breakthroughs, as he did in this video interview:
“But when I started thinking about color and how a computer program might handle it and trying to do that in the classic AI terms of modeling my own behavior, I was up against a brick wall. I didn’t know how to proceed. And it was only finally when I realized — what seemed very obvious once I realized it — that machines and human beings a really quite different entities, that a program has one set of capabilities that don’t correspond at all to human beings’ capabilities, then I started to make progress.”
The following clip from a 1987 documentary illustrates the state and issues of the discipline back then:
Of course, progress has been made since then. The University of Konstanz, Germany, develops and operates a painting machine called e-David. As you can see in the video below, it’s a robot that uses paint and brushes. One of the system’s key features are its visual sensors. They are involved in the painting process and create a “visual control loop” that allows e-David to correct its own mistakes.
While a painting robot is impressive in its own right, the more fundamental progress has been made on the software side. As you might already expect, the recent progress in machine learning has also reached the visual arts.
One of the more astonishing publications of late was published by a team of researchers from Rutgers University and Facebook’s AI lab. They built an artificial intelligence system that aims to create pictures which qualify as having an original style. Their system consists of two neural networks. One is a generative algorithm that creates the pictures, the other one is a so-called discriminator that learned to categorize pictures by artist and style from a database of more than 80,000 pictures. While the first system creates new pictures, the second one is used to evaluate the output. The goal was to create pictures which don’t look quite like already known art while also not deviating too far from what qualifies as visually pleasing.
Here’s Dr. Ahmed Elgammal, one of the researchers, describing the process in a Medium post:
The first signal is the discriminator’s classification of “art or not art”. The second signal is a “style ambiguity” signal that measures how confused the discriminator is in trying to identify the style of the generated art as one of the known styles. The generator will use this signal to improve its ability to generate art that does not follow any of the established styles, and has an increased level of style ambiguity. On one hand it tries to fool the discriminator into thinking it is “art,” and on the other hand it tries to confuse the discriminator about the style of the work generated.
The authors call their new system a creative adversarial network (CAN) — a play on the generative adversarial network design which is the common name for such generative systems comprised of two competing neural networks. It is a technique of the unsupervised machine learning variety which uses unlabeled data. Instead of training the algorithm with data that humans described up front, this approach leaves it to the system to extrapolate its own categories from the raw data.
As I have explained in part one of this series: machine learning is about analyzing statistical properties and recognizing patterns. In the case of visual arts, the algorithm might, for instance, recognize certain properties of successful paintings that have been hidden to the human eye. The CAN system uses those insights to evaluate the generative algorithm’s output against the criteria it identified. With every iteration, the output gets “better” (or rather: closer to the optimum, as defined by the discriminating network). To some success, as Artnet reports:
The Abstract Expressionist works rated the highest, with 85 percent of respondents correctly identifying them as the work of a human artist. Users believed that 53 percent of the CAN images were made by people, as compared to only 35 percent of the GAN images, and, interestingly, 41 percent of the Art Basel works.
Where things get interesting, however, is when respondents were asked to rate how intentional, visually structured, communicative, and inspiring the images were. They “rated the images generated by [the computer] higher than those created by real artists, whether in the Abstract Expressionism set or in the Art Basel set.”
The quality of computer generated visual art is certainly increasing (even though “quality” is hard to reasonably quantify, particularly in the arts). Still, so far no computer is selling its art at top art auctions (though Google did this). Instead, artists like Sougwen Chung start to use/collaborate with artificial intelligence systems to create new works.
On the consumer product front, meanwhile, apps like Pikazo or Deepart.io allow users to modify pictures by applying artistic styles to them — basically Instagram filters on steroids — and to eventually print them. While that might result in some users hanging up the pictures they created this way, I haven’t found any evidence of art buyers reallocating their investments to computer generated/modified art at large.
Video Games
Video games are, in many regards, the opposite of paintings. They are entirely digital. They are commercial products. Efficient creation matters. In the case of art, the artist and the narrative which accompanies the artifact are just as important as the piece itself. That mostly isn’t the case with video games. While some game designers do have a loyal fan base, most people buy games for other reasons than its creator.
Video games, naturally, have been at the forefront of technological developments and advances in computing. Therefore, it doesn’t come as a surprise that artificial intelligence and particularly creative computing play a bigger role in the gaming industry today than in most other creative industries. Since this series is focused on creative AI, we’ll look at AI in the game design and development process (apart from that, AI also controls all non-player elements in a game and some AIs are trained in gaming environments).
Procedural generation is a computational creativity technique that game developers have employed since the 1980’s. The most common use case is the random creation of a game’s environment (e.g. maps, levels). The dungeon crawler Rogue and the space exploration game Elite are two popular titles that used procedural generation in the 80's. Diablo, Sim City, Civilization, and many others have also used the technique since.
Thanks to the progress in computer power (Moore’s Law), the scope of procedural generation projects grew, too. The most prominent recent example is No Man’s Sky. The game contains 18 quintillions (that’s 18 zeros) algorithmically generated planets for the players to explore. Creating such a universe by hand would have taken the work of many, many people for a pretty long time. Hello Games, the studio behind No Man’s Sky, employed a team of 13. Procedural generation is about efficiency. But it still takes some human intervention in order to achieve the desired results. An article from RollingStone illustrates the process:
Using procedural generation isn’t simply about offloading the creative process onto an algorithm — the real challenge is that it requires developers to teach an algorithm the difference between good and bad game design.
The true craft of making a game like No Man’s Sky was creating a system from which a variety of interesting results can emerge, but with no boring or obtuse results. For instance, Hello Games built a base template for sea creatures that could be seeded to produce an endless variety of results: a sleek shark-like species, or a whale-sized leviathan, or a bizarre aquatic nightmare straight out of an H.P. Lovecraft story. Everything from the size to the coloration to the presence of scales will be unique — even if you lived long enough to visit every habitable planet in the game, you would never see the exact same species of aquatic life twice.
While the result is a technologically impressive game, many reviewers and players weren’t satisfied with the actual gameplay. NMS is certainly an impressive sandbox — made by algorithms —but whether or not it’s a great game is up for debate.
AI is also being used in other areas of the game design process. For instance, it can help with the task of balancing a game. If you aren’t familiar with game design: All games rely on a variety of game mechanics, i.e. rules and variables. If you think of a traditional role playing game, characters often have certain attributes like strength, dexterity, or health. The combination of values determines how strong or weak a character is. Another example would be the particular movements which chess pieces are allowed to make. The sum of all game mechanics determines what strategies are going to be successful in a given game. Thus, they shape the overall experience. Balancing, then, is the process of getting the right mix of all values and parameters so the game plays as intended by the designer.
As modern games often have a fairly big amount of such variables, balancing can quickly become a complicated task. In a 2013 Interview, veteran game designer and founder of indie studio Mothership Entertainment Paul Tozour, explained how machine learning-based approaches can help:
You may not be able to write a fitness function to put an exact number on entertainment value, but if you can state your design goals clearly, then you can very often write a fitness function that measures whether some part of your game satisfies them.
And then it’s about selecting the right design goals in the first place to create the kind of entertainment experience you’re looking for, and determining which of those design goals are appropriate targets for any kind of AI-assisted tuning.
On its own, this isn’t creative AI yet. At least not in the sense that it autonomously generates something. Of course, though, people are working on algorithmically generated games as well. One of the most prolific figures in the field is Michael Cook, who currently is a senior research fellow at Falmouth University’s MetaMakers Institute. He also organizes the annual event PROCJAM — its tagline says it all: Make Something That Makes Something — and developed a program called ANGELINA that autonomously creates entire games, including the setting and the mechanics.
While the results are still rather crude games, the following excerpt from a read-worthy feature on the procedural generation scene nicely depicts the motivation behind the efforts — and their potential:
“AI will invent genres that humans could never have possibly conceived of, I believe,” says Cook. “One day people will steal ideas from software, not because they want the fame or the pride, but because it’s the current mobile trend and it’s too good not to steal. It’s a cynical and sad aspect of the future, perhaps, but also I like to think it would be a moment of huge validation for AI.”
In the meantime, AI’s benefits will be more subtle.
“Computers are quite good at considering options equally,” explains Cook. “They can’t forget things, they don’t get tired, they don’t get confused by emotional needs.” People, however, are prone to fatigue and conscious or unconscious biases. An AI can show them something that they stopped considering as an option earlier in the process.
The catch, however, is that the two need to work together for best results. AI and human, collaborating.
As far as I can tell, the last sentence appears to be part of an emerging pattern. (Beware though, I arrived at this conclusion using only my wetware.)
That’s it for part three. Next time, we’ll look at the current state-of-the-AI in advertising. Make sure to subscribe to the newsletter below, so you don’t miss it.
