Inside ‘The Next Rembrandt’: How JWT Got a Computer to Paint Like the Old Master

The project leaders explain their brilliant, troubling masterpiece

CANNES, France—Rembrandt van Rijn finished his last painting in 1669, the year he died. So it was enthralling, and a little unsettling, to step on to a boat at the Cannes Lions festival for a private viewing of the first new Rembrandt in 347 years.

In a fascinating merging of creativity and technology, the humans at J. Walter Thompson Amsterdam taught a computer to paint like Rembrandt by having it study the old master's works for months. The resulting painting is a completely new portrait, not a replica, and it's indistinguishable—to my eye, at least—from the real thing.

The project, created for banking client ING, won 16 Lions at Cannes last week, including two Grand Prix (in Cyber and Creative Data) and a coveted Innovation Lion. It's a piece of marvelous technical and philosophical complexity. And since its official unveiling in Amsterdam earlier this year, it's become a controversial flash point between the worlds of technology and creativity—raising uncomfortable questions about the future of artificial intelligence and art.

We've come to accept that computers are, in some ways, smarter than humans, or at least more powerfully logical. (They can beat us at chess and Jeopardy, after all.) We have a harder time, especially those of us in the creative industries, entertaining the question of whether machines could ever be as creative as humans. Creativity is supposed to be our exclusive province, the spark that makes us special, the thing computers could never dream of mastering.

"The Next Rembrandt" questions that, much to the glee of many technologists and the consternation of many art historians.

The Idea

In Cannes, I learned the backstory of the painting while visiting it on the boat. It began in October 2014 when JWT Amsterdam executive creative director Bas Korsten got an interesting brief from ING. The bank always thought of itself as innovative, and it worried that its 10-year sponsorship of Dutch arts and culture was feeling stuffy.

" 'It's not the most innovative domain. Can you bring some of our innovative spirit to it?' That was the brief," Korsten tells me as we sit on the upper deck of the Santa Maria in the marina next to the Palais—steps away from the artwork itself, at the moment covered up, that would result from the ensuing 18-month project.

After getting the brief, Korsten brainstormed ideas that might connect art and innovation. He thought at first about having a robot arm try to paint a painting. That turned out to be a dead end, as robotics just aren't advanced enough yet.

But then one day, he had a breakthrough.

"All of a sudden I saw the face of Jesus," he said. "Not in some religious epiphany, but in a magazine. It was an article about how an archeologist in Jerusalem had reconstructed Jesus' face out of skulls they had found around Jerusalem. So I thought, if you can create something new out of historical material, why can't we take all the paintings from an old master and recreate something new out of that? That's how the idea was born."

Specifically, Korsten wondered if a computer, if fed enough data about Rembrandt's paintings, could learn to paint like him. It was a wild, ambitious idea. And the technologists at the agency, including head of technology Emmanuel Flores, didn't know if it was possible.

JWT's Bas Korsten and Emmanuel Flores

"When I mentioned it to Emmanuel, there wasn't as much color in his face as there is right now. He was quite white," says Korsten, motioning across the table to his co-conspirator, who indeed looks quite happily flushed here in the Cannes harbor with one of the festival's great pieces to show off to visitors.

They knew they couldn't do it alone, however. "We needed experts to pull this off," says Korsten. So they assembled quite the team—art historians, material researchers, data scientists and engineers among them. The group included supporting partner Microsoft and advisers from TU Delft, The Mauritshuis and The Rembrandt House Museum.

Data Collection and Painting

The project website explains this part in great detail.

To teach Rembrandt's style to the computer, the team gathered enormous amounts of data about his paintings—the geometries, the composition patterns, even the height of the brush strokes off the canvas—and fed it into the machine. This gathering process took months and involved getting as much visual information about the originals as possible, and then resizing them to match each other.

Then they wrote deep learning algorithms and used facial recognition techniques to get the computer to recognize all the different patterns—everything that goes into a painting—and be able to create a similar one on its own using the fragments but without taking an average, which would be the easy way.

"We took the features out of the paintings, and we started to generate new features from that. We were not averaging," says Flores. "The system was classifying the images and the features based on recurrence and relevance. What are the most dominant patterns that make a Rembrandt look like a Rembrandt?"

The team had the computer generate a portrait of a male between 30 and 40 years old, looking to the right. The painter's real-life subjects were fairly evenly divided between men and women; the team chose a man because the source data of the paintings with men was of higher quality overall than those with women.

Through the process of gathering the data, the staggering physical complexities that actually make up a painting became apparent very quickly—the geometries, the brush strokes, even the canvas itself, which affects the physicality of a painting. The team also 3-D scanned the original Rembrandts to understand their topography—the height of the brush strokes off the canvas.

"Brush strokes means the layering of material. Geometry means how it looks," says Flores. "We needed to split all these components into smaller processes, because it was way too complex."

Finally, the computer finished its work. The resulting painting was 3-D printed in some 14 layers. It consists of over 148 million pixels based on 168,263 painting fragments from all 346 of Rembrandt's paintings. The result was something uncanny: a work of art that looked remarkably familiar, closer to a typical Rembrandt than the creatives had likely ever dreamed—good enough to thrill the public, and to get the technologists and art historians talking, too.

What It All Means

Computers tend to do what they're told. This can be reassuring to anyone who's ever watched a dystopian science fiction movie or two. But teaching computers to be more flexible in their thinking could open up much richer applications for their brain power—in essence, giving them street smarts as well as book smarts.

Another Grand Prix winner at Cannes this year dealt with this very issue. Google DeepMind's "AlphaGo" project, which eclipsed "The Next Rembrandt" and seven other Innovation Lion winners to nab the Innovation Grand Prix, attempts to teach computers to move past logic and into something akin to intuition through gameplay.

Despite their parallels, "The Next Rembrandt" isn't quite that ambitious. At its core, it is an elaborate, unusually beautiful and thought-provoking data visualization project. The machine learning it employs is existing technology, Digital Craft jury president Wesley Ter Haar of MediaMonks tells me, and that jury decided that "The Next Rembrandt" represented a use of the technology and not an evolution of it. (The other criticism of the piece at Cannes was that the connection to the brand was weak, which is a valid point.) 

Still, the fact that they awarded it a Grand Prix anyway shows its raw power—"the emotional value it has," as Ter Haar puts it. "Everybody reacts to it." 

Even if "The Next Rembrandt" is a long way from a computer exhibiting true creativity, the machine did surprise its masters on one magical morning during the long months of testing.

The team had been running test prints overnight and looking at the results the next day. "One morning, one of my developers said, 'You have to come and see something very, very quickly,' " Flores says. "I looked and I saw wrinkles [around the eye of the man in the painting]. I thought, 'Where is this coming from? Is this a glitch?' And he was like, 'No, no, they're wrinkles!' The facial hair and the wrinkles emerged without us expecting it. I didn't expect to see such mastery or such awareness of how to make craft with variations that are magical to the eye of the viewer."

So, how believable is it as a Rembrandt?

Korsten, Flores and I walk over to it on the boat. The draped covering is taken off. And it hits you pretty immediately. This is a completely exquisite work, and a remarkable thing to gaze at. As with any painting, the details are much more vivid in person than in photos. The brush strokes look like brush strokes. The colors and shadowing are magnificent.

Most of all—and perhaps most disconcertingly—the man looking out at you from the canvas just feels real. For a massive data project to visualize something that feels so human is extraordinary indeed.

"You don't expect something actually happening behind those eyes. That's pretty scary," Korsten admits. "We're really happy with it. I think the layman would look at it in a museum and say, 'Hey, that's one of those from the commissioned portraits series.' "

Actual Rembrandt scholars, however, are not as uniformly impressed.

"We had one expert who was really negative [about certain features and colors]," Korsten admits. "But he treated it as though it were an actual painting, which is good. One of the other experts said this is such a valuable tool in the toolbox of Rembrandt experts, because it teaches us about what he is."

What's Next?

After the unveiling in April, the painting set off a heated global discussion.

"People were asking, 'Is this good?' 'What's the relationship between artificial intelligence and creativity?' 'Is creativity flashes of genius that are reserved for mankind?' " says Korsten. "Those questions have gotten a lot of attention from people in the art and tech worlds, who are actually quite opposed in their reactions."

"I've been living with this painting for a long time, and it just generates discussion, even about taboos," adds Flores. "Who owns this? Who made it?"

Does the agency team have any answers in that regard?

"Personally I think humans are still special. I like to think we're special," says Korsten. "These algorithms have to be steered in the right direction. We're like a father teaching a kid how to write—you still need the father. In teaching a computer these algorithms about what is Rembrandt and what isn't, they still needed us."

Also, Korsten thinks one human in particular would find the whole thing a bit farcical.

"I think Rembrandt would laugh himself silly," Korsten says, "if he saw there was a team of 20 people, really clever people, working for 18 months and this is what they come up with." 

CREDITS
Client: ING
Campaign: The Next Rembrandt

ING
Director of Communications: Johan van der Zanden
Head of Sponsoring: Tjitske Benedictus
Teamlead Internal & External Communication: Marc Smulders
Sr. Marketing Communications Manager: Mirjam Smit
Sponsor Manager Culture: Eline Overkleeft
Event Manager: Marleen Hasselo
Social Media Specialist: Thijs Jaski

Microsoft
Director Small And Midmarket Solutions: Ron Augustus
Microsoft Azure Lead: Erik-­‐Jan van Vuuren
Product Marketing Manager: Niels Lohuis
Corporate Communciation Manager: Yvette Lansbergen
Marcom Manager: Eva de Vries
Solution Architect: Thijs Jaski

J. Walter Thompson Amsterdam
Executive Creative Director: Bas Korsten
Concept: Bas Korsten, Robert Nelk, Mark Peeters
Creative Art: Guney Soykan
Creative Copy: Bas Korsten, Kasia Haupt
Head of Technology: Emanuel Flores
Design: Vinesh Gayadin
Project Director: Jesse Houweling
Strategy: Agustin Soriano
Developer: Morris Franken, Ben Haanstra
3D Artist: Andre Ferwerda
Editor: Tim Arnold
PR Director: Jessica Hartley
Brand Manager: Elisah Boektje
Screen Producer: Frederique van der Hoeven, Mariska Fransen
Print Producer: Chariva Geurtsen

Animations: Kreukvrij (Olaf Gremie)
Website Production: Superhero Cheesecake
Special Advisor: David Navarro, Jeroen Van Der Most, Ferran Lopez

Film Production: New Amsterdam Film Company
Director: Juliette Stevens
Executive Producer: Sander Verdonk

Sound Studio: Studio Alfred Klaassen