Louvre’s Mona Lisa (C2RMF scan) split-screen with my neural-net restoration. (More youthful based on Islesworth painting. No veil, restored eyebrows, longer more natural curls)

Mona Lisa Machinima

Restoring Mona Lisa with Neural Networks

I combined three paintings of Lisa Gherardini, one famous (Louvre Mona Lisa) and two lesser known (Prado Mona Lisa and Isleworth Mona Lisa) to create a photorealistic synthetic composite.

Final composite. Creative Commons BY-NC-SA-4.0. Prints/Licensing available.

My obsession with Leonardo da Vinci’s painting goes backs six years to when I did a digital colour restoration of Louvre painting based on the colour-preserved and newly (2012) authenticated ‘Prado’ version made concurrent by a pupil of DaVinci. The result was a blurry and unfamiliarly saturated version of the green-yellow, varnished, eyebrow-less painting people expect.

One reason it isn’t believed to be a copy is the perspective shift. The perspective difference is consistent with a pupil sitting to left (and maybe back somewhat) of his master.

Mona Lisa Had Eyebrows

Artbreeder, the neural-net tool I’ve been using s deceptively simple but difficult to predict. In May 2020, I made a first attempt at a GAN/ neural-net assisted restoration. It wasn’t very good. In July, Bas Uterwick, another AI Portrait creator, did an excellent and more careful enhancement. (Sometimes media confuses us. We both have Dutch names and have a background in computer graphics and photography. 🤷)

For my second attempt, with more attention to detail, I again used Artbreeder but also Photoshop for refinement. The finished face proportions and expression is based entirely on the Louvre painting. The lighting, hair and youthfulness is a result of blending aspects of the Isleworth and Prado versions.

The background is a mixture of Prado and Louvre versions interpreted through Artbreeder’s new (v3) landscape creator which allows you to upload your own landscape for remixing. A fair compromise since da Vinci’s landscape is imagine and not based on any single known location.

Each painting was individually restored. Da Vinci’s Louvre and Isleworth were restored with assistance of neural net. Prado (middle) was just Photoshop. The background was imagined to begin with and the AI took many liberties in trying to interpret it.

Too add extra photorealism, I mixed in several high-resolution photographs: images of two sisters who claim to be descendants of Lisa Gherardini and several portraits of the actress Taissa Farmiga who, according to a celebrity facial recognition search, was a close match. (The accuracy of these faces are secondary to the ease and quantity of photographs available.)

Actress: Taissa Farmiga, who according to StarByFace, was a 40% face match to my partially completed restoration.

Is this a more accurate depiction of the real Lisa Gherardini? That depends if Leonardo was himself aiming for photorealism. The similarities and age progression and between the Isleworth and Louvre versions lead me to believe Leonardo was trying to depict a real person. The difference in the Prado version I think is attributable to a lack of skill of the pupil.

Interestingly, Mona Lisa’s smile was one of several aspects of this painting the neural net couldn’t capture. The combination of pursed lips and asymmetrical smirk is something outside this neural net parameters. I suspect this is wholesale creation of da Vinci’s brush. (Something I had to modify substantially Photoshop to match)

ABOUT THE AUTHOR

Daniel Voshart works in the Art Department of Star Trek: Discovery as a Virtual Reality Specialist. In his spare time he writes and does forensic related work and research. Find him on Twitter and IG.