AI that can paint images

Image style transfer

image from paper [3]

Image style transfer is an approach that you can apply for image painting, where the goal is to extract the style from one image and apply it to a second image while preserving the content of this second image, just like it’s shown in the image above.

In order to achieve this, the style of an image needs to be defined.

In fact, the style of an image is very much related to its texture. So by using texture extraction methods, we are essentially extracting the style of an image.

These methods are called Visual Texture Modeling methods. There are 2 types of these methods :

  • Parametric Texture Modelling with Summary Statistics
  • Non-parametric Texture Modelling with Markov Random Fields

In order to apply this style on an image, we can use image reconstruction techniques. There are 2 types of these techniques as well:

  • Image-Optimisation-Based Online Image Reconstruction.
  • Model-Optimisation-Based Offline Image Reconstruction.

Convolutional neural networks can be used for both Visual Texture Modeling and Image Reconstruction.

Neural style transfer (NST)

One of the fairly simple approaches to do style transfer in images is by using the approach described in the paper “A Neural Algorithm of Artistic Style”.

What you need :

  • A CNN (VGG, Inception, …etc) that’s pretrained on a large dataset (ImageNet).
  • An image that you want to style but keep its content, let’s call it “content image”.
  • An image that you want to extract its style, let’s call it “style image”.

What to do:

  • Pass both images through the CNN.
  • Extract feature maps from both the content image and the style image.
  • To get the style from the style image, you need to use what’s called a “Gram matrix”. It’s a fancy way of saying : compute the means and correlations across the different feature maps.
  • So the feature maps will represent the content of an image and the Gram matrix will represent the style of an image.
  • Next, you need to optimize a loss function. The goal of this loss function is to make the style of the content image as close as possible to the style of the style image.
  • At the end you would have basically transferred the style of one image to another image while preserving the content of the latter one.​
image of a lion created using NST approach — made by the author

The image above I created using open source code from a tutorial provided by Tensorflow.

Btw, if you want to learn how to use Tensorflow then check out my free Tensorflow course that’s focused on Deep Learning for Computer Vision.

Real time neural style transfer

The previous approach gives some great results from an artistic perspective, but the downside is that it’s too long.

Every image has to go through a set of iterations where there is extraction of the content and the style of two images and then there is an optimization process to make one image’s style close to another one’s.

Because of this hurdle, another approach [2] has been proposed in the literature. With this method, there are two neural networks working together.

One network extracts the style from the style image. The other network has two inputs : the content image and the output of the previous network.

With this approach, the full system learns representations of painting styles from a large dataset during the training phase.

At test time, the network directly extracts the style from the style images and applies it to the content image. No optimization needed!

Conclusion

AI can be used for artistic styling. There are deep learning approaches that can transfer the style of one image to another image while preserving the content of the latter.

There are different approaches to achieve this, some of them require online optimization. Which means that whenever you want to style an image, there has to be an optimization routine, which can take some time.

Other approaches use offline optimization, which enables the neural style transfer system to learn how to extract the style of an image and apply it to a new image during a training process. At test time, the transfer of the style is fast.

Original Source

Author

Machine Learning Engineer at Orpalis | Founder of AIFEE