Style transfer can be achieved by extracting features and styles from layers of a Convolutional Neural Network (CNN). To generate the synthesized image, we start with a white noise image and perform gradient descent to find an image that best represents the features and style of the desired image. The key algorithms involved are called content loss, gram matrix, and style loss.

The CNN model used here is a pre-trained version of SqueezeNet.

Figure 3: Feature Map Demonstration [3]

Convolution is performed in every layer of a CNN. Small image features such as edges are in the first few layers, while global features such as object parts are in the deeper layers.  

If we strategically choose the feature map layer, we can obtain information about the subject/content in the image.

If we are given 2 feature maps, one from the content image, and the other from the generated image, we can then use the equation on the right to calculate its content loss.

This equation resembles a Mean Square Error (MSE) function that calculates the squared ℓ2 distance between two convolutional feature maps.

If we then perform the gradient descent to minimize the content loss, we can achieve the goal of preserving the maximum amount of content from the input image. 

Figure 4: Gram Matrix Visualized [4]

We can also analyze the style of an image using the feature maps of a CNN. Think of the "texture" of an image as how often some specific colors and edges appear together. 

How can we find correlations? Well, as we learned in class, the dot product can find similarities between the two vectors. If two features often appear together, then their dot product should be a relatively big number.

We can take a similar approach to analyze a matrix, the gram matrix. Again, we start by extracting the feature maps of the desired image from the CNN. Since the feature maps have three dimensions (height, width, and channel), we can simplify later operations by flattening one of its dimensions first. Then by performing matrix multiplication with its transpose to measure the similarity between the rows of the flattened feature map.

The style loss is very similar to the content loss in ways that it uses Mean Square Error to calculate the distance between the generated style image with the input style image. This time the difference measured is the style image's gram matrix and the generated image's gram matrix. We can also add a weight parameter to help us easily tune the amount of style we want to preserve in the final generated image.

We can also perform gradient descent to preserve the most style.