AI Upscalers

Introduction

Artificial Intelligence uses a network of simulated neurons (neural network) to augment the ability of a computer to perform tasks that rigid classical programming cannot easily achieve such as perception, interpretation, and prediction. In the case of upscaling images (sometimes referred to as super resolution), the algorithm is essentially tasked with creating extra pixel data to fill in the missing gaps between the known pixel values. Classical approaches to upscaling is called interpolation, which often simply fills in the missing pixels with color values that split the difference between their nearest neighboring pixels.

Of course, a human would be capable of perceiving the objects in the image, and having prior knowledge of the characteristics of these objects, be capable of predicting what the missing pixel values could be.

This is where artificial intelligence comes into play. Having been trained on a large volume of image sets containing high and low resolution pairs, it becomes capable of generalizing the process of upscaling similar images.

The downside is that it requires a LOT of examples and a massive amount of computational power in order to properly train an AI algorithm to do this task well. Furthermore, the trained model does very well on images that are similar to image pairs that it was trained on and does not do as well on images that are very different. In the case of upscaling retro graphics, this means that we must do due diligence to either use a model that was trained on images similar to our use-case, pre-process the images we wish to upscale so that it more closely resembles our chosen model's low resolution training inputs, or we train our own model to do the task. While it may require an extraordinary amount of computational power to train the model, it does not require too much to run the model. But please be advised that at the time of writing this, the only way to use AI on a personal computer in a reasonable amount of time is by using the GPU which is only available on fairly modern NVIDIA graphics cards which support CUDA.

Below is a list of currently available AI image enhancement algorithms with notes. ESRGAN and SFTGAN are a little tricky to get installed, so I have included a tutorial on how to install them on a Windows 10 environment.

  1. ESRGAN
    • An open source AI upscaler (super sampler) based on a generative adversarial network architecture.
    • Can be run in CPU mode, but requires a CUDA enabled Nvidia graphics card to get the most out of it.
    • Installation and Operation tutorials here.
  2. SFTGAN
    • An open source image enhancer that seeks to restore lost texture details from known types, made by the same team who developed ESRGAN.
    • Can be run in CPU mode, but requires a CUDA enabled Nvidia graphics card to get the most out of it.
    • Installation and Operation tutorial here.
  3. Gigapixel AI
    • Commercial super sampler that uses artificial intelligence and is made by Topaz Labs. It costs $100, but comes with a 30 day free trial.
    • Produces good results with little fuss. Easy to install, but also requires a CUDA enabled Nvidia graphics card.
  4. Sharpen AI
    • Commercial image enhancer that uses artificial intelligence and is made by Topaz Labs. It costs $80, but comes with a 30 day free trial.
    • The algorithm often compliments other super resolution results, effectively sharpening some blurriness and enhancing detail.
  5. Nvidia Gameworks
    • Apparently, it works quite well, but it requires a beta access pass which very few people were awarded.