Generalizing texture transformers for super-resolution and inpainting

Abstract

The new multi-camera smartphones and recent advancements in generalized Machine Learning models make it possible to bring new types of photo editing neural networks to the market. This thesis covers methods of image enhancement with texture transfer. The known high-resolution regions (reference) can be utilized to restore degraded areas of an image. The task of restoring partially degraded images can be defined “partial super-resolution.” The task of restoring missing parts of images is called inpainting. We propose to use the novel Texture Transformer Network for Image Super-Resolution (TTSR) to solve the partial super-resolution and inpainting tasks. The fully convolutional networks are unable to copy image patches. This inability forces the model to store textures using the train weights. The usage of the attention mechanism allows taking advantage of joint feature learning in low-resolution and high-resolution parts of images simultaneously, in which deep feature correspondences can be discovered by attention. This approach exhibits an accurate transfer of texture features. The experiments confirm that the TTSR network can be used to solve the partial super-resolution and inpainting tasks simultaneously. Modifications of the network (different embedding sizes, soft-attention, trainable projections) study the architecture capacity to solve the specified tasks. The evaluation of results includes comparing the TTSR network with an inpainting network for the inpainting task.