Aim

1. To refine synthetic handwriting data using limited real data so as to improve word-level classification accuracy of a model trained on the synthetic data.
2. To study various GANS model on how they can be used for handwritten text generation.

Using DC Gan

Link to the tutorial => https://medium.com/@ekss1121/generative-adversarial-networks-b9f80e6d7679

Results(after 2400 epochs)

Loss Original Data Generated Data

Conclusion

1. Some letters were easy to generate such as B, e, U.
2. They do not provide desirable result on single letter generation. As the fonts are still blur and some are not recognizable.
3. We did not provide any image to the generator to create a synthetic image.
4. We were not able to set any condition on the generates images, such as font or which character to produce

Using Pix2Pix

Link => https://arxiv.org/pdf/1611.07004.pdf

Requirements

1. Put conditions on the output, hence use a input image to be converted from one font to another

2. Produce font conversion from one word to another

Hence we move to the pix2pix model which use a encoder-decoder network to get the input and produce output in different font

Why use gans(according to the paper)?

Minimizing euclidean distance between predicted and ground truth produces blurry images because it minimizes by averaging all plausible outputs, but we need sharp and realistic images.

Hence we use gans who's goal is to "make the output in distinguish-able from reality”(blurry images will be considered fake)

Architecture

Generator Discriminator

Results(64 epochs)

Input Output Expected Input Output Expected

Conclusion

1. Model learns a good representation of the desired font
2. It makes spelling errors. Not consistent with the input image
3. Sharp images are produced with no blurring.

About the architecture

• Generator uses U-net( like resnet) for efficient learning instead of simple encoder-decoder network
• Very deep U-net
• Paper proposes, using Patch Gan(where discriminator checks the NxN patches to predict whether the image is real or fake but currently we used the basic architecture

Using Cycle Gan

Link => https://arxiv.org/abs/1703.10593

Why use Cycle Gan?

1. While using pix2pix we encountered many spelling errors.
2. They do not require paired images as in case of pix2pix

What are CycleGan?

1. Cycle gan is a modification to the pix2pix architecture, generator and the discriminator use the same architecture.
2. Instead of training one GAN, we are training 2 GANS. First (F(x)) learns the mapping from X -> Y and the second (G(x)) learns the inverse mapping from Y->X. We want to make sure that G(F(x)) = x. This would mean that the model is cyclic consistent. Hence we have 2 GANs and a new loss function.

Cyclic consistency loss = Ex∼pdata(x) [F(G(x)) − x] +Ey∼pdata(y) [G(F(y)) − y].

3. Unlike pix2pix which required paired images, it does not require them but will use unpaired images of both sets(words of different font).

Network Model

Results

Real Font A Fake Font A Real Font B Fake Font B

Conclusion

1. Model learns a good representation of the desired font
2. Reduced spelling errors. Consistent with the input image
3. Capital Letters are not learned correctly even with increase in training time. Mostly because the data set has less capital letters.
4. We get font conversion and also its inverse font conversion.

About the architecture

• Very similar to pix2pix
• Loss now has 3 parts, 2 for GANS and 1 for cyclic consistency => L(G, F, DX, DY ) =LGAN(G, DY , X, Y ) + LGAN(F, DX, Y, X) + λLcyc(G, F)

TODO:

simGAN

New Results(40 epochs, 2 days)

More results can be found on this link -> https://web.iiit.ac.in/~shubh.maheshwari/test_12/index.html

We did the same experiment on Cycle Gans using a much larger data-set.

1. We took the training set of 6000, validation set 500.
2. Results were very similar to the previous experiments where we got good results for small words
3. But the model couldn't generate capital letters. I increased the the number of capital letters in the data-set but instead of learning, it either didn't write them or created straight lines, spelling errors etc.

One of the reasons can be =>

We also think that this model is not good fit to change the shape of object. We tried to run the model for converting a men's face to a look alike women's face. For that we used celebA dataset but the results are not good and images produced are quite distorted.

https://hardikbansal.github.io/CycleGANBlog/

4. The training data-set and test data-sets where all unique, no words were repeated.

Real Font A Fake Font A Real Font B Fake Font B

Capital Letter Conversion

To understand the issue of capital letter conversion we ran the same experiment but only for converting from one font to another.

Results are not as promising as the for small letters but its a start.

Results(Best)

more results => https://web.iiit.ac.in/~shubh.maheshwari/test_latest/index.html

Real Font A Fake Font A Real Font B Fake Font B

Using Bicycle-Gan

Why switch from cycle gans?

- It gave better results on capital letters.

- Can account for variance.

What is a Bicycle Gan

- It is based on the pix2pix architecture

- Similar to cycle gan which used the inverse relation to map back to the same image. Bicycle gan does that with the noise

- It uses an encoder for the noise used to generate the image and using KL divergence make sure the encoded noise was indeed the original noise used to generate the data.

Understanding Bicycle-Gan

We need a way to enforce the random noise to reduce the mode collapse on different z values

• Method 1

1. One way to do this is is by using z to generate B using A and then use an encoder to get z^ (z -> B -> z^).

2. Now using an L1 loss together with loss for G and D. Figure (d), we can enforce the noise.

• Method 2

1. We can reverse this by using E on B to produce Q(z/B) then using this and G generate B^. (B -> z -> B^)

2 . Authors noted that this method alone didn't provide the good results at test time. Hence they enforced KL-divergence on Q(z/B) and z(random Gaussian ) to produce much better results

3. Look at figure c for reference.

• Finally the authors combined both methods to create 2 cyclic constraints. Hence its called the Bicycle Gan

Results on Small & Capital letters

If the video doesn't load => https://drive.google.com/open?id=1KrjMRw4oQC1wnEEsFYzpj5DMf7_xmDh7

More results => https://web.iiit.ac.in/~shubh.maheshwari/test_bicycle

Result on Multiple Fonts( Very Similiar)

- We can see a case of mode collapse as their is very less variance in the generated images

Input Image Output Image Generated Image(Encoded) Generated Image(Random)

Understanding the data-set and checking for any bias

Report => https://web.iiit.ac.in/~shubh.maheshwari/Visualize_fonts.html

Currently I have just evaluated using PCA.

please recommend more evaluation methods for image comparisons

TODO

Link to the meeting -> https://drive.google.com/file/d/1KbcyuZCA9U-ZySEsXovowWJK1NSju5Jh/view

1. Training the cycle on a disjoint and a larger data-set

2 . Training on single to multiple fonts

3. Variable length

4. Class Information should be preserved (eg. spelling)

5. We need some kind of qualitative method to analyse how good the generated fonts are in general(not for a particular font)

1. We could try to train an ocr together with the gan for analysing how close is the generator to the required target space

2. We could use an already good trained OCR to predict the labels. Wrong labels and less confident labels would get a bad score

6. Analyse the training data, whether we have a bias during training

7. Use a trimmed images for font