# Which neural network topology is the most efficient to generate randomly shaped letters?

I have created some unique shapes, so-called "letters" for a custom alphabet, all of which can fit into 9x9 pixels. Instead of drawing countless more, I try to combine two solutions I saw in a relevant part of Reddit, and decided to let a neural network create some additional examples.

Desconstructing the problem: letters are formed in a graph of 25 nodes (always ordered into a 5x5 square) by spontaneously connecting only the adjacent nodes - no diagonal or non-adjacent edges are present.

For a neural network input, I drew these runes into 9x9 blocks, where each row has 5 "pixels" for each nodes, and 4 more as a place for indicating connection.

Below is the current letter set, with an example of an empty graph and the only example generated by my network in the last line.

I've made a perceptron (tried 1, 2 and even 3 hidden layers) where input layer had 6 neurons, using them as a binary code (zero values mean -1 activation, and one values are the 1 activation), andthe output layer had 81 neurons. (9x9 to plot out the desired shapes)

My aim was to be able to produce additional letters by defining letters as sample and making the network to learn it. Then, I assumed that by activating the network with undefined inputs, I can find new letter shapes.

I used built-in functions froM JavaNNS and backpropagation as learning method.

Which are the parts of my topology, where I could possibly go wrong? What is the most suitable solution for this task?

• I know nothing at all about neural nets, so it's perfectly possible that's why I don't understand your question. But I don't understand your question at all. It seems that you've generated some stuff using neural nets and you want to generate some more stuff. But you don't clearly say what you've already done, what you're trying to achieve, why your existing technology can't generate more stuff. Apologies if it's just me being dumb. – David Richerby Jul 8 '16 at 19:48
• @DavidRicherby I haven't generated nothing yet. I've made some monochrome pictures by hand (not literally, but rather a composition of 81 binary values), fourteen to be precise. I want them to be the output in a neural network, by learning - and based on them, I want to see other similar outputs. I'm unsure if I've chosen the proper input, the proper topology and the proper hidden layer size. – Zoltán Schmidt Jul 8 '16 at 19:54
• @D.W. I worried that if I'm not asking generally enough, my question would not add value to the site. If you think I definitely need to explain to be able to answer at all, I do it. – Zoltán Schmidt Jul 9 '16 at 14:49
• Thanks for the links. Following them leads to github.com/alexjc/neural-doodle, github.com/mxgmn/SynTex, nucl.ai/blog/neural-doodles -- those might be helpful. – D.W. Jul 10 '16 at 18:50
• @D.W. tried SynTex, and though it was useful, its functionality didn't precisely helped my purposes. For example, it couldn't "memorize" the fact that diagonal connections are not allowed - I wouldn't even expect it from a texture synthesis algorithm, but still, a bit different from what I wanted. – Zoltán Schmidt Jul 12 '16 at 13:46

I think it's unlikely you're going to get good results. Neural networks typically require a large training set. With only a few dozen or so examples, I doubt you'll have a large enough training set for this to work well.

So, what could possibly go wrong? I suspect the most likely outcome is that you'll get useless results.

That said, if you really insist on trying neural networks, you could try using a variational autoencoder. An autoencoder consists of two halves: the first half maps the input $x$ to a value $z$ in the "latent space", and the second half maps $z$ to an output $x'$. You train the autoencoder to ensure that (hopefully) $x$ and $x'$ are very similar, for all the instances $x$ in your training set. The idea is that you make the latent space much smaller than the space of inputs/outputs, and you hope this forces it to generalize.

For instance, the input might be a 40-vector $x \in \mathbb{R}^{40}$, encoded by a one-hot encoding ($x_i=1$ means that you include a black line in the $i$th position; $x_i=-1$ means you don't include it). The latent space might be $\mathbb{R}^2$ or $\mathbb{R}^3$ or so. You might include a few layers of a convolutional network to map from $x$ to $z$, and then the inverse architecture to map from $z$ to $x'$. Then, you could train the network using a standard loss function for variational autoencoders, along with some regularization term.

• "With only a few dozen or so examples, I doubt you'll have a large enough training set for this to work well." It does not matter if you have 10 or 1000 samples to train your Network with, look at OR, AND and XOR Neural Networks where each has only 4 training samples. What the network essentially needs is just a lot of training; The larger the training set the more time the network will take to get ready (final accurate weights), but with wider output classification. – Kyle Khalaf Jul 12 '16 at 19:30
• @FirstStep, You seem to have some misconceptions. The number of training samples does matter. The number needed will depend on the task. Just because some functions (e.g., XOR) can be learned with a few samples doesn't mean that all functions can be learned with a few samples. It just doesn't work that way. – D.W. Jul 12 '16 at 19:37
• I did not say that because XOR takes few samples, that means that is how Neural Network works. Of course it depends on the task, so no it is not "Neural networks typically require a large training set", it just depends on the problem. OCR needs 26? ODR needs only 10.. To classify objects in an image you will need all the objects in the universe.. and so on. (And I think you are up voting your own answers and comments) – Kyle Khalaf Jul 12 '16 at 19:38
• At this point, I don't know what point you are trying to make. I merely said that I doubt that a few training samples will be enough for this problem. Of course that's merely speculation. The only way to prove or disprove this definitively is to try it. Mentioning OR, AND, and XOR does not disprove it. You can't disprove a "typically" with one example; and you can't disprove a "for this example..." with a different example. You're welcome to write your own answer if you would answer the question differently. – D.W. Jul 12 '16 at 19:43
• I just like the subject :) – Kyle Khalaf Jul 12 '16 at 19:44