What algorithm to use for training combinations

Question

I would be very glad if someone could help me with my machine learning task. I have palettes of 5 colors each (in RGB format), and would like to train the neural network so that I can input a color, and the model would give me 4 other colors that would go best with it.

My training set consists of 1500 palettes (1500 * 5 = 7500 examples - each color from palettes is combined with four other). I've trained the network with multi-label classifier: one color as input (3 variables RGB) and one-hot encoded vector as output (I've predefined 68 classes, so it looks like [1 0 0 ... 1 1 1]). So basically, I have two questions:

1) Very strange, but when I try to predict colors that go best with my color, I always get the same colors, just with different probabilities (with sigmoid activation function). Why could this possibly happen? Why my probabilities never reach over 30%?

2) This still doesn't solve my problem - I get only suggestions on different colors, but these colors may not match between them. I need an algorithm that would take into account all 5 colors at once. What kind of algorithm could possibly solve this problem? Maybe combine several?

Thank you for your time reading this!

Answer 1

Your current setup might not be the best one for this. You need to think about the structure of the model, and the set of features you use.

Model

I would suggest a different approach: see if you can construct a model that captures how well a set of 5 colors match. In particular, perhaps you can fit a model $f$, where $f(c_1,\dots,c_5)$ describes how well the five colors $c_1,\dots,c_5$ match each other (larger values of $f$ correspond to a better match). For instance, one plausible model for $f$ is

$$f(c_1,\dots,c_5) = \prod_{i<j} g(c_i,c_j)$$

where $g$ is some function (that you try to fit to the data). Then, given a training set that describes many different combinations of five colors and how well they match, you could try to fit a model to that (e.g., find $g$ such that $f$ matches the training set as closely as possible). You can then consider some architecture for $g$, e.g., a small neural network with a few hidden layers, and use gradient descent to train the model.

Features

Constructing a model based solely on the RGB values might be a lot to ask. I suspect you can do a lot better by choosing some features that are likely to be relevant to color matching.

Often it helps to have some domain knowledge. Do you know anything about color matching? Are there any properties of colors that are relevant or informative or affect whether they match?

I would suggest that you pick multiple plausible features, then let the model training process work out which ones are most relevant. Here are some ideas for you to consider:

• Consider adding other color space as additional features. I would suggest you consider adding the CMYK, HSV, and CIELAB.

• It appears that primary colors are best matched with primary colors, and secondary colors best matched with secondary colors, etc. Therefore, when determining how well two colors $c_i,c_j$ match, perhaps the difference between their hues (modulo 360 degrees) might be relevant. You could add this as an additional feature that is input to $g$.

Maybe your study of the art of what colors match will suggest other features you can try as well.

Answer 2

So far, I have come to the following conclusions:

1) It is not correct to mix all colors with all, usually palettes are compiled the way that only neighbor colors match really good.

2) I have found a similar problem which is solved here http://colormind.io/. They use Conditional GANs. As input, they use pairs of two images (color palettes are treated as images with colors order preserved) - one full palette, and another with one or more colors filled into palette. The goal of the network is to learn to generate palettes from the second partial palette. I would suggest looking into this project https://phillipi.github.io/pix2pix/