Supervised learning is: given $X$ and $Y$, we want to find a good predictive function $f$ such that $f(X)$ is "close" to $Y$. Examples include XGBoost (ensemble of decision trees), neural networks, $\ell_1$-regularized linear fit, etc.
In the problem I am encountering I have $X$. Each row of $X$ describes each day's scenario, for example, the financial market descriptors of that day. I don't have $Y$. $Y$ is the decision for each day, for example, whether I should buy/hold SP500. For each $Y$, I will have some scoring, say
score(Y) system to describe whether my decision is good over a long period of time. Now, I want to find simple, robust $f$ such that $f(X)$ gets the low score. Of course, for arbitrarily complex $f(X)$, the score
score(f(X)) can be very low but the higher complexity generally means worse model for the problem I am tackling on. So there should be some regularization parameter that I could tune to control the complexity of $f$. The $f$ can be anything, like supervised learning. It could be ensemble of decision trees (XGBoost, random forest, etc...), linear (L1-regularized fitting etc.). I am currently essentially hard coding my problem into a optimization problem to solve on. But I think there should be a field out there tackling such problems.
If so, what is this type of problems generally called. And what's the state of the art packages currently in python. Thank you.