Policy dependent on initial state distribution in finite horizon MDPs

Consider an MDP defined as the tuple $$\langle S,A,R,P,\mu,\lambda\rangle$$ where $$S$$ is the state space, $$A$$ the action space, $$R:S\times A\times S\to\mathbb{R}$$ the reward function, $$P$$ the transition probabilities, $$\mu$$ the initial state distribution, and $$\lambda$$ is the discount factor.

Let the decision horizon be finite. The expected discounted return under a given policy $$\pi = (\pi_1,\ldots,\pi_{T-1})$$, where each $$\pi_t:S\to A$$, is defined as \begin{align*} F(\pi,\mu) = E_{\pi,\mu}\big[\sum_{t=1}^{T-1}\lambda^tR(s_t,a_t,s_{t+1}) + \lambda^TR(s_{T-1},s_T)\mid s_1\sim \mu\big] \end{align*} where $$a_t = \pi_t(s_t)$$ and the expectation is taken with respect to the probability distribution on the states induced by the initial state distribution $$\mu$$ and the policy $$\pi$$. An optimal policy is defined as $$\pi^*\in\arg \max_\pi F(\pi,\mu)$$.

It seems to me that the math says that since we are only maximizing over $$\pi$$, there is still a residual dependence of the optimal policy on $$\mu$$, i.e., mathematically it should be written: $$\pi_{\mu}^*\in\arg \max_\pi F(\pi,\mu)$$.

Question: Are optimal policies in MDPs dependent upon the initial state distribution? If not, what conditions allow us to rule out this dependence?

• I just would like to note that there's also Artificial Intelligence SE, where there are a lot of RL enthusiasts. – nbro Sep 15 '20 at 10:56