I was reading this paper when I stumbled upon what the authors call ``regularized UCB algorithm" (see Appendix A.1). In particular, they define the optimistic mean reward estimator to be $$\tilde{r}_t^\lambda(a) \triangleq \hat{\theta}_t^\lambda(a)+\sqrt{\frac{6 \sigma^2 \log (T)}{\lambda+N_a(t-1)}}+\frac{\lambda\left\|\theta^{\star}\right\|_{\infty}}{\lambda+N_a(t-1)},$$ where $\lambda>0$ is a regularization parameter, $\theta^\star:=\max_a\theta_a,$ $\theta_a$ is the mean reward of arm $a$, $T$ is the number of iterations, $N_a(t):=\sum_{\ell=1}^t\mathbf{1}(A_\ell=a),$ $\sigma^2$ is the sub-Gaussian parameter of the noise in the realized rewards (i.e., $R_a^t=\theta_a+\epsilon_a^t$) where $\epsilon_a^t$ are iid sub-Gaussian random variables with parameter $\sigma^2$, and $\hat{\theta}_t(a)={N_a(t-1)}^{-1}\sum_{\ell=1}^tR_a^\ell $.
The authors refer to Lattimore and Szepesvári (2020) as a citation for this version of the UCB algorithm, but I was not able to find it. Can someone point me to a reference or a similar version of the UCB algorithm?
Many thanks!
References:
- Lattimore, Tor and Csaba Szepesvári, Bandit Algorithms, 1 ed., Cambridge University Press, July 2020.
