I recently faced this problem in CLRS ed.4 and couldn't find out how to attack it and solve it. Here's the recurrence: $$T(n)=3T(\frac{n}{3})+8T(\frac{n}{4})+\frac{n^2}{\log{n}}$$
Here's what I tried: $$\frac{3}{3^p}+\frac{8}{4^p}=1$$ For $p=1$, we get $\frac{3}{3}+\frac{8}{4}>1$ and for $p=2$, we get $\frac{3}{3^2}+\frac{8}{4^2}<1$. So we conclude that $1<p<2$. Using numerical methods, we get $p\approx1.85674$. Then we can write: $$T(n)=\Theta\left(n^p(1+\int_1^n{\frac{x^2}{x^{p+1}\log{x}}}\,\mathrm{d}x)\right)=\Theta\left(n^p+n^p\int_1^n{\frac{1}{x^{p-1}\log{x}}}\,\mathrm{d}x\right)$$
Here's where I'm stuck. I know $1<p<2$ but I can't figure out or guess the intergral answer. I suspect we don't need to calculate the integral actually, but then how can we solve the recurrence then?
Any hint or help is so much appreciated!