If I have (I-Z) where I is a 3x3 identity matrix while Z is a 3x3 lower triangular matrix, how many subtractions that I should count from this process? Is it costs K subtractions or (K^2+K)/2 subtractions?
It’s a 3x3 matrix, so it can be done in constant time O(1). I can’t see where a “k” comes into this problem.
Since you are subtracting $Z$ from $I$, you must compute the lower part of the matrix. If it was reversed, you only had to compute the new diagonal. The total number of subtractions is therefore $(k^2 + k)/2$.