...while calculating time complexity and operations such as swap are ignored
You are wrong. All the operations are taken into account when calculating time complexity. But as loops are dominant compared to other operations, we ignore other operations and only consider dominant operations(Because for large input value, cost of all other operations are much smaller than the dominant operations) .
As an example with selection sort: When you consider all statement costs into account then you get a function $f(n)=an^2+bn+c$ ($a$,$b$ and $c$ are constants and depend on machine architecture). Here dominant term is $an^2$.So we can say Time complexity of selection sort $O(an^2)$.We also ignore leading terms coefficient $a$ ,as $a$ does not change the rate of growth.
I suggest you to study some standard algorithm textbook to understand it properly.