Must you work with all qubits in a circuit when applying a gate?

When looking at this question: How to apply a 1-qubit gate to a single qubit from an entangled pair?

And other questions, it makes it seem like when you have a quantum circuit involving $N$ qubits, that when you want to peform a gate operation on a subset of the qubits, that you must still make a matrix that applies to all qubits.

Is this always true, or is it only true in the case where qubits are entangled?

In other words... is it only entangled qubits that we have to process together, or must all qubits in a circuit be processed together?

This can be formalized as evolving certain qubits via $U$ while the other qubits evolve via $I$, thus the joint unitary is $U \otimes I$.
• @AlanWolfe note that what you ask is still true: you have to construct a matrix which applies to all qubits, but, as Ran G points out, you do that by first making your matrix U, and then taking the tensor with the identity matrix. More generally, if you want to do a 3-qubit gate $U$ on qubits 3,4 and 5 out of 6 qubits, you take the matrix $I \otimes I \otimes U \otimes I$. You can construct matrices for arbitrary subsets (e.g. $U$ for qubits 1,2 and 5), but it's a little more work, and it probably warrants a question of its own. Oct 16 '15 at 21:28
I think you will need that in the case that your operation is not using qubits in sequence. That is if you have 4 qubits (1, 2, 3 and 4) and you want to apply a 2-qubit quantum operation on qubit 1 and 4, e.g. a CNOT then you should construct a matrix of size $2^4 \times 2^4$.