Would it be possible to form such a group using the ADD
instruction and the NOT
instruction?
Sure. The integers modulo $2^{32}$ form a group. The group operation is addition modulo $2^{32}$, which can be implemented by the ADD instruction. You don't need the NOT instruction.
There are other groups you could form, such as the integers modulo 2, and many more. I recommend you read the definition of a group and play around with some examples.