I have benn wondering about the following possibility: if I have multiple hardware threads on the same core, and then someone decides to use shared register to allow fast synchronization among threads that run on the CPU, and implements test-and-set on the register instead of doing it in main memory. Will this work? and if it works, why?
Synchronization based on test-and-set requires a shared data storage location that all the involved threads can read and modify. Your shared register is a shared data storage location.
The reason synchronizing on a register when the threads are running on different CPUs is that the register would not be shared (not be visible) to one of the threads. You should think of registers as memory locations with some features removed to make them faster.
In fact, something similar to your scenario was possible on SPARC processors. The SPARC had a large register file (something like 64 or 128 registers) with 8 registers that were always visible and then 24 register "windows" into the rest of the register file. Usually these windows were used to implement fast stack frames, but there were several experimental runtime systems that used the windows to support multiple threads.