However, it seems like intuitively a compiler should just be able to compile down to instructions that say "allocate N bits of space" and similar for other related instructions rather than "give me 32-bits of space" or "give me 64 bits of space"?
Only if those instructions exist, and that's the decision of the instruction set designers rather than the compiler designers. Besides, the width of the instructions varies between instruction sets, as do things like the number of registers (very important for an optimising compiler).
In principle it would be possible to write some clever code which executes correctly in two different instruction sets and jumps to different places. You could then use this as a kind of "loader", placing it at the start of the executable to pack a 32-bit and a 64-bit version into the same file. However, then you run up against a problem which is not easily worked around, which may be considered the "real" answer, although perhaps not at the level of abstraction you want.
Executables have headers with metadata, which includes a flag for the instruction set. For example, the Windows Portable Executable format has a COFF header which requires you to choose one of a list of machine types:
I suppose you might be able to get away with using a clever loader if you choose machine type
0x0000, but I wouldn't count on Windows agreeing to execute your program. And technically you really ought to implement support for every CPU on the list if you do that.