The notion of round should be quite similar to the intuitive idea of "one step of the protocol". At any round you are allowed to send 1 message (of any size) and/or receive one message. The message you send at round $i$ may depend on all the information you have up to round $i$, but it cannot depend on the message you receive at round $i$ (or later). It follows that any information that depends on the the message you receive at round $i$ may be sent only at round $\ge i+1$. This sequentiality is the core of splitting into rounds, they just formulate the causality of the protocol
Since the above doesn't determine the size of each message sent, one can define the communication complexity to be exactly the number of bits communicated throughout.
As for simultaneous transmission of messages:
In the above informal description I wrote:
At any round you are allowed to send 1 message (of any size) and/or receive one message.
note the and/or
part. If simultaneous transmission is allowed, then each party sends a message and receives one. If no simultaneous messages allowed, then the party either sends or receives a message, but not both. This makes the partitioning to rounds more crisp: every time the sender party changes, it is a new round.
finally, I repeat the recommendation of @hengxin, and refer you to the book of Kushilevitz and Nissan.