One of the key uses of state transition systems, also known as labelled transition systems, is for modelling concurrent systems.
One very nice, delightful even, book that uses labelled transition systems to give semantics to concurrent formalisms CCS and the $\pi$-calculus is
Communicating and Mobile Systems: the Pi-Calculus by Robin Milner. Definitely worth reading.
Principles of Model Checking by Christel Baier and Joost-Pieter Katoen is about model checking concurrent systems, so early in the book labelled transition systems are introduced to give semantics to such systems. This book also talks about probabilistic labelled transitions too.
Another possibility is Concurrency: State Models & Java Programs by Jeff Magee & Jeff Kramer. This book takes a more practical approach, but comes with an analyzer for systems described using labelled transition systems.
I don't know much about continuous dynamic systems. Perhaps you want to look at hybrid systems?