A system is several things working together. A real time system might include computer hardware and software. Depending on where you draw the line it might include sensors and actuators reaching into the "real" world (sorry for the pun ). The software might include an operating system, or not.
A realtime operating system is a set of software components with the purpose of making it easier for you as a programmer to write the application software of a real time system. Typically the OS helps in handling hardware and dividing the processor among several tasks. It could include tools used to create the system, to find bugs and to measure performance. This is a kind of software reuse, and I tend to think of it as "me beeing a midget, standing on the shoulders of the titans previous to me".
A real time system can loosely be defined by the statement "even if the response is correct, if it is too late it is as bad as if it was wrong". The concept of too late is the major concept in a real time system. This could be on a time scale of pico- or microseconds (sort of difficult with software), milliseconds or seconds (typical realtime areas) or even minutes, hours, days or years (probably not called realtime, but could be).
Often a real time system includes some mechanism to handle serious errors. The concept of fail-safe is often used. As example, even if the software stops totally, or part of the hardware becomes unserviceable, some hardware function in the system turns outputs to a safe state and maybe restarts the software. Another exemple might be that cars with computer controlled ignition and fuel injection has a special "limp-home" mode allowing you to do something useable even when some of the sensors are broken. Here the computer might be working, but not all parts of the system.
When human life is at risk, say in space exploration, the fail-safe solution might include running several computers in parallell and turning off suspicious ones using external hardware. The space shuttle is a famous example with several computers running in parallell and special hardware selecting to kick out erraneous ones. In addition there was a special fallback computer, using different hardware and software written by a different team to handle if the main computers would go totally off.
