8
$\begingroup$

How is video playback done on a computer? It's obviously not relying purely on the CPU, since video playback continues when a user performs another activity, such as typing into a YouTube comment field. This task seems further complicated in the case of streaming video, where data presumably needs to be transferred from the network interface to the video/graphics controller.

EDIT: I should have made clear that I understand that the CPU switches among multiple processes and threads all of the time. My question was meant to be more specifically about video playback. Is the video processing done on the graphics chipset/card? Do those typically offer buffering? Or can it all be handled by a single-core CPU with time to spare for other tasks, or am I wrong to think that low-end CPUs can play back video without (not-network) delays?

$\endgroup$
  • 2
    $\begingroup$ Any core can just execute one operation at any given discrete time step - but since it's really fast, it can switch between doing things so quickly that we can't detect it. If we could hear or see in time steps in the range of nanoseconds, it would look/sound like it's stuttering. Scheduling would be the rough topic for this, I assume; the trick is probably to schedule time sensitive things (like playing media or changing the screen content) frequently enough so our brain can't figure out that something else was done in between. (Note: the above is rough guesswork, but that's where I'd look.) $\endgroup$ – G. Bach Sep 4 '14 at 20:29
  • 1
    $\begingroup$ Add to the previous comment that modern chips have several cores (read: several CPUs), and that there are other processors to share the workload, such as the graphic processor. $\endgroup$ – babou Sep 4 '14 at 22:12
  • $\begingroup$ Thanks, but I thought about multiple cores and dismissed it, since it seemed like even old low end computers can do something else while showing videos. $\endgroup$ – Ellen Spertus Sep 5 '14 at 14:33
3
$\begingroup$

A modern graphics card works more or less (somewhat) the same as a regular cpu.

On a graphics card, you will have (sometimes) multiple processors, each processor will have (sometimes) multiple multiprocessors, which each have multiple cores.

When a video is loaded into a graphics card it gets transcoded into the output buffer by a certain allocation of multiprocessor cores. This output buffer is generally globally accessible memory, which means that any core from any multiprocessor from at least one processor (but often all) has direct r/w access to it.

Usually one multiprocessor is limited to one instruction set, which means that a graphics card can only concurrently operate as many different "flavours" of threads as there are multiprocessors on the card. A "flavour" might for instance be a physics simulation, a video render, an OS render, or a cryptographic function. Naturally one multiprocessor can schedule multiple thread types to run intermittently, but this is usually not necessary.

Decoding a video is often a lot of work, considering how it's usually compressed and sometimes encrypted, so the displaybuffer isn't always busy. Thus it's relatively easy to render a mouse cursor traveling over a video frame. However, sometimes this doesn't quite work, and you will see how in some applications your cursor disappears. This is not necessarily because the application is "on top" of the os, but simply because it's hogging part of the output buffer.

$\endgroup$
3
$\begingroup$

This is implemented using several different techniques. Within the application (for example, the web browser), the program can execute different threads of execution. One thread can perform the video streaming/playback, while a different thread can handle the user typing into the comment field.

These threads, along with many other threads and processes, are in turn scheduled by the operating system (even if you only have the web browser open, the operating system has many background tasks running, such as the windowing manager, the print spooler, the USB device manager, etc). It is the job of the OS scheduler to determine which thread can actively run, on which processor core, and for how long. In a typical system, a thread can run for up to some threshold value, suppose 100 milliseconds, before the scheduler allows another task to run.

This task switching is fundamental to all modern operating systems (Windows, Linux, UNIX, etc), and is usually a significant portion of an undergraduate operating systems course.

$\endgroup$
  • $\begingroup$ Thanks for the reply. I should have made clear that I know about multi-threading and multi-programming. (In fact I'm teaching an undergraduate operating systems course for about the tenth time.) I just don't understand specifically how video playback works $\endgroup$ – Ellen Spertus Sep 5 '14 at 14:35
  • 1
    $\begingroup$ @espertus Here's a paper on scheduling continuous media, maybe that can clear up some of your concerns. While having to read up is far more work than someone just providing an answer, I figured at least it's a starting point since I'm not sure we have someone on this forum versed in the relevant issues. $\endgroup$ – G. Bach Sep 5 '14 at 15:17
2
$\begingroup$

Video playback isn't that special. Sure, GPU-accelerated decoding is common these days, which frees up the CPU for other work, but it wasn't always so. Smooth video playback with only a single CPU is definitely possible, as long as there's enough CPU available for the resolution and complexity of the compression algorithm, and doing it along with other tasks is just as possible, as long as there's enough CPU for both, and the OS schedules well. 30fps video is common, which allows 33ms to decode each frame. Even fairly old chips can manage that for SD-quality MPEG-2, and modern chips can manage it for HD MPEG-4; the laptop I'm typing this on requires about 20% of the CPU to playback 720p High Profile AVC.

As for buffering, there's a limited amount available. The minimum for anything that doesn't want to have screen tearing is "double buffering", where there are two frames in memory: the video card displays one while the software updates the other, and the roles of the two are swapped during the vertical refresh. With "triple buffering", there are two offscreen frames, the next and next + 1, which reduces the amount of jitter. On modern cards with large amounts of RAM it's possible to go beyond that to larger numbers of frames, with the only downsides being user-visible latency in seeking, starting playback, etc. and the fact that other apps might want that video RAM for themselves.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.