In a nutshell. I coded a video player application (on linux). I used a proprietary library/API, with this, video and audio frames comes with well defined format. After decoding (via API), I call scheduling() function for video and audio frames and library does the rest. For argument sake I oversimplified the example.
Now let's say we switched to open source API (like Libav) with this I have everything similar as before except scheduling functions.
I need help to devise a scheduling algorithm to use in my video player.
I have video frames and audio frames with timestamps (can be converted to wallclock time via a simple function). Naive approach I did was, rendering frames same order immediately as they arrive. This looks like working but overtime lips sync problem occurs (as expected). I don't quite get it what scheduling function I used before actually doing and since these are closed source (comes as pre-compiled) I can't check it.
The original function takes these parameters (if these give some idea/help):
theFrame in frame to display
displayTime in time at which to display the frame in
displayDuration in duration for which to display the frame in
timeScale in time scale for
I have all of this parameters/values for each frame.
What is a scheduling algorithm supposed to do?
What I know is, Scheduling algorithm buffers frames and outputs (video or audio) on scheduled time.
What are its outputs?
Audio and video uses different function but basically does same thing. It outputs video or audio. Simplest terms you see the video and hear audio on the player application thanks to this function correctly in sync.
What are the requirements?
1) Frame to output (video or audio). i.e. YUV Frame Buffer or a PCM packet for audio.
2) Timestamp (stamped on encoder side). i.e. 64bit signed integer, after simple conversion function gives a wall clock kinda time (HH:MM:SS.ms).
3) Timing information (stream time_base and frame duration) i.e. time_base for video : 1(num)/50(den=framerate), duration is tricky let's say it's 1/framerate secs (20 ms in this example)
What is the "overtime lips sync problem"?
Video and audio goes out of sync. In other words, video and audio doesn't match anymore.
What I'm asking is a generic logic for a Scheduling Algorithm, I don't wanna bore with you with details, so I kept is simple.
Let me ask different way:
Think I have a file (like a SRT subtitle file) it has text on every line and timestamp. And I wanna print it on linux terminal line by line according to it's timestamps.
My naive thinking will be:
1) Start a timer when program starts, load the file (SRT) on the memory.
2) Parse lines and timestamps.
3) Find the line with (in the SRT) closest to our timer and print it.
4) Repeat step (3) until EOF
In my original question/problem, I can't load whole media file, it can be a live stream or very big file. I can only buffer few seconds worth of frames (i.e. 25 frames (each for video or audio)).
So what should I do ?