I have just started discovering the machine learning world! I found many tutorials showing how to deal with different types of data. I walked through some tensorFlow tutorials and I think they have very good way for showing how the (x, y) pair for supervised learning is "formed".
I was not able to find a way for representing the (x, y) pair for a video though. Are there any resources that can help me get started with labeling video sequences?
Try Computer Vision Algorithms and Applications by Richard Szeliski and label propagation in video sequences.
To get a good grasp on the topic also Supervised Sequence Labeling with Recurrent Neural Networks. I proposed these resources as they cover the labeling, machine learning and are very in-depth materials (book, dissertation and the paper about propagating labels).
Video is the natural extension of the image, but to get fully working methods we also have to account for scene movement and changes. And I belive that in the video point is referenced by $x, y, t$.
This is studied in the computer vision world. Start by understanding how to do that for an image; then video is a straightforward extension of those techniques. So, read some good textbooks and tutorials on computer vision.
