Suppose I have a video in which there are two objects $A$ and $B$. Object $A$ appears suddenly at time $Ta1$, and then disappears suddenly at time $Ta2$. Object $B$ appears suddenly at time $Tb1$, and then starts to fade gradually at time $Tb2$, and at $Tb3$ the object $B$ completely fades away. The background intensity is nearly constant, but is changing at a slope $s$ ($s$ can be a very small positive, zero or very small negative) in the long run. For simplicity, we can assume the objects never move. Also, the time $Ta1$, $Ta2$, $Tb1$, $Tb2$ and $Tb3$ are not known beforehand, they can only be inferred from the data or the graphs.

In the graphs below, (a) shows the intensity of a pixel where object $A$ appears against time, (b) shows the intensity of a pixel where object $B$ appears against time, and (c) shows the intensity of a background pixel against time. The graph on the left hand side is the theoretical plot, while the graph on the right hand side is the actual plot from my data. In all graphs, the slope $s$ is a very small positive number.


I tried to use the Running Gaussian Average with Selective Background Update method. It is quite successful in segmenting the background from the objects $A$ and $B$, except that a slight amount of background may be wrongly regarded as foreground at a late time (when the intensity of the background increases by a certain amount). But leaving this issue aside, I would like to do one step further, that is, to distinguish between objects $A$ and $B$. Surely if $Ta1=Tb1$ and $Ta2=Tb2$, the distinction can only be made after $Tb2$ (or even $Tb3$). But what methods can be applied to perform the distinction?

  • $\begingroup$ could you say more about the underlying (real?) process going on here? in what kinds of (continuous) video scenes of anything do objects suddenly appear out of nowhere (without moving)? $\endgroup$
    – vzn
    Commented Sep 23, 2015 at 15:58
  • $\begingroup$ @vzn: In reality object A does move (and may overlap object B), but here I just would like to simplify the situation, because I would like to consider solutions that focus on single pixel (i.e. not considering neighboring pixels or shapes of foreground). $\endgroup$ Commented Sep 23, 2015 at 17:55

1 Answer 1


There are probably many possible approaches, but I'll suggest one candidate step-by-step approach that I think might work well, given the information shown in your question:

1. Find the times

As a first step, I suggest that you estimate $Ta1,Ta2,Tb1,Tb2,Tb3$.

The good thing is that $Ta1,Ta2$ will be the same for all pixels of object A, and $Tb1,Tb2,Tb3$ will be the same for all pixels of object B. This means you only need to estimate 5 numbers, and you have an entire video to work with -- so you should have more than enough data to infer these 5 times.

One reasonable way to do this would be to do foreground-background segmentation to identify the foreground pixels, using the Running Gaussian Average with Selective Background Update method you already tried; as you said, this is already fairly accurate. It won't need to be perfect -- just getting it approximately right should be enough for what I'm going to suggest.

When you've done that, pixels inferred to be foreground pixels will be a mixture of object A and object B (plus a little bit of error). Now looking at just the inferred foreground pixels, you should be able to deduce $Ta1$, $Ta2$, $Tb1$, $Tb2$, $Tb3$. For instance, at each time, compute the average pixel intensity over all foreground pixels, and graph this as a function of time. You should find that this time series is roughly constant until $Ta1$, then has a large increase at $Ta1$, another large increase at $Tb1$, a big decrease at $Ta2$, and a smooth decrease from $Tb2$ to $Tb3$. Each of those increases/decreases should be easily detectable in the signal.

For instance, you could take the first derivative of this time series (the average intensity of the foreground pixels) and look at its peaks. You should find large positive peaks at $Ta1$ and $Tb1$ and a large negative peak at $Ta2$. Apart from those three times, the first derivative should be close to zero for all time periods, except that it is a significant negative value from $Tb2$ and $Tb3$. I suspect these two characteristics should be enough to identify $Ta1,Ta2,Tb1,Tb2,Tb3$.

If that's not accurate enough, there are alternative approaches; ask if you need them. But I expect it will be relatively easy to implement this step and find the times $Ta1,Ta2,Tb1,Tb2,Tb3$.

2. Build a three-class classifier

Now that you know the times $Ta1,Ta2,Tb1,Tb2,Tb3$, I suggest that as your next step you build a classifier that takes the time series for a particular pixel and classifies it as one of three classes: A (object A), B (object B), or Bg (background).

It will probably suffice to build a simple classifier by hand. My guess is you probably won't need terribly sophisticated methods. I'd suggest you identify a few features, and for each pixel, compute a feature vector (the values of those features, for that particular pixel). There are some natural features you could use, for instance:

  • $\text{avg}([Ta1+\varepsilon,Ta1+2\varepsilon])-\text{avg}([Ta1-2\varepsilon,Ta1-\varepsilon])$, where $\text{avg}([t_1,t_2])$ is the average intensity of the pixel taken over times in the interval $[t_1,t_2]$. This basically tells you how large a "jump" in the signal there is, at time $Ta1$; for A you expect it to be large, whereas it should be approximately zero for B and Bg.

  • $\text{avg}([Ta2+\varepsilon,Ta2+2\varepsilon])-\text{avg}([Ta2-2\varepsilon,Ta2-\varepsilon])$. For $A$ you expect this to be very negative, whereas it should be approximately zero for B and Bg.

  • $\text{avg}([Tb1+\varepsilon,Tb1+2\varepsilon])-\text{avg}([Tb1-2\varepsilon,Tb1-\varepsilon])$. For B you expect it to be large, whereas it should be approximately zero for A and Bg.

  • $\text{avg}([Tb3+\varepsilon,Tb1+3\varepsilon])-\text{avg}([Tb2-2\varepsilon,Tb2-\varepsilon])$. For B you expect it to be very negative, whereas it should be approximately zero for A and Bg. (unless $Ta1$ or $Ta2$ falls within the interval $[Tb2,Tb3]$, in which case you can also figure out what you expect it to be: very positive for A if $Ta1$ is in the interval but $Ta2$ isn't; very negative if $Ta2$ is in the interval but $Ta1$ isn't; or zero if they're both in the interval)

  • The fraction of the time points that your Running Gaussian Average with Selective Background Update method classifies the pixel as background. For A and B we expect it to be small; for Bg we expect it to be large.

You might be able to build a classifier simply by setting thresholds for each of these features and using the result to select which object you have.

If that's not accurate enough, you could imagine training a classifier using some machine learning method (e.g., logistic regression, linear SVM, etc.).

Now apply this classifier to each pixel separately, to classify it as belonging to object A, object B, or the background.

3. Optional: Clean up the results

You could optionally clean up the results by using spatial locality: we expect object A to be a connected region in the image, and same for object B. For instance, if a pixel belongs to object A, then that makes it incrementally a bit more likely that the neighboring pixels belong to object A. Or, a more sophisticated version of this is: if two adjacent pixels have approximately the same color, then they're a bit more likely to belong to the same class; if they have very different colors, there are no restrictions (they might belong to the same class, or might belong to a different class; each is equally plausible).

There are methods to take this information into account. For instance, suppose your classifier outputs not just its predicted class, but also a probability for each of the three classes (e.g., 90% chance this particular pixel is object A, 8% chance it is object B, 2% chance it is background). Then you can combine this with a graph cut method to take into account spatial locality, i.e., to find a segmentation that both takes into account your classifier's predictions and also takes into account spatial locality.

This kind of post-processing might further enhance the accuracy of your method, by cleaning up the boundaries of the objects.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.