Face Detection in Video

Hi guys, I'm using Viola Jones technique as my face detection. For my case, my face detection is apply on video which are frame by frame. For example, for the first frame, there're 3 faces are detected. What if there're only 2 faces detected at 2nd frame. Which meant, how i address 1st face of first frame as 1st frame of 2nd frame. And 3rd face of 1st frame as 3rd face of 2nd frame. Moreover, both frames can't apply arithmetic operation to make comparison for both frames since both of the frame have different illumination and different movement from users.