For face tracking in a video sequence, many face tracking methods have been developed. However, most of them have a problem that the initial position of the face should be known. In this paper, we present a fast and efficient algorithm for fully automatic detection and tracking a face. Face detection is performed by binary template matching. Each sub-image extracted from the input image is binarized by an effective threshold value. The value is computed by an average intensity value. Then, the binarized sub-image is compared to predefined binary face templates. Once a face is reliably detected, we extract a skin color distribution from the detected facial region for fast face tracking. Face tracking is realized using skin color, motion information, and binary template matching. Our proposed algorithm is computationally efficient, so that it can be executed in real-time. In addition, our algorithm is robust to face orientation, scale changes, and lighting condition changes.