Commercial active camera surveillance systems require a novel algorithm for fast and smooth motion tracking of a fast moving object as well as the improvement of video coding performance. So various algorithms based on displaced frame difference (DFD) have been developed due to their relative simplicity. However, they still need large computation for accurate global motion estimation (GME), suffer from noisy DFD, and cannot detect slow object due to the constant frame interval. In this paper, we propose an efficient DFD based motion tracking algorithm consisting of three stages: GME, global motion compensation (GMC), and localization stages. In the GME stage, we choose several confident blocks as reference ones guaranteeing the true global motion. Thereby, the number of search points can be greatly reduced while keeping global motion accuracy. In the GMC stage, we can detect a slow and stationary, or fast object from a DFD. This DFD is obtained by using a previous frame which is properly selected with the proposed double buffering algorithm. Finally, in the localization stage, we extract the moving object area from noisy DFD by using statistics between DFD and the current frame. We implement the proposed algorithm into an active camera system equipped with a pan-tilt unit and a standard PC with an AMD 800MHz processor. It is proved that the system operates well with a camera rotating speed up to 40?sec. It can perform the exhaustive search for a search range of ?0, and achieve the processing speed of about 50 frames/sec for an image sequence of 320x240.