In this paper we introduce a new target recognition algorithm, group-based Fisherfaces, which is robust to change in pose and brightness variation. The change in pose and brightness variation of target causes the data distribution to be highly non-linear and complex. Proposed group-based Fisherfaces method separates training data into some groups according to clusters and then finds respective basis vectors for each group. Separation of data makes the data distribution become simpler and group-based basis vectors can discriminate classes more precisely than the basis vectors achieved by whole space linear projection methods. In proposed methods, the basis vectors of each group are achieved by Fisherfaces method that finds basis vectors minimizing the within-class scatter and maximizing between-class scatter. If the illumination change is a dominant component of variance in a class, the Fisherfaces method is robust to the illumination change. The proposed algorithm was applied to the experiments recognizing the pose and identity of target in FLIR (forward-Iooking infrared) images. FLIR images contain severe brightness change that is difficult to be modeled. The experimental results show that the proposed algorithm achieves higher recognition rate than the general linear projection methods-whole space PCA and Fisherfaces.