The recent exponential growth in data traffic and the Internet places urgent demands on high capacity WDM optical transmission systems. To fulfill these demands, it is very attractive to widen the bandwidth of optical amplifiers. Erbium-doped fiber amplifier(EDFA) is key technology in WDM system and its usable bandwidth was limited by its physical characteristics. In order to solve this problem, it was suggested to use acousto-optic tunable filters(AOTF) as active gain-flattening filters in the mid-stage of EDFA. To flatten the gain, an algorithm to control the spectral characteristic of AOTF and pump power is required. However, the algorithm that has been adopted in gain flattening of EDFA is heuristic and it has many weighting factors which are determined by trials and errors.
In this paper, we propose a new algorithm to flatten the gain spectrum of EDFA. This algorithm is based on the well-known numerical method, nonlinear least square, to estimate spectral characteristic of AOTF which modifies the output spectrum of EDFA to flatten the gain. The new algorithm gives better performance with respect to the previous heuristic methods. The maximum gain ripple of the gain flattened EDFA was less than 0.5 dB. In addition, the new algorithm requires three times small iterations for gain flattening. We also suggest some guidelines to design of dynamically gain flatted EDFA with AOTF. The dynamic range of the dynamically gain flattened amplifier is much wider than that of the conventional amplifiers that have the gain flattening filters and a variable optical attenuator.