It is well known that the performance of video coder can be improved using rate-distortion optimization, which selects each parameter to minimize rate and distortion simultaneously. If we use a quantization parameter (OP) as an optimization parameter, we can get remarkable improvement in PSNR. But the video coder like H.263 uses the quantization parameter as an optimization parameter has no way to control output bitrate of video coder any more, because now the quantization parameter is selected to minimize rate and distortion simultaneously. Rate-control in real video coder is important because it can save the size of a decoder buffer and also can prevent it from overflowing or underflowing of the decoder buffer.
We propose two new algorithms and show the results of these algorithms applied to the H 263 codec. In both algorithms, we suggest new method using Lagrange multiplier in rate-distortion optimization instead of the quantization parameter to control the output bitrate of video coder. First, we present the result of PSNR and bitrate using H.263 TMN5 and the existing rate-distortion optimization algorithm. In the thesis, it is shown that existing algorithm, which uses Lagrange multiplier in frame level has poor performance of rate control. The first proposed algorithm gives the value of new parameter at every GOB. The fluctuation of the bitrate in this algorithm decreases remarkably, but PSNR gain also decreases compared to the existing algorithm. To recover this defect, we propose the second algorithm in which the value of the new parameter is determined by two different ways. This algorithm gives about 1dB improvement in PSNR and shows a less fluctuation of the bitrate.
