Optical duobinary signals have been applied to dense wavelength division multiplexing (DWDM) systems with high spectral efficiency to fully utilize a limited gain bandwidth of about 35 nm (4.4 THz) for an erbium-doped fiber amplifier (EDFA). These signals are one type of partial response signals and have narrower bandwidth than conventional intensity modulation signals. In this thesis, we investigate an optimal duobinary filter for long distance transmission and obtain minimum channel spacing for high spectral efficiency through numerical simulations of transmission characteristics of optical duobinary signals. The simulation is done at 10 Gb/s using $2^7-1$ PRBS (pseudo random bit sequence). The results are compared with optical binary transmission system that is a signal format for conventional optical transmission system. For linear transmission, 10 Gb/s signal can be transmitted up to 226 km of non-dispersion shifted fiber when we use optimum duobinary filter that is a 5th order Bessel-Thomson filter having a 2.7 GHz bandwidth. We also investigate the symmetry requirements of modulator-driving electronics. We can tolerate the amplitude mismatch of modulator-driving signal by 20 %, the phase mismatch by (/8, and the bandwidth mismatch by +200 MHz for obtaining 200 km transmission. In nonlinear multi-channel transmission simulations considering fiber nonlinearities, we find the minimum channel spacing for transmitting a given length of fiber according to the fiber input power. We consider single span, double spans and 4 spans. An amplifier spacing is 80 km. In case of low fiber input power, the optical duobinary transmission has advantage compared with the binary transmission. The minimum channel spacing for the duobinary signal is 18 GHz, while that of the binary signal is 30 GHz, when the fiber input power is 1 mW. However, the optical duobinary transmission has more sensitive dependence on the optical nonlinearity. Thus the advantage disappears, when we increase the transmission power.
