Coherent detection employing high modulation formats has become one of the most
attractive technologies for long-haul transmission systems due to the high power and
spectral efficiencies. Appropriate digital signal processing (DSP) is used to equalize and
compensate for distortion caused by laser and fiber characteristics and impairments, such
as polarization mode dispersion (PMD), polarization rotation, laser phase noise and
nonlinear effects. Research on the various DSP algorithms in the coherent optical
communication systems is the most promising investigation.
In this research, two new modulation formats; polarization QAM modulation (POL-QAM)
and set-partitioning QAM (SP-QAM) are investigated due to their high spectral
efficiency and novelty. For PMD and polarization rotation equalization, a new modified
constant modulus algorithm (CMA) is proposed for POL-QAM. We investigate the bit
error rate (BER) performance of the two modulation formats over fiber channel
considering PMD and polarization rotation effects. Furthermore, we investigate the BER
performance of the two modulation formats over long-haul fiber optic transmission
systems. Carrier phase estimation (CPE) algorithms are also investigated, which are used
to mitigate phase noise caused by the transmitter.