Multiple-input multiple-output (MIMO) scheme is able to improve the modern communications system performance in terms of increased throughput or reliability. As a virtually distributed antenna scheme, the relaying can enhance the communication system performance while there is no physical size limitation at the end user. Through decades, many relaying schemes have been extensively investigated for different channels. When the relay is close to the destination in a static channel and perfect channel state information (CSI) is available at the relay in a slow fading channel, the compress-and-forward (CF) scheme is often applied since it performs better than other relaying schemes. However, the CF scheme requires the relay to perform two-step operation (quantization and WZ binning), which increases the cost of implementing such scheme. In addition, having perfect CSI at the relay is not always possible in a wireless channel. To address these problems caused by the nature of the CF scheme, the generalized quantize-and-forward (GQF) scheme is proposed in this dissertation for the half-duplex (HD) multi-user channel.

In this dissertation, the first part focuses on studying the half-duplex (HD) relaying in the Multiple Access Relay Channel (MARC) and the Compound Multiple Access Channel with a Relay (cMACr). A GQF scheme has been proposed to establish the achievable rate regions. Such scheme is developed based on the variation of the Quantize-and-Forward (QF) scheme and single block with two slots coding structure. The achievable rates results obtained can also be considered as a significant extension of the achievable rate region of Half-Duplex Relay Channel (HDRC). Furthermore, the rate regions based on GQF scheme are extended to the Gaussian channel case. The scheme performance is shown through some numerical examples. In contrast to conventional Full-Duplex (FD) MARC and Interference Relay Channel (IRC) rate achieving schemes which apply the block Markov encoding and decoding in a large number of communication blocks, the GQF developed are based on the single block coding strategies, which are more suitable for the HD channels.

When the relay has no access to Channel State Information (CSI) of the relay-destination link, the GQF is implemented in the slow Rayleigh fading HD-MARC. Based on the achievable rates inequalities, the common outage probability and the expected sum rates are derived. Through numerical examples, we show that in the absence of CSI at the relay, the GQF scheme outperforms other relaying schemes. When the end users have different quality-of-service (QoS) requirements for slow fading channel, it is more precise to use the individual outage related parameters to quantify the scheme performance. The individual outage probability and total throughput are characterized for the HD-MARC. The numerical examples show that the outage probability of the individual users is lower than that of classic Compress-and-Forward (CF) scheme.

The Diversity Multiplexing Tradeoff (DMT) is often applied as a figure of merit for different communication schemes in the asymptotically high SNR slow fading channels. The CF scheme achieves the optimal DMT for high multiplexing gains when the CSI of the relay-destination (R-D) link is available at the relay. However, having the CSI of R-D link at relay is not always possible due to the practical considerations of the wireless system. In this dissertation, the DMT of the GQF scheme is derived without R-D link CSI at the relay. Moreover, the GQF scheme achieves the optimal DMT for the entire range of multiplexing gains.