Detector randomization and stochastic signaling for minimum probability of error receivers

Optimal receiver design is studied for a communications system in which both detector randomization and stochastic signaling can be performed. First, it is proven that stochastic signaling without detector randomization cannot achieve a smaller average probability of error than detector randomization with deterministic signaling for the same average power constraint and noise statistics. Then, it is shown that the optimal receiver design results in a randomization between at most two maximum a-posteriori probability (MAP) detectors corresponding to two deterministic signal vectors. Numerical examples are provided to explain the results.