The light field and holographic displays constitute two important categories of advanced three-dimensional displays that are aimed at delivering all physiological depth cues of the human visual system, such as stereo cues, motion parallax, and focus cues, with sufficient accuracy. As human observers are the end-users of such displays, the delivered spatial information (e.g., perceptual spatial resolution) and view-related image quality factors (e.g., focus cues) are significantly dependent on the characteristics of the human visual system. Retinal image formation models enable rigorous characterization and subsequently efficient design of light field and holographic displays. In this chapter the ray-based near-eye light field and wave-based near-eye holographic displays are reviewed, and the corresponding retinal image formation models are discussed. In particular, most of the discussion is devoted to characterization of the perceptual spatial resolution and focus cues.