Rate Distortion Optimization for H.264 Interframe Coding: A General Framework and Algorithms

• Published in: IEEE transactions on image processing Vol. 16; no. 7; pp. 1774 - 1784
• Main Authors: YANG, En-Hui, XIANG YU
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.07.2007; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Code standards; Coding; Coding, codes; Computer Graphics - standards; Costs; Data Compression - methods; Data Compression - standards; Decoding; Distortion; Documentation - standards; Encoding; Entropy; Entropy coding; Exact sciences and technology; Fixed-slope lossy compression; Frames; H.264 hybrid coding; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Image processing; Information, signal and communications theory; Internationality; Iterative algorithms; Miscellaneous; Motion estimation; Multimedia - standards; Numerical Analysis, Computer-Assisted; Optimization; Optimization algorithms; Quantization; rate distortion (RD) optimization; Rate-distortion; Sampling, quantization; Signal and communications theory; Signal processing; Signal Processing, Computer-Assisted; soft decision quantization (SDQ); Software algorithms; Studies; Telecommunications and information theory; Video Recording - methods; Lossy compression; Image processing; Video signal; Iterative method; Implementation; Optimization; Interframe encoding; Signal quantization; Image sequence; Fixed-slope lossy compression; Hybrid model; Soft decision; Variable length code; soft decision quantization (SDQ); H.264 hybrid coding; Motion estimation; Optimal algorithm; Rate distortion theory; rate distortion (RD) optimization; Entropy codes; Decoding; Baseline; Adaptive coding; Signal processing; Optimal code; Compatibility; Signal to noise ratio
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2007.896685

Rate distortion (RD) optimization for H.264 interframe coding with complete baseline decoding compatibility is investigated on a frame basis. Using soft decision quantization (SDQ) rather than the standard hard decision quantization, we first establish a general framework in which motion estimation, quantization, and entropy coding (in H.264) for the current frame can be jointly designed to minimize a true RD cost given previously coded reference frames. We then propose three RD optimization algorithms-a graph-based algorithm for near optimal SDQ in H.264 baseline encoding given motion estimation and quantization step sizes, an algorithm for near optimal residual coding in H.264 baseline encoding given motion estimation, and an iterative overall algorithm to optimize H.264 baseline encoding for each individual frame given previously coded reference frames-with them embedded in the indicated order. The graph-based algorithm for near optimal SDQ is the core; given motion estimation and quantization step sizes, it is guaranteed to perform optimal SDQ if the weak adjacent block dependency utilized in the context adaptive variable length coding of H.264 is ignored for optimization. The proposed algorithms have been implemented based on the reference encoder JM82 of H.264 with complete compatibility to the baseline profile. Experiments show that for a set of typical video testing sequences, the graph-based algorithm for near optimal SDQ, the algorithm for near optimal residual coding, and the overall algorithm achieve on average, 6%, 8%, and 12%, respectively, rate reduction at the same PSNR (ranging from 30 to 38 dB) when compared with the RD optimization method implemented in the H.264 reference software.