Improved rate-distortion optimized video coding using non-integer bit estimation and multiple Lambda search

• Published in: Frontiers of Computer Science Vol. 10; no. 1; pp. 157 - 166
• Main Authors: Kei IM, Sio, Mahdi GHANDI, Mohammad
• Format: Journal Article
• Language: English
• Published: Beijing Higher Education Press 01.02.2016; Springer Nature B.V
• Subjects: Algorithms; Arithmetic coding; Codec; Coders; Computer Science; Distortion; H.264/AVC; H265/HEVC video coding; Integers; Lambda adjustment; Motion simulation; non-integer bit estimation; rate distortion optimization; Research Article; Searching; 决策算法; 成本计算; 搜索; 点估计; 率失真优化; 编解码器; 视频编码器; 非整数; rate distortion optimization; H265/HEVC video coding; Lambda adjustment; H.264/AVC; non-integer bit estimation
• ISSN: 2095-2228; 2095-2236
• DOI: 10.1007/s11704-015-5066-1

Many modern video encoders use the Lagrangian rate-distortion optimization (RDO) algorithm for mode decisions during the compression procedure. For each encoding stage, this approach involves minimizing a cost, which is a function of rate, distortion and a multiplier called Lambda. This paper proposes to improve the RDO process by applying two modifications. The first modification is to increase the accuracy of rate estimation, which is achieved by computing a non-integer number of bits for arithmetic coding of the syntax elements. This leads to a more accurate cost computation and therefore a better mode decision. The second modification is to search and adjust the value of Lambda based on the characteristics of each coding stage. For the encoder used, this paper proposes to search multiple values of Lambda for the intra-4×4mode decision. Moreover, a simple shift in Lambda value is proposed for motion estimation. Each of these modifications offers a certain gain in RDO performance, and, when all are combined, an average bit-rate saving of up to 7.0% can be achieved for the H.264/AVC codec while the same concept is applicable to the H.265/HEVC codec as well. The extra added complexity is contained to a certain level, and is also adjustable according to the processing resources available.