Academia.eduAcademia.edu

Figure 39 – uploaded by WENQING GE

See full PDFdownloadDownload figure
Table 8.3 Coding of delta QP in HEVC and H.264/AVC Underlined symbols are bypass coded

Table 8 3 Coding of delta QP in HEVC and H.264/AVC Underlined symbols are bypass coded

Related Figures (230)

Fig. 2.1 Overview of HEVC encoding and decoding coding other pictures. These stored pictures are called reference pictures. Each picture in HEVC is partitioned into one or multiple slices. Each slice is independent of other slices in the sense that the information carried in the slice is coded without any dependency on data from other slices within the same picture. A slice consists of one or multiple slice segments, where the first slice segment of a slice is called independent slice segment and is independent of other slice segments. The subsequent slice segments, if any, are called dependent slice segments since they depend on previous slice segments.
Fig. 2.1 Overview of HEVC encoding and decoding coding other pictures. These stored pictures are called reference pictures. Each picture in HEVC is partitioned into one or multiple slices. Each slice is independent of other slices in the sense that the information carried in the slice is coded without any dependency on data from other slices within the same picture. A slice consists of one or multiple slice segments, where the first slice segment of a slice is called independent slice segment and is independent of other slice segments. The subsequent slice segments, if any, are called dependent slice segments since they depend on previous slice segments.
Fig. 2.3. Temporal sub-layer examples
Fig. 2.3. Temporal sub-layer examples
2.2.2 VCL NAL Unit Types Table 2.1 shows all 32 VCL NAL unit types and their NAL unit type (NALType in Fig. 2.2) values in the NAL unit header. All VCL NAL units of the same access unit must have the same value of NAL unit type and that value defines the type of the access unit and its coded picture. For example, when all VCL NAL units of an access unit have NAL unit type equal to 21, the access unit is called a CRA access unit and the coded picture is called a CRA picture. There are three basic classes of pictures in HEVC: intra random access point (IRAP) pictures, leading pictures, and trailing pictures.
2.2.2 VCL NAL Unit Types Table 2.1 shows all 32 VCL NAL unit types and their NAL unit type (NALType in Fig. 2.2) values in the NAL unit header. All VCL NAL units of the same access unit must have the same value of NAL unit type and that value defines the type of the access unit and its coded picture. For example, when all VCL NAL units of an access unit have NAL unit type equal to 21, the access unit is called a CRA access unit and the coded picture is called a CRA picture. There are three basic classes of pictures in HEVC: intra random access point (IRAP) pictures, leading pictures, and trailing pictures.
There are three types of trailing pictures in HEVC: temporal sub-layer access (TSA) pictures, step-wise temporal sub-layer access (STSA) pictures, and ordinary trailing pictures (TRAIL). 2.2.2.3 Temporal Sub-layer Access (TSA) Pictures A TSA picture is a trailing picture that indicates a temporal sub-layer switching point. The TSA picture type can only be used for a picture if it is guaranteed that no picture that precedes the TSA picture in decoding order with a temporal ID that is greater than or equal to the temporal ID of the TSA picture itself is used for prediction of the TSA picture or any subsequent (in decoding order) pictures in the same or higher temporal sub-layer as the TSA picture. For example, picture Pg in Fig. 2.5 can use the TSA picture type since only previous pictures in temporal sub- layer 0 are used for prediction of the TSA picture itself and subsequent pictures in decoding order.
There are three types of trailing pictures in HEVC: temporal sub-layer access (TSA) pictures, step-wise temporal sub-layer access (STSA) pictures, and ordinary trailing pictures (TRAIL). 2.2.2.3 Temporal Sub-layer Access (TSA) Pictures A TSA picture is a trailing picture that indicates a temporal sub-layer switching point. The TSA picture type can only be used for a picture if it is guaranteed that no picture that precedes the TSA picture in decoding order with a temporal ID that is greater than or equal to the temporal ID of the TSA picture itself is used for prediction of the TSA picture or any subsequent (in decoding order) pictures in the same or higher temporal sub-layer as the TSA picture. For example, picture Pg in Fig. 2.5 can use the TSA picture type since only previous pictures in temporal sub- layer 0 are used for prediction of the TSA picture itself and subsequent pictures in decoding order.
random access at the position of the CRA picture will not decode the P2 picture, and therefore cannot decode these RASL pictures and will discard them. Even though it is not forbidden to use the RASL type for decodable leading pictures, such as the RADL picture in Fig. 2.6, it is recommended to use the RADL type when possible to be more network friendly. Only other RASL pictures are allowed to be dependent on a RASL picture. This means that every picture that depends on a RASL picture must also be a RASL picture. RADL and RASL pictures may be mixed in decoding order, but not in output order. RASL pictures must precede RADL pictures in output ardear
random access at the position of the CRA picture will not decode the P2 picture, and therefore cannot decode these RASL pictures and will discard them. Even though it is not forbidden to use the RASL type for decodable leading pictures, such as the RADL picture in Fig. 2.6, it is recommended to use the RADL type when possible to be more network friendly. Only other RASL pictures are allowed to be dependent on a RASL picture. This means that every picture that depends on a RASL picture must also be a RASL picture. RADL and RASL pictures may be mixed in decoding order, but not in output order. RASL pictures must precede RADL pictures in output ardear
Fig. 2.7 Examples of disallowed referencing structures Fig. 2.8 Original (a) and new (b) referencing structures before splicing has occurred
Fig. 2.7 Examples of disallowed referencing structures Fig. 2.8 Original (a) and new (b) referencing structures before splicing has occurred
.2.11 Reserved and Unspecified VCL NAL Unit Types
.2.11 Reserved and Unspecified VCL NAL Unit Types
Table 2.2. The 32 HEVC non-VCL NAL unit types payload so they dot CONsIst OF only te two Dyte INAL unit neader, Filler data NAL units have no impact on the decoding process. The payload is a series of bytes equal to ‘11111111’ followed by a byte equal to ‘10000000’. It can be used to fill up a data channel to a desired bit rate in the absence of an adequate amount of VCL data. Filler data NAL units must signal the same temporal ID as the coded picture and they are not allowed to precede the first VCL NAL unit in the ACCeSS unit.
Table 2.2. The 32 HEVC non-VCL NAL unit types payload so they dot CONsIst OF only te two Dyte INAL unit neader, Filler data NAL units have no impact on the decoding process. The payload is a series of bytes equal to ‘11111111’ followed by a byte equal to ‘10000000’. It can be used to fill up a data channel to a desired bit rate in the absence of an adequate amount of VCL data. Filler data NAL units must signal the same temporal ID as the coded picture and they are not allowed to precede the first VCL NAL unit in the ACCeSS unit.
sequence, and is intended for use in the upcoming layered extensions of HEVC, which will enable scalable and multiview coding. While the PPS may differ for separate pictures, it is common for many or all pictures in a coded video sequence to refer to the same PPS. Reusing parameter sets is bit rate efficient because it avoids the necessity to send shared information multiple times. It is also loss robust because it allows parameter set content to be carried by some more reliable external communication link or to be repeated frequently within the bitstream to ensure that it will not get lost. This ability to reuse the content of a picture parameter set in dif- ferent pictures and to reuse the content of SPSs and VPSs in different CVSs is what primarily distinguishes the concept of a “parameter set” from the “picture header” and “sequence header” syntax used in older standards established prior to AVC. To identify for a given slice the active parameter set at each level of the parameter eet tune hierarchy each eclice header caontaine a PPS identifier which referencec 9
sequence, and is intended for use in the upcoming layered extensions of HEVC, which will enable scalable and multiview coding. While the PPS may differ for separate pictures, it is common for many or all pictures in a coded video sequence to refer to the same PPS. Reusing parameter sets is bit rate efficient because it avoids the necessity to send shared information multiple times. It is also loss robust because it allows parameter set content to be carried by some more reliable external communication link or to be repeated frequently within the bitstream to ensure that it will not get lost. This ability to reuse the content of a picture parameter set in dif- ferent pictures and to reuse the content of SPSs and VPSs in different CVSs is what primarily distinguishes the concept of a “parameter set” from the “picture header” and “sequence header” syntax used in older standards established prior to AVC. To identify for a given slice the active parameter set at each level of the parameter eet tune hierarchy each eclice header caontaine a PPS identifier which referencec 9
Fig. 2.12 Two referencing structures that need a DPB size of at least three pictures
Fig. 2.12 Two referencing structures that need a DPB size of at least three pictures
Table 2.3 The five RPS lists
Table 2.3 The five RPS lists
Table 2.4 RPS for each picture of Fig. 2.15 An IDR picture resets the codec which includes turning all pictures in the DPB into non-reference pictures. Since the RPSs of IDR pictures are empty, there is no RPS syntax signaled for IDR pictures. All lists in Table 2.4 are therefore empty for the IDR picture Ip. At picture B3, picture P; is put in RefPicSetStFoll since P| is not referenced by B3. P; is however kept in the DPB since it is used for future pictures. At picture Ba, Ip is made into a long-term picture in this example, it is therefore put in RefPicSetLtFoll since it is not referenced by By. At picture Ps, the encoder makes pictures By and B3 non-reference pictures by not including them in the RPS at all. At the same time, picture Ip is moved to RefPicSetLtCurr since it is referenced by Ps. Ip is thereafter kept in RefPicSetLtFoll for later use. The encoder is required to ensure that every picture that is indicated in RefPic-
Table 2.4 RPS for each picture of Fig. 2.15 An IDR picture resets the codec which includes turning all pictures in the DPB into non-reference pictures. Since the RPSs of IDR pictures are empty, there is no RPS syntax signaled for IDR pictures. All lists in Table 2.4 are therefore empty for the IDR picture Ip. At picture B3, picture P; is put in RefPicSetStFoll since P| is not referenced by B3. P; is however kept in the DPB since it is used for future pictures. At picture Ba, Ip is made into a long-term picture in this example, it is therefore put in RefPicSetLtFoll since it is not referenced by By. At picture Ps, the encoder makes pictures By and B3 non-reference pictures by not including them in the RPS at all. At the same time, picture Ip is moved to RefPicSetLtCurr since it is referenced by Ps. Ip is thereafter kept in RefPicSetLtFoll for later use. The encoder is required to ensure that every picture that is indicated in RefPic-
Fig. 2.15 Example referencing structure
Fig. 2.15 Example referencing structure
Table 2.5 RPS syntax for picture Bz in Fig. 2.15
Table 2.5 RPS syntax for picture Bz in Fig. 2.15
——— — First, temporary lists for LO and L1 are constructed. Temporary list LO start with the pictures in RPS list RefPicSetStCurrBefore sorted in descending POC valu order. These are all short-term pictures that are available for reference for the curre1 picture and have a POC value that is less than the POC value of the current pictur [he temporary list LO for picture B7 in Fig. 2.16 therefore starts with P, followe by By. These are followed by the pictures in the RPS list RefPicSetStCurrA fte sorted in ascending POC value order, which are Bg and Ps in Fig. 2.16. Finalh he available long-term pictures are added, so picture Ip is the last picture to b added. The final temporary list LO for picture B7 is then {P,, Bz, Be, Ps, Ip}. Th emporary list L1 is set up similar to LO, but the order of RefPicSetStCurrBefore an RefPicSetStCurrAfter is swapped. The temporary list L1 for picture B7 therefor becomes {Be, Ps, Pi, Bo, Io}. OO a a a ae I a ee ge PR ON ane AT 4 ieee | 6OThkam Le ae
——— — First, temporary lists for LO and L1 are constructed. Temporary list LO start with the pictures in RPS list RefPicSetStCurrBefore sorted in descending POC valu order. These are all short-term pictures that are available for reference for the curre1 picture and have a POC value that is less than the POC value of the current pictur [he temporary list LO for picture B7 in Fig. 2.16 therefore starts with P, followe by By. These are followed by the pictures in the RPS list RefPicSetStCurrA fte sorted in ascending POC value order, which are Bg and Ps in Fig. 2.16. Finalh he available long-term pictures are added, so picture Ip is the last picture to b added. The final temporary list LO for picture B7 is then {P,, Bz, Be, Ps, Ip}. Th emporary list L1 is set up similar to LO, but the order of RefPicSetStCurrBefore an RefPicSetStCurrAfter is swapped. The temporary list L1 for picture B7 therefor becomes {Be, Ps, Pi, Bo, Io}. OO a a a ae I a ee ge PR ON ane AT 4 ieee | 6OThkam Le ae
Table 2.6 HEVC SEI messages
Table 2.6 HEVC SEI messages
Table 2.6 (continued)
Table 2.6 (continued)
new features of tiles and wavefront parallel processing (WPP). Also, it is described how these picture partitioning features can be used together with the concept of slice segments for improved bitstream access. Section3.3.3 deals with the same combination of tools but from the different perspective of support for ultra-low delay applications. Finally, in Sect. 3.3.4, the assets and drawbacks of tiles and WPP are discussed. The whole chapter is concluded in Sect. 3.4.
new features of tiles and wavefront parallel processing (WPP). Also, it is described how these picture partitioning features can be used together with the concept of slice segments for improved bitstream access. Section3.3.3 deals with the same combination of tools but from the different perspective of support for ultra-low delay applications. Finally, in Sect. 3.3.4, the assets and drawbacks of tiles and WPP are discussed. The whole chapter is concluded in Sect. 3.4.
Fig. 3.2 Illustration of the partitioning of a picture with 1280 x 720 luma samples into macro- blocks and coding tree units: (a) Partitioning of the picture into 16 x 16 macroblocks as found in all prior video coding standards of the ITU-T and ISO/IEC; (b) Partitioning of the picture into 64 x 64 coding tree units, the largest coding tree unit size supported in the Main profile of HEVC
Fig. 3.2 Illustration of the partitioning of a picture with 1280 x 720 luma samples into macro- blocks and coding tree units: (a) Partitioning of the picture into 16 x 16 macroblocks as found in all prior video coding standards of the ITU-T and ISO/IEC; (b) Partitioning of the picture into 64 x 64 coding tree units, the largest coding tree unit size supported in the Main profile of HEVC
Fig. 3.3. Macroblock partitioning modes supported in the High profile of H.264 | MPEG-4 AVC for inter-picture coding (top line) and intra-picture coding (bottom line). If the Inter-8 x 8 is chosen, the 8 x 8 sub-macroblocks can be further partitioned into 8 x 4, 4 x 8, or 4 x 4 blocks
Fig. 3.3. Macroblock partitioning modes supported in the High profile of H.264 | MPEG-4 AVC for inter-picture coding (top line) and intra-picture coding (bottom line). If the Inter-8 x 8 is chosen, the 8 x 8 sub-macroblocks can be further partitioned into 8 x 4, 4 x 8, or 4 x 4 blocks
Fig. 3.4 Example for the partitioning of a 64 x 64 coding tree unit (CTU) into coding units (CUs) of 8 X 8 to 32 x 32 luma samples. The partitioning can be described by a quadtree, also referred to as coding tree, which is shown on the right. The numbers indicate the coding order of the CUs
Fig. 3.4 Example for the partitioning of a 64 x 64 coding tree unit (CTU) into coding units (CUs) of 8 X 8 to 32 x 32 luma samples. The partitioning can be described by a quadtree, also referred to as coding tree, which is shown on the right. The numbers indicate the coding order of the CUs
rediction mode is selected for a CU and signaled inside the bitstream. The chroma ntra prediction mode applies to both chroma CBs and can be selected among five andidates, where one of the candidates represents the intra prediction mode chosen or the luma CB or the first luma intra block in case four intra prediction modes re transmitted for the luma CB. The reason why the signaling of four luma intra rediction modes is only supported for the minimum CU size is the following. f we consider a picture block of 2% x 2% luma samples that is larger than the ninimum CU size, then partitioning the block into four CUs and transmitting a uma intra prediction mode for each of the CUs is, for the luma component, the ame as coding the entire block as one CU and transmitting an intra prediction node for the four luma subblocks. Even though these partitioning methods do not nield the same result for the chroma components, they can be seen as redundant yntax features. And since the impact of coding the chroma components on the verall coding efficiency is usually very small, the corresponding redundant syntax s avoided and a subdivision of the luma CB for signaling intra prediction modes is mly supported for the minimum CU size. The actual intra prediction is not always applied to the blocks for which the intra
rediction mode is selected for a CU and signaled inside the bitstream. The chroma ntra prediction mode applies to both chroma CBs and can be selected among five andidates, where one of the candidates represents the intra prediction mode chosen or the luma CB or the first luma intra block in case four intra prediction modes re transmitted for the luma CB. The reason why the signaling of four luma intra rediction modes is only supported for the minimum CU size is the following. f we consider a picture block of 2% x 2% luma samples that is larger than the ninimum CU size, then partitioning the block into four CUs and transmitting a uma intra prediction mode for each of the CUs is, for the luma component, the ame as coding the entire block as one CU and transmitting an intra prediction node for the four luma subblocks. Even though these partitioning methods do not nield the same result for the chroma components, they can be seen as redundant yntax features. And since the impact of coding the chroma components on the verall coding efficiency is usually very small, the corresponding redundant syntax s avoided and a subdivision of the luma CB for signaling intra prediction modes is mly supported for the minimum CU size. The actual intra prediction is not always applied to the blocks for which the intra
Fig. 3.6 Supported partitioning modes for splitting a coding unit (CU) into one, two, or four prediction units (PU). The (M/2) x (M/2) mode and the modes shown in the bottom row are not supported for all CU sizes
Fig. 3.6 Supported partitioning modes for splitting a coding unit (CU) into one, two, or four prediction units (PU). The (M/2) x (M/2) mode and the modes shown in the bottom row are not supported for all CU sizes
Table 3.1 Comparison of motion compensation block sizes supported in different standards
Table 3.1 Comparison of motion compensation block sizes supported in different standards
Fig. 3.7 Example for the partitioning of a 64 x 64 luma coding tree block (black) into coding blocks (blue) and transform blocks (red). In the illustration on the right, the blue lines show the corresponding coding tree with the coding tree block (black square) at its root and the coding blocks (blue circles) at its leaf nodes; the red lines show the non-degenerated residual quadtrees with the transform blocks (red circles) as leaf nodes. Note that the transform blocks chosen identical to the corresponding coding blocks are not explicitly marked in this figure. The numbers indicate the coding order of the transform blocks
Fig. 3.7 Example for the partitioning of a 64 x 64 luma coding tree block (black) into coding blocks (blue) and transform blocks (red). In the illustration on the right, the blue lines show the corresponding coding tree with the coding tree block (black square) at its root and the coding blocks (blue circles) at its leaf nodes; the red lines show the non-degenerated residual quadtrees with the transform blocks (red circles) as leaf nodes. Note that the transform blocks chosen identical to the corresponding coding blocks are not explicitly marked in this figure. The numbers indicate the coding order of the transform blocks
Table 3.2. Summary of the test sequences for the two considered application scenarios that are used in the comparisons. All sequences are given in the 4:2:0 chroma sampling format with a bit depth of 8 bit per sample and have a duration of 10s. The resolution is specified in luma samples The hierarchical prediction structure chosen for the entertainment application scenario provides a very high coding efficiency, but it is associated with a structural delay of eight pictures. For interactive video applications, such a high delay is not acceptable. Here, we used a dyadic low-delay hierarchical prediction structure with groups of four pictures [36] and four active reference pictures. For this coding structure, all pictures are coded in display order, but the QP is varied depending on the hierarchy level. Similarly as for the coding structure chosen for the entertainment scenario, the QP is increased by 1 from one hierarchy level to the next. Except the first picture of a video sequence, which is coded as I picture, all pictures are coded as B pictures. The QP for the I picture is decreased by | relative to the QP for the B pictures of the lowest hierarchy level. Furthermore, all pictures are coded as a single slice. For the application scenario of interactive video application, we selected six sequences with typical video conferencing content. All of these sequences have a resolution of 1280 x 720 luma samples. The chosen test sequences for both scenarios are summarized in Table 3.2. They
Table 3.2. Summary of the test sequences for the two considered application scenarios that are used in the comparisons. All sequences are given in the 4:2:0 chroma sampling format with a bit depth of 8 bit per sample and have a duration of 10s. The resolution is specified in luma samples The hierarchical prediction structure chosen for the entertainment application scenario provides a very high coding efficiency, but it is associated with a structural delay of eight pictures. For interactive video applications, such a high delay is not acceptable. Here, we used a dyadic low-delay hierarchical prediction structure with groups of four pictures [36] and four active reference pictures. For this coding structure, all pictures are coded in display order, but the QP is varied depending on the hierarchy level. Similarly as for the coding structure chosen for the entertainment scenario, the QP is increased by 1 from one hierarchy level to the next. Except the first picture of a video sequence, which is coded as I picture, all pictures are coded as B pictures. The QP for the I picture is decreased by | relative to the QP for the B pictures of the lowest hierarchy level. Furthermore, all pictures are coded as a single slice. For the application scenario of interactive video application, we selected six sequences with typical video conferencing content. All of these sequences have a resolution of 1280 x 720 luma samples. The chosen test sequences for both scenarios are summarized in Table 3.2. They
Table 3.3 Coding efficiency improvement for successively enabling PUs of selected sizes. The shown average bit-rate savings are measured relative to a configuration with 16 x 16 PUs only. In all configurations, the transform coding is done using a 4 X 4 transform Table 3.4 Coding efficiency improvement for successively increasing the maximum TU size. The shown average bit-rate savings are measured relative to a configuration with 4 x 4 TUs only. In all configurations, all supported CU and PU sizes are enabled and the minimum TU size is 4 x 4
Table 3.3 Coding efficiency improvement for successively enabling PUs of selected sizes. The shown average bit-rate savings are measured relative to a configuration with 16 x 16 PUs only. In all configurations, the transform coding is done using a 4 X 4 transform Table 3.4 Coding efficiency improvement for successively increasing the maximum TU size. The shown average bit-rate savings are measured relative to a configuration with 4 x 4 TUs only. In all configurations, all supported CU and PU sizes are enabled and the minimum TU size is 4 x 4
Table 3.6 Coding efficiency improvement for successively increasing the CTU size and the number of hierarchy levels in the coding tree. The shown average bit-rate savings are measured relative to a configuration with a CTU size of 16 x 16 and a minimum CU size of 8 x 8 luma samples is subdivided for the purpose of signaling intra prediction modes. The results are summarized in Table 3.5. It can be seen that providing the possibility of applying the transform across PU boundaries yields a bit-rate saving of 0.7 % for both application scenarios. By allowing more than one level of the residual quadtree, the coding efficiency can be further increased, even though the improvement is small compared to the impact of other coding options.
Table 3.6 Coding efficiency improvement for successively increasing the CTU size and the number of hierarchy levels in the coding tree. The shown average bit-rate savings are measured relative to a configuration with a CTU size of 16 x 16 and a minimum CU size of 8 x 8 luma samples is subdivided for the purpose of signaling intra prediction modes. The results are summarized in Table 3.5. It can be seen that providing the possibility of applying the transform across PU boundaries yields a bit-rate saving of 0.7 % for both application scenarios. By allowing more than one level of the residual quadtree, the coding efficiency can be further increased, even though the improvement is small compared to the impact of other coding options.
Table 3.5 Coding efficiency improvement for successively increasing the maximum depth of the residual quadtree (RQT). The shown average bit-rate savings are measured relative to a configuration in which the RQT depth is equal to 0. All supported CU, PU, and TU sizes are enabled
Table 3.5 Coding efficiency improvement for successively increasing the maximum depth of the residual quadtree (RQT). The shown average bit-rate savings are measured relative to a configuration in which the RQT depth is equal to 0. All supported CU, PU, and TU sizes are enabled
Table 3.7 Coding efficiency improvement for increasing the set of PU and TU sizes as has been done in the development of video coding standards. The shown average bit-rates are measured relative to a configuration corresponding to H.262 | MPEG-2 Video (16 x 16 PUs and 8 x 8 TUs only) restricted to 16 x 16 PUs and 8 x 8 TUs and thus corresponds to the block sizes supported in H.262 | MPEG-2 AVC. A configuration corresponding to H.263 and MPEG-4 Visual is obtained by additionally allowing PU sizes of 8 x 8 luma samples. For simulating the partitioning features of H.264 | MPEG-4 AVC, we set the maximum CTU size to 16 x 16 luma samples, allowed all subdivisions supported in HEVC, except the asymmetric partitioning modes, and additionally enabled 4 x 4 PUs by slightly modifying the syntax. Even though this configuration differs in some aspects from the actual syntax features of H.264 | MPEG-4 AVC, it still provides a suitable comparison point. Finally, in the HEVC configuration, the maximum set of supported block sizes was enabled. The average bit-rate savings relative to the H.262 | MPEG-2 Video configuration are summarized in Table 3.7. Additionally, diagrams with rate-distortion curves for a representative sequence for both application scenarios are shown in Fig. 3.8. For the tested HD sequences, the increase of partitioning modes from H.262 | MPEG-2 Video to H.264 | MPEG-4 AVC yields bit-rate savings of 7.3 and 5.7 % for the entertainment and interactive scenario, respectively. The experiment indicates that most of the coding efficiency improvements [31] that are obtained by H.264 | MPEG-4 AVC for HD material are related to other coding tools, not the additional partitioning modes. In contrast to that, a significant part of the coding gain of HEVC relative to H.264 | MPEG-4 AVC can be attributed to supporting increased block sizes for motion-compensated prediction and transform coding.
Table 3.7 Coding efficiency improvement for increasing the set of PU and TU sizes as has been done in the development of video coding standards. The shown average bit-rates are measured relative to a configuration corresponding to H.262 | MPEG-2 Video (16 x 16 PUs and 8 x 8 TUs only) restricted to 16 x 16 PUs and 8 x 8 TUs and thus corresponds to the block sizes supported in H.262 | MPEG-2 AVC. A configuration corresponding to H.263 and MPEG-4 Visual is obtained by additionally allowing PU sizes of 8 x 8 luma samples. For simulating the partitioning features of H.264 | MPEG-4 AVC, we set the maximum CTU size to 16 x 16 luma samples, allowed all subdivisions supported in HEVC, except the asymmetric partitioning modes, and additionally enabled 4 x 4 PUs by slightly modifying the syntax. Even though this configuration differs in some aspects from the actual syntax features of H.264 | MPEG-4 AVC, it still provides a suitable comparison point. Finally, in the HEVC configuration, the maximum set of supported block sizes was enabled. The average bit-rate savings relative to the H.262 | MPEG-2 Video configuration are summarized in Table 3.7. Additionally, diagrams with rate-distortion curves for a representative sequence for both application scenarios are shown in Fig. 3.8. For the tested HD sequences, the increase of partitioning modes from H.262 | MPEG-2 Video to H.264 | MPEG-4 AVC yields bit-rate savings of 7.3 and 5.7 % for the entertainment and interactive scenario, respectively. The experiment indicates that most of the coding efficiency improvements [31] that are obtained by H.264 | MPEG-4 AVC for HD material are related to other coding tools, not the additional partitioning modes. In contrast to that, a significant part of the coding gain of HEVC relative to H.264 | MPEG-4 AVC can be attributed to supporting increased block sizes for motion-compensated prediction and transform coding.
Fig. 3.9 Picture structuring in HEVC using slices, showing the CTUs belonging to independent slice segments (blue shaded) and dependent slice segments of two slices of a picture. Note that the slice boundaries are marked by a solid line, whereas the boundaries between slice segments of the same slice are marked by dashed lines
Fig. 3.9 Picture structuring in HEVC using slices, showing the CTUs belonging to independent slice segments (blue shaded) and dependent slice segments of two slices of a picture. Note that the slice boundaries are marked by a solid line, whereas the boundaries between slice segments of the same slice are marked by dashed lines
Fig. 3.11 Two different ways of fragmenting an exemplary tile partitioning using two tiles into slices and slice segments. Left: One slice including four dependent slice segments. Right: Three slices, each containing one dependent slice segment. Note that the tile boundaries are marked with bold dashed lines, the slice boundaries with bold solid lines, and the boundaries between slice segments with dotted lines. The CTUs belonging to the independent slice segments are blue shaded
Fig. 3.11 Two different ways of fragmenting an exemplary tile partitioning using two tiles into slices and slice segments. Left: One slice including four dependent slice segments. Right: Three slices, each containing one dependent slice segment. Note that the tile boundaries are marked with bold dashed lines, the slice boundaries with bold solid lines, and the boundaries between slice segments with dotted lines. The CTUs belonging to the independent slice segments are blue shaded
uffer (DPB) removal for the whole access unit, a timestamp for each DU. The DU iming gives a separate, possibly earlier timing for decoding of the DU compared o the whole picture. This has the merits that parts of the picture may be encoded yy the sender, transmitted to the receiver and decoded at the receiver earlier than if he whole access unit is processed at once. This means that this indication of earlier lecoding times of sub-pictures allows to reduce the whole end-to-end delay of a apturing, coding, transmission, decoding and presentation chain [8]. In ultra-low delay applications, such as remote-video or broadcast contribution vhere delays below picture durations are required, the encoder needs to output a aC ee Se | Ss i, i i a : ee: | a
uffer (DPB) removal for the whole access unit, a timestamp for each DU. The DU iming gives a separate, possibly earlier timing for decoding of the DU compared o the whole picture. This has the merits that parts of the picture may be encoded yy the sender, transmitted to the receiver and decoded at the receiver earlier than if he whole access unit is processed at once. This means that this indication of earlier lecoding times of sub-pictures allows to reduce the whole end-to-end delay of a apturing, coding, transmission, decoding and presentation chain [8]. In ultra-low delay applications, such as remote-video or broadcast contribution vhere delays below picture durations are required, the encoder needs to output a aC ee Se | Ss i, i i a : ee: | a
of all the defined prediction modes for all the block sizes. Due to the very large amount of different block size—prediction mode combinations all the prediction modes have been designed in a way to allow easy algorithmic implementations for arbitrary block sizes and prediction directions as discussed in the following sections.
of all the defined prediction modes for all the block sizes. Due to the very large amount of different block size—prediction mode combinations all the prediction modes have been designed in a way to allow easy algorithmic implementations for arbitrary block sizes and prediction directions as discussed in the following sections.
Fig. 4.2, Examples of 8 x 8 luma prediction blocks generated with all the HEVC intra prediction modes. Effects of the prediction post-processing can be seen on the top and left borders of the DC prediction (mode 1), top border of horizontal mode 10 and left border of vertical mode 26
Fig. 4.2, Examples of 8 x 8 luma prediction blocks generated with all the HEVC intra prediction modes. Effects of the prediction post-processing can be seen on the top and left borders of the DC prediction (mode 1), top border of horizontal mode 10 and left border of vertical mode 26
Fig. 4.3. An example of reference sample substitution process. Non-available reference samples are marked as grey: (a) reference samples before the substitution process (b) reference samples after the substitution process
Fig. 4.3. An example of reference sample substitution process. Non-available reference samples are marked as grey: (a) reference samples before the substitution process (b) reference samples after the substitution process
Fig. 4.4 Two types of filtering process of reference samples: (a) shows a three-tap filtering using two neighboring reference samples. Reference sample X is replaced by the filtered value using A, X and B while reference sample Y is replaced by the filtered value using C, Y and D. (b) Shows a strong intra smoothing process using corner reference samples. Reference sample X is replaced by a linearly filtered value using A and B while Y is replaced by a linearly filtered value using B and C
Fig. 4.4 Two types of filtering process of reference samples: (a) shows a three-tap filtering using two neighboring reference samples. Reference sample X is replaced by the filtered value using A, X and B while reference sample Y is replaced by the filtered value using C, Y and D. (b) Shows a strong intra smoothing process using corner reference samples. Reference sample X is replaced by a linearly filtered value using A and B while Y is replaced by a linearly filtered value using B and C
Table 4.2 Angular parameter A defines the directionality of each angular intra prediction mode array. For the modes with positive angular parameter A (vertical modes 26 to 33 and horizontal modes 2 to 10) the 1-D reference array is simply created by copying reference samples from the direction of the prediction:
Table 4.2 Angular parameter A defines the directionality of each angular intra prediction mode array. For the modes with positive angular parameter A (vertical modes 26 to 33 and horizontal modes 2 to 10) the 1-D reference array is simply created by copying reference samples from the direction of the prediction:
Fig. 4.5 Angle definitions of angular intra prediction in HEVC numbered from 2 to 34 and the associated displacement parameters. H and V are used to indicate the horizontal and vertical directionalities, respectively, while the numeric part of the identifier refers to the sample position displacements in 1/32 fractions of sample grid positions. Reproduced with permission from [8], © 2012 IEEE
Fig. 4.5 Angle definitions of angular intra prediction in HEVC numbered from 2 to 34 and the associated displacement parameters. H and V are used to indicate the horizontal and vertical directionalities, respectively, while the numeric part of the identifier refers to the sample position displacements in 1/32 fractions of sample grid positions. Reproduced with permission from [8], © 2012 IEEE
Table 4.3 Inverse angle parameter B as a function of angular parameter A If angular parameter A is negative for the selected mode, samples from both left and above of the block are required for the reference array. In these cases the samples in ref[x] array with non-negative indexes are populated as described above. The sample values for negative indexes are obtained by projecting the left reference column to extend the top reference row towards left, or projecting the top reference row to extend the left reference column upwards in the case of vertical and horizontal predictions, respectively. This projection is defined as follows:
Table 4.3 Inverse angle parameter B as a function of angular parameter A If angular parameter A is negative for the selected mode, samples from both left and above of the block are required for the reference array. In these cases the samples in ref[x] array with non-negative indexes are populated as described above. The sample values for negative indexes are obtained by projecting the left reference column to extend the top reference row towards left, or projecting the top reference row to extend the left reference column upwards in the case of vertical and horizontal predictions, respectively. This projection is defined as follows:
Fig. 4.7 Example of planar prediction: (a) illustrates calculation of the horizontal component, (b) illustrates generation of the vertical component and (c) provides an example of a final planar prediction created by averaging the horizontal and vertical components
Fig. 4.7 Example of planar prediction: (a) illustrates calculation of the horizontal component, (b) illustrates generation of the vertical component and (c) provides an example of a final planar prediction created by averaging the horizontal and vertical components
This design is based on the finding that often structures in the chroma signal follow those of the luma. In the case the derived mode is indicated for a chroma PU, intra prediction is performed by using the corresponding luma PU mode. Angular intra mode 34 is used to substitute the four chroma prediction modes with individual identifiers when the derived mode is one of those four modes. The substitution process is illustrated in Table 4.4.
This design is based on the finding that often structures in the chroma signal follow those of the luma. In the case the derived mode is indicated for a chroma PU, intra prediction is performed by using the corresponding luma PU mode. Angular intra mode 34 is used to substitute the four chroma prediction modes with individual identifiers when the derived mode is one of those four modes. The substitution process is illustrated in Table 4.4.
Table 4.4 Determination of chroma intra prediction mode according to luma intra prediction mode
Table 4.4 Determination of chroma intra prediction mode according to luma intra prediction mode
Table 4.5 Binarization of the syntax element intra_chroma_pred_mode with reenect to the chroma intra nrediction mode .4.3 Syntax Design for Intra Mode Coding Let three sorted most probable modes in an ascending order for luma intra mode be SMPM[0], SMPM[1] and SMPM[2], respectively. The syntax element prev_intra_luma_pred_flag specifies whether the luma intra mode is equal to one of the three most probable modes. In this case, the additional syntax element mpm_idx indicates that SMPM[mpm_idx] is the selected luma intra mode.
Table 4.5 Binarization of the syntax element intra_chroma_pred_mode with reenect to the chroma intra nrediction mode .4.3 Syntax Design for Intra Mode Coding Let three sorted most probable modes in an ascending order for luma intra mode be SMPM[0], SMPM[1] and SMPM[2], respectively. The syntax element prev_intra_luma_pred_flag specifies whether the luma intra mode is equal to one of the three most probable modes. In this case, the additional syntax element mpm_idx indicates that SMPM[mpm_idx] is the selected luma intra mode.
Similar levels of improvement over legacy codecs were reported in [16] using the JCT-VC test set and Bjgntegaard delta metrics [1]. This test set includes material mostly captured with video cameras, but it also contains some computer generated material as well as typical screen content. Table 4.7 summarizes the results of this study which analyzed performance at three different quality ranges: high (HEVC QPs 12-27), medium (HEVC QPs 22-37) and low (HEVC QPs 32-47). In these tests corresponding reference implementations were used and the resulting PSNRs for each codec were aligned by adjusting the respective quality factors. Coding efficiency improvements over H.264/AVC intra coding has been analyzed for example in [9]. Results of that study indicate that HEVC requires on average
Similar levels of improvement over legacy codecs were reported in [16] using the JCT-VC test set and Bjgntegaard delta metrics [1]. This test set includes material mostly captured with video cameras, but it also contains some computer generated material as well as typical screen content. Table 4.7 summarizes the results of this study which analyzed performance at three different quality ranges: high (HEVC QPs 12-27), medium (HEVC QPs 22-37) and low (HEVC QPs 32-47). In these tests corresponding reference implementations were used and the resulting PSNRs for each codec were aligned by adjusting the respective quality factors. Coding efficiency improvements over H.264/AVC intra coding has been analyzed for example in [9]. Results of that study indicate that HEVC requires on average
Table 4.6 Average objective and subjective bit rate reductions provided by HEVC with reenect tao TPBRG and TPBG O0N0 ac renorted in
Table 4.6 Average objective and subjective bit rate reductions provided by HEVC with reenect tao TPBRG and TPBG O0N0 ac renorted in
Fig. 5.1 Inter-picture prediction concept and parameters using a translational motion model
Fig. 5.1 Inter-picture prediction concept and parameters using a translational motion model
Fig. 5.3 Maximum number of left neighbors with motion data in HEVC with CTUO having 16 8x4 luma PBs next to CTU1 with one 6464 luma PB (left) and in H.264/AVC with MBO having four 4x4 partitions next to MB1 with one 16x 16 partition (right)
Fig. 5.3 Maximum number of left neighbors with motion data in HEVC with CTUO having 16 8x4 luma PBs next to CTU1 with one 6464 luma PB (left) and in H.264/AVC with MBO having four 4x4 partitions next to MB1 with one 16x 16 partition (right)
Fig. 5.4 Motion vector predictor and merge candidates. (a) Temporal. (b) Spatial motion vectors need to be scaled according to the temporal distances between the candidate reference picture and the current reference picture. Equation (5.3) shows how the candidate motion vector MVcang is scaled according to a scale factor. ScaleFactor is calculated in Eq. (5.4) based on the temporal distance between the current picture and the reference picture of the candidate block td and the temporal distance between the current picture and the reference picture of the current block tb. The temporal distance is expressed in terms of difference between the picture order count (POC) values which define the display order of the pictures. The scaling operation is basically the same scheme that is used for the temporal direct mode in H.264/AVC. This factoring allows pre-computation of ScaleFactor at slice-level
Fig. 5.4 Motion vector predictor and merge candidates. (a) Temporal. (b) Spatial motion vectors need to be scaled according to the temporal distances between the candidate reference picture and the current reference picture. Equation (5.3) shows how the candidate motion vector MVcang is scaled according to a scale factor. ScaleFactor is calculated in Eq. (5.4) based on the temporal distance between the current picture and the reference picture of the candidate block td and the temporal distance between the current picture and the reference picture of the current block tb. The temporal distance is expressed in terms of difference between the picture order count (POC) values which define the display order of the pictures. The scaling operation is basically the same scheme that is used for the temporal direct mode in H.264/AVC. This factoring allows pre-computation of ScaleFactor at slice-level
Fig. 5.6 Detail of a video sequence exemplifying the merge concept. (a) In the foreground, the scene contains a moving object (a pendulum) with motion indicated by the arrow. (b) A rate- distortion optimized quadtree partitioning for inter-picture prediction parameters is indicated by the white borders. (c) Only effective block borders are shown, i.e. borders that separate blocks with different motion parameters. Reproduced with permission from [12], © 2012 IEEE
Fig. 5.6 Detail of a video sequence exemplifying the merge concept. (a) In the foreground, the scene contains a moving object (a pendulum) with motion indicated by the arrow. (b) A rate- distortion optimized quadtree partitioning for inter-picture prediction parameters is indicated by the white borders. (c) Only effective block borders are shown, i.e. borders that separate blocks with different motion parameters. Reproduced with permission from [12], © 2012 IEEE
Table 5.1 Order in which motion data combinations of different merge candidates, that have been already inserted in the merge list, are tested to create combined bi- predictive merge candidates included in the merge candidate list. In the event of a lost co-located picture, the decoder could not derive the temporal candidates and hence not check whether it would be redundant. The indexing of all subsequent candidates would be affected by this.
Table 5.1 Order in which motion data combinations of different merge candidates, that have been already inserted in the merge list, are tested to create combined bi- predictive merge candidates included in the merge candidate list. In the event of a lost co-located picture, the decoder could not derive the temporal candidates and hence not check whether it would be redundant. The indexing of all subsequent candidates would be affected by this.
Table 5.2 Average bit rate savings of HEVC merge and skip mode using HM8.0
Table 5.2 Average bit rate savings of HEVC merge and skip mode using HM8.0
temporal candidate (if available) and zero MV candidates. In order to enable an encoder to trade-off parallelism and coding eff ciency, the parallel merge estimation level is adaptive and signaled « log2_ parallel merge level _minus2 in the picture parameter set. TI following MER sizes are allowed: 4x4 (no parallel merge estimation possible 8x8, 16x16, 32x32 and 64 x 64. A higher degree of parallelization, enabled t a larger MER, excludes more potential candidates from the merge candidate lis That, on the other hand, decreases the coding efficiency. The coding efficienc losses in terms of BD-rate [2] for different MER sizes are reported in [28] ar summarized in Table 5.3. Results are generated using HM5.0 [13] with rando access high efficiency (RA-HE) and low delay B high efficiency (LB-HE) codir configurations as described in [3]. When the merge estimation region is larger than a 4x4 block, another modific Fy ot, , . er i . ) oe
temporal candidate (if available) and zero MV candidates. In order to enable an encoder to trade-off parallelism and coding eff ciency, the parallel merge estimation level is adaptive and signaled « log2_ parallel merge level _minus2 in the picture parameter set. TI following MER sizes are allowed: 4x4 (no parallel merge estimation possible 8x8, 16x16, 32x32 and 64 x 64. A higher degree of parallelization, enabled t a larger MER, excludes more potential candidates from the merge candidate lis That, on the other hand, decreases the coding efficiency. The coding efficienc losses in terms of BD-rate [2] for different MER sizes are reported in [28] ar summarized in Table 5.3. Results are generated using HM5.0 [13] with rando access high efficiency (RA-HE) and low delay B high efficiency (LB-HE) codir configurations as described in [3]. When the merge estimation region is larger than a 4x4 block, another modific Fy ot, , . er i . ) oe
Fig. 5.8 Comparison of magnitude response of half-pel interpolation filters of H.264/AVC and HEVC relative to an ideal filter
Fig. 5.8 Comparison of magnitude response of half-pel interpolation filters of H.264/AVC and HEVC relative to an ideal filter
Table 5.6 Average bit rate savings of HEVC interpolation filter for the luma component using HM6.0
Table 5.6 Average bit rate savings of HEVC interpolation filter for the luma component using HM6.0
Table 5.7 Average bit rate savings of HEVC interpolation filter for the chroma component
Table 5.7 Average bit rate savings of HEVC interpolation filter for the chroma component
Fig. 6.1 Block-based hybrid video coding. (a) Encoder, (b) Decoder. C is the transform matrix and Qstep is the quantization step size. Reproduced with permission from [6]. © IEEE 2013
Fig. 6.1 Block-based hybrid video coding. (a) Encoder, (b) Decoder. C is the transform matrix and Qstep is the quantization step size. Reproduced with permission from [6]. © IEEE 2013
Table 6.1 Comparison of transform design methods factor @ is defined as dyypN'’~. As a result of careful investigation, it was decided to represent each matrix coefficient with 8 bit (including sign bit), and to choose the elements of the first basis vector to be equal to 64 (i.e. do; = 64,7 =0, ...,N —1). Note that this results in a scale factor of 2°*™” for the HEVC transform matrix when compared to the orthonormal DCT. The remaining matrix elements were hand-tuned (within the constraints of properties 4-6 of Sect. 6.2.1) to achieve a good balance between properties 1-3 of Sect. 6.2.1. The hand-tuning was performed as follows. First, the real-valued scaled DCT matrix elements, acy, were derived. Next, for each unique number in the resulting matrices, each integer value in the interval [—1.5, 1.5] around a@c;; was examined and the resulting values of oj, mj, and n; were calculated. Since there are only 31 unique numbers in the transform matrices (see Sect. 6.2.4), various permutations can be examined systematically (although not exhaustively). The final integer matrix elements were chosen to give a good compromise between all measures 0, mj, and n;. The resulting worst case values of oy, mj, and n,; are shown in the second column of Table 6.1. The norm was considered to be sufficiently close to | (i.e. the norm measure n; is sufficiently close to 0) to justify not using a non-flat default de-quantization matrix in HEVC (i.e. all transform coefficients scaled equally). nmr... a a ey ae a ay er.
Table 6.1 Comparison of transform design methods factor @ is defined as dyypN'’~. As a result of careful investigation, it was decided to represent each matrix coefficient with 8 bit (including sign bit), and to choose the elements of the first basis vector to be equal to 64 (i.e. do; = 64,7 =0, ...,N —1). Note that this results in a scale factor of 2°*™” for the HEVC transform matrix when compared to the orthonormal DCT. The remaining matrix elements were hand-tuned (within the constraints of properties 4-6 of Sect. 6.2.1) to achieve a good balance between properties 1-3 of Sect. 6.2.1. The hand-tuning was performed as follows. First, the real-valued scaled DCT matrix elements, acy, were derived. Next, for each unique number in the resulting matrices, each integer value in the interval [—1.5, 1.5] around a@c;; was examined and the resulting values of oj, mj, and n; were calculated. Since there are only 31 unique numbers in the transform matrices (see Sect. 6.2.4), various permutations can be examined systematically (although not exhaustively). The final integer matrix elements were chosen to give a good compromise between all measures 0, mj, and n;. The resulting worst case values of oy, mj, and n,; are shown in the second column of Table 6.1. The norm was considered to be sufficiently close to | (i.e. the norm measure n; is sufficiently close to 0) to justify not using a non-flat default de-quantization matrix in HEVC (i.e. all transform coefficients scaled equally). nmr... a a ey ae a ay er.
Fig. 6.3 Additional scale factors S71, S72, Siri, Sir2, Sq, Sig required to implement HEVC integer transform and quantization. (a) Forward transform and quantization, (b) inverse transform and quantization. The 2D forward and inverse transform are implemented as separable 1D column and row transforms. C is the orthonormal DCT matrix. D is the scaled approximation of the DCT matrix. M = log>(N) where N is the transform size. Reproduced with permission from [6]. © IEEE 2013
Fig. 6.3 Additional scale factors S71, S72, Siri, Sir2, Sq, Sig required to implement HEVC integer transform and quantization. (a) Forward transform and quantization, (b) inverse transform and quantization. The 2D forward and inverse transform are implemented as separable 1D column and row transforms. C is the orthonormal DCT matrix. D is the scaled approximation of the DCT matrix. M = log>(N) where N is the transform size. Reproduced with permission from [6]. © IEEE 2013
Fig. 6.4 Intermediate scaling factor determination for the forward transform so that the interme- diate and output values fit within 16-bits. B is video bit depth and M = logs(N) where N is the transform size. Worst case bit-depth analysis is done assuming a residual block with all samples having maximum amplitude equal to —2? (where B = 8 is the video bit depth), as input to the first stage of the forward transform. (a) First stage of the forward transform, (b) Second stage of the forward transform. Reproduced with permission from [6]. © IEEE 2013
Fig. 6.4 Intermediate scaling factor determination for the forward transform so that the interme- diate and output values fit within 16-bits. B is video bit depth and M = logs(N) where N is the transform size. Worst case bit-depth analysis is done assuming a residual block with all samples having maximum amplitude equal to —2? (where B = 8 is the video bit depth), as input to the first stage of the forward transform. (a) First stage of the forward transform, (b) Second stage of the forward transform. Reproduced with permission from [6]. © IEEE 2013
Fig. 6.5 Use of the inverse transform scale factors assuming the input to be the final output of Fig. 6.4. Video bit depth B = 8 (a) First stage of the inverse transform, (b) Second stage of the inverse transform. Reproduced with permission from [6]. © IEEE 2013
Fig. 6.5 Use of the inverse transform scale factors assuming the input to be the final output of Fig. 6.4. Video bit depth B = 8 (a) First stage of the inverse transform, (b) Second stage of the inverse transform. Reproduced with permission from [6]. © IEEE 2013
decoder as shown in Fig. 6.8) and the quantization matrix entry at the DC (zero-frequency) position are transmitted. HEVC also allows for prediction of a quantization matrix from another quantization matrix of the same size. The use of quantization matrix (termed as scaling matrix in HEVC) is enabled by setting the flag scaling list enabled flag in SPS. When this flag is enabled, additional flags in SPS and PPS control whether the default quantization matrices or non-default quantization matrices are used.
decoder as shown in Fig. 6.8) and the quantization matrix entry at the DC (zero-frequency) position are transmitted. HEVC also allows for prediction of a quantization matrix from another quantization matrix of the same size. The use of quantization matrix (termed as scaling matrix in HEVC) is enabled by setting the flag scaling list enabled flag in SPS. When this flag is enabled, additional flags in SPS and PPS control whether the default quantization matrices or non-default quantization matrices are used.
Table 6.4 Quantization group size for different coding tree unit sizes
Table 6.4 Quantization group size for different coding tree unit sizes
The delta QP is transmitted only in coding units with non-zero transform coefficients. If the CTU is split into coding units that are greater than the QG size, then delta QP is signaled at a coding unit (with non-zero transform coefficients) that is greater than the QG size. If the CTU is split into coding units that are smaller than the QG size, then the delta QP is signaled in the first coding unit with non-zero transform coefficients in the QG. If a QG has coding units with all zero transform coefficients (e.g. if the merge mode is used in all the coding units of the QG), then delta QP will not be signaled. The OP nredictor need for calcnlatingo the delta OP neec a camMpbjnatinn af OP Vella Lt Will Ut Ue sietiaivts. The QP predictor used for calculating the delta QP uses a combination of QP values from the left, above and the previous QG in decoding order as shown in Fig. 6.9 [21]. The QP predictor uses a combination of two predictive techniques: spatial QP prediction (from left and above QGs) and previous QP prediction. It uses spatial prediction from left and above within a CTU and uses the previous QP as predictor at the CTU boundary. This is shown in Fig. 6.9. The spatially adjacent QP values, QPLerr and QPagpove are considered to be not available when they are in a different CTU or if the current QG is at a slice/tile/picture boundary. When a spatially adjacent QP value is not available, it is replaced with the previous QP value, QPprey, in decoding order. The previous QP, QPprey, is initialized to the slice QP value at the beginning of the slice, tile or wavefront.
The delta QP is transmitted only in coding units with non-zero transform coefficients. If the CTU is split into coding units that are greater than the QG size, then delta QP is signaled at a coding unit (with non-zero transform coefficients) that is greater than the QG size. If the CTU is split into coding units that are smaller than the QG size, then the delta QP is signaled in the first coding unit with non-zero transform coefficients in the QG. If a QG has coding units with all zero transform coefficients (e.g. if the merge mode is used in all the coding units of the QG), then delta QP will not be signaled. The OP nredictor need for calcnlatingo the delta OP neec a camMpbjnatinn af OP Vella Lt Will Ut Ue sietiaivts. The QP predictor used for calculating the delta QP uses a combination of QP values from the left, above and the previous QG in decoding order as shown in Fig. 6.9 [21]. The QP predictor uses a combination of two predictive techniques: spatial QP prediction (from left and above QGs) and previous QP prediction. It uses spatial prediction from left and above within a CTU and uses the previous QP as predictor at the CTU boundary. This is shown in Fig. 6.9. The spatially adjacent QP values, QPLerr and QPagpove are considered to be not available when they are in a different CTU or if the current QG is at a slice/tile/picture boundary. When a spatially adjacent QP value is not available, it is replaced with the previous QP value, QPprey, in decoding order. The previous QP, QPprey, is initialized to the slice QP value at the beginning of the slice, tile or wavefront.
The direct 1D 4-point transform using (6.12) would require 16 multiplications and 12 additions. The 2D transform will require 128 multiplications and 96 additions. Even—Odd decomposition on the other hand requires a total of six multiplications and eight additions for 1D transform using (6.14—6.16). The 2D transform using Even—Odd decomposition will require a total of 48 multiplications and 64 additions which is 62.5 % savings in number of multiplications and 33.3 % savings in number of additions when compared to direct matrix multiplication. The 8-point 1D inverse transform is defined by the following equation:
The direct 1D 4-point transform using (6.12) would require 16 multiplications and 12 additions. The 2D transform will require 128 multiplications and 96 additions. Even—Odd decomposition on the other hand requires a total of six multiplications and eight additions for 1D transform using (6.14—6.16). The 2D transform using Even—Odd decomposition will require a total of 48 multiplications and 64 additions which is 62.5 % savings in number of multiplications and 33.3 % savings in number of additions when compared to direct matrix multiplication. The 8-point 1D inverse transform is defined by the following equation:
The number of arithmetic operations for the inverse transform can be further reduced if knowledge about zero-valued input transform coefficients is assumed. In an HEVC decoder, this information can be obtained from the entropy decoding or de-quantization process. For typical video content many blocks of size N x N will have non-zero coefficients only in a K x K low frequency sub-block. For example in [5] it was found that on average around 75 % of the transform blocks had non-zero coefficients only in K x K low frequency sub-blocks. Computations can be saved in two ways for such transform blocks. Figure 6.11 shows the first way. Columns that are completely zero need not be inverse transformed. So only K 1D IDCTs along columns needs to be carried out. However, all N rows will need to be transformed subsequently. The second way to reduce computations is by exploiting the fact that each of the column and row IDCT is on a vector that has non-zero values only in the first K locations. For example with K = N/2, x4 = x5 = x6 = x7 = 0, roughly half the computations for the inverse transformation can be eliminated by simplifying Eqs. (6.19-6.20) to
The number of arithmetic operations for the inverse transform can be further reduced if knowledge about zero-valued input transform coefficients is assumed. In an HEVC decoder, this information can be obtained from the entropy decoding or de-quantization process. For typical video content many blocks of size N x N will have non-zero coefficients only in a K x K low frequency sub-block. For example in [5] it was found that on average around 75 % of the transform blocks had non-zero coefficients only in K x K low frequency sub-blocks. Computations can be saved in two ways for such transform blocks. Figure 6.11 shows the first way. Columns that are completely zero need not be inverse transformed. So only K 1D IDCTs along columns needs to be carried out. However, all N rows will need to be transformed subsequently. The second way to reduce computations is by exploiting the fact that each of the column and row IDCT is on a vector that has non-zero values only in the first K locations. For example with K = N/2, x4 = x5 = x6 = x7 = 0, roughly half the computations for the inverse transformation can be eliminated by simplifying Eqs. (6.19-6.20) to
(6.21) resulting in a total of 28 additions. The 2D transform using Even—Odd decomposition will require a total of 352 multiplications and 448 additions. The computational complexity calculation for the 4-point and 8-point inverse transform can be extended to inverse transforms of larger sizes. In general, the resulting number of multiplications and additions (excluding the rounding opera- tions associated with the shift operations) for the two-dimensional N-point inverse transform can be shown to be tions associated with the shift operations) for the two-dimensional N-point inverse
(6.21) resulting in a total of 28 additions. The 2D transform using Even—Odd decomposition will require a total of 352 multiplications and 448 additions. The computational complexity calculation for the 4-point and 8-point inverse transform can be extended to inverse transforms of larger sizes. In general, the resulting number of multiplications and additions (excluding the rounding opera- tions associated with the shift operations) for the two-dimensional N-point inverse transform can be shown to be tions associated with the shift operations) for the two-dimensional N-point inverse
Table 6.7 BD-Rate savings of the HEVC 4 x4 and 8 x 8 transforms versus the H.264/AVC 4 x 4 and 8 x 8 transforms
Table 6.7 BD-Rate savings of the HEVC 4 x4 and 8 x 8 transforms versus the H.264/AVC 4 x 4 and 8 x 8 transforms
Table 6.6 BD-rate savings of using larger transform sizes (16 X 16 and 32 x 32) on top of the smaller transform sizes (4 x 4 and 8 x 8)
Table 6.6 BD-rate savings of using larger transform sizes (16 X 16 and 32 x 32) on top of the smaller transform sizes (4 x 4 and 8 x 8)
HEVC encoder can turn on and off each of the in-loop filters independently. Modern video coding standards try to remove as much redundancy from the coded representation of video as possible. One of the sources of redundancy is the temporal redundancy, i.e. similarity between the subsequent pictures in a video sequence. This type of redundancy is effectively removed by motion prediction. Another type of redundancy is spatial redundancy and is removed by intra- prediction from the neighboring samples and spatial transforms. In HEVC, both the motion prediction and transform coding are block-based. The size of motion- predicted blocks varies from 8 x 4 and 4 x 8, to 64 x 64 luma samples, while the size of block transforms and intra-predicted blocks varies from 4 x 4 to 32 x 32 samples.
HEVC encoder can turn on and off each of the in-loop filters independently. Modern video coding standards try to remove as much redundancy from the coded representation of video as possible. One of the sources of redundancy is the temporal redundancy, i.e. similarity between the subsequent pictures in a video sequence. This type of redundancy is effectively removed by motion prediction. Another type of redundancy is spatial redundancy and is removed by intra- prediction from the neighboring samples and spatial transforms. In HEVC, both the motion prediction and transform coding are block-based. The size of motion- predicted blocks varies from 8 x 4 and 4 x 8, to 64 x 64 luma samples, while the size of block transforms and intra-predicted blocks varies from 4 x 4 to 32 x 32 samples.
used in HEVC, coarse quantization can also create discontinuities at the block boundaries. In highly detailed areas with high-frequency content, such artifacts can be masked by the human visual system. However, in the smooth areas, discontinu- ities between the blocks are easily noticed by a viewer and may cause significant degradation of the perceived video quality. The example of a block artifact in one dimension is shown in Fig. 7.3. The horizontal axis shows the sample positions along a horizontal or vertical 1-D line, and the vertical axis shows the sample values. Deblocking filter attenuates the artifacts in the areas, where they are mostly visible, i.e. in the smooth, uniform areas. The excessive filtering in the highly detailed areas should be avoided since it can cause undesirable blurring. The artifacts in those areas are rarely noticed by the human eye, while it is also more difficult to determine whether the discontinuity is caused by a block boundary or belongs to the original signal [27]. Therefore, an important part of the deblocking filter is the deblocking filtering decisions, which determine whether a particular part of a block boundary is to be filtered. In these decisions, the HEVC deblocking filter uses the mode and motion information from the decoded bitstream as well as analyses the values of reconstructed samples on the sides of the block boundary. The strength of the deblocking filter can also be adjusted by the encoder on the picture and the slice basis. Section 7.2.2 provides a description of the HEVC deblocking filer, while
used in HEVC, coarse quantization can also create discontinuities at the block boundaries. In highly detailed areas with high-frequency content, such artifacts can be masked by the human visual system. However, in the smooth areas, discontinu- ities between the blocks are easily noticed by a viewer and may cause significant degradation of the perceived video quality. The example of a block artifact in one dimension is shown in Fig. 7.3. The horizontal axis shows the sample positions along a horizontal or vertical 1-D line, and the vertical axis shows the sample values. Deblocking filter attenuates the artifacts in the areas, where they are mostly visible, i.e. in the smooth, uniform areas. The excessive filtering in the highly detailed areas should be avoided since it can cause undesirable blurring. The artifacts in those areas are rarely noticed by the human eye, while it is also more difficult to determine whether the discontinuity is caused by a block boundary or belongs to the original signal [27]. Therefore, an important part of the deblocking filter is the deblocking filtering decisions, which determine whether a particular part of a block boundary is to be filtered. In these decisions, the HEVC deblocking filter uses the mode and motion information from the decoded bitstream as well as analyses the values of reconstructed samples on the sides of the block boundary. The strength of the deblocking filter can also be adjusted by the encoder on the picture and the slice basis. Section 7.2.2 provides a description of the HEVC deblocking filer, while
Fig. 7.4 Order of boundaries processing in HEVC and H.264/AVC deblocking. In each group (first to fourth), boundaries are processed from left to right and from top to bottom. In HEVC all vertical (horizontal) boundaries can be processed in parallel
Fig. 7.4 Order of boundaries processing in HEVC and H.264/AVC deblocking. In each group (first to fourth), boundaries are processed from left to right and from top to bottom. In HEVC all vertical (horizontal) boundaries can be processed in parallel
Fig. 7.6 Four-sample segment of vertical block boundary between adjacent blocks P and Q. Deblocking decision are made based on lines 0 and 3 (dashed)
Fig. 7.6 Four-sample segment of vertical block boundary between adjacent blocks P and Q. Deblocking decision are made based on lines 0 and 3 (dashed)
Fig. 7.5 Derivation of boundary strength (Bs) for block boundary. P denotes the left (or top) block and Q denotes the right (or bottom) block at the block boundary
Fig. 7.5 Derivation of boundary strength (Bs) for block boundary. P denotes the left (or top) block and Q denotes the right (or bottom) block at the block boundary
Fig. 7.10 Dependency of B on QP. Reproduced with permission from [37], © 2012 IEEE
Fig. 7.10 Dependency of B on QP. Reproduced with permission from [37], © 2012 IEEE
concave edges/corners, and local valleys are marked with solid circles and none- of-the-above samples with empty circles. One can observe from the figure that the distortion can be reduced by applying negative offsets to local peaks and convex comers and positive offsets to concave corners and valleys. The operation of the odge offset is based on this observation. EO uses four one-directional patterns for sample classification: horizontal, vertical, 135° diagonal, and 45° diagonal, as shown in Fig. 7.14, where label “c” represents the current sample and labels “a” and “b” represent the two neighboring samples. These four sample patterns form four EO classes. Only one EO class can be selected for each CTB that enables EO. Based on the rate-distortion optimization, one EO class is chosen and an index indicating which EO class is selected is signaled in the bitstream. For a given EO class with the specific direction, each sample inside the CTB Ly ST SNe Ye ee cee Le ee eS i. Stee ene eee ey Lemme NY PCT fe: Mey Pees MME | Ser |
concave edges/corners, and local valleys are marked with solid circles and none- of-the-above samples with empty circles. One can observe from the figure that the distortion can be reduced by applying negative offsets to local peaks and convex comers and positive offsets to concave corners and valleys. The operation of the odge offset is based on this observation. EO uses four one-directional patterns for sample classification: horizontal, vertical, 135° diagonal, and 45° diagonal, as shown in Fig. 7.14, where label “c” represents the current sample and labels “a” and “b” represent the two neighboring samples. These four sample patterns form four EO classes. Only one EO class can be selected for each CTB that enables EO. Based on the rate-distortion optimization, one EO class is chosen and an index indicating which EO class is selected is signaled in the bitstream. For a given EO class with the specific direction, each sample inside the CTB Ly ST SNe Ye ee cee Le ee eS i. Stee ene eee ey Lemme NY PCT fe: Mey Pees MME | Ser |
Fig. 7.15 Positive offsets for EO categories 1 and 2 and negative offsets for EO categories 3 and 4 results in smoothing, where the x-axis is sample index and the y-axis is sample value. Reproduced with permission from [13], © 2012 IEEE
Fig. 7.15 Positive offsets for EO categories 1 and 2 and negative offsets for EO categories 3 and 4 results in smoothing, where the x-axis is sample index and the y-axis is sample value. Reproduced with permission from [13], © 2012 IEEE
a aa a a a aa aa be signaled to the decoder. There is no constraint on the offset sign for the BO. Figure 7.16 demonstrates how the BO compensates sample intensity offset o a region. The horizontal axis denotes the sample position and the vertical axi denotes the sample value. The dotted curve represents the original samples, whil the solid curve denotes the reconstructed samples, affected by quantization errors 0 prediction residues and phase shifts because of the coded motion vectors that deviat from the true motion. As shown in Fig. 7.16, if there is a phase shift (difference between the reconstructed motion vector and the “true” motion vector, a smoot! region with a gradient may be offset with a certain value compared to the origina signal. In this example, the reconstructed samples are shifted to the left compare to the original samples, which results in a systematic negative error that can b corrected by BO for bands k, k+ 1, k+2, and k+3, where the samples rangin from k*8 to ((k+1)*8)—1 are classified as belonging to band k, and can b modified by using the corresponding offset value. In HBRVC only offeete af four cancecntive hande and the ctartinga (or minimim PMicinsn ek eimaretaae “Rees RUB Uimeawmeviires BER Baro term SARS | MRE eenaures murs aerses wie umes ironrin cir igen he mwcrmeney
a aa a a a aa aa be signaled to the decoder. There is no constraint on the offset sign for the BO. Figure 7.16 demonstrates how the BO compensates sample intensity offset o a region. The horizontal axis denotes the sample position and the vertical axi denotes the sample value. The dotted curve represents the original samples, whil the solid curve denotes the reconstructed samples, affected by quantization errors 0 prediction residues and phase shifts because of the coded motion vectors that deviat from the true motion. As shown in Fig. 7.16, if there is a phase shift (difference between the reconstructed motion vector and the “true” motion vector, a smoot! region with a gradient may be offset with a certain value compared to the origina signal. In this example, the reconstructed samples are shifted to the left compare to the original samples, which results in a systematic negative error that can b corrected by BO for bands k, k+ 1, k+2, and k+3, where the samples rangin from k*8 to ((k+1)*8)—1 are classified as belonging to band k, and can b modified by using the corresponding offset value. In HBRVC only offeete af four cancecntive hande and the ctartinga (or minimim PMicinsn ek eimaretaae “Rees RUB Uimeawmeviires BER Baro term SARS | MRE eenaures murs aerses wie umes ironrin cir igen he mwcrmeney
Fig. 7.18 CTU consists of three CTBs of color components; the current CTU can reuse SAO parameters of the left or above CTU. Reproduced with permission from [13], © 2012 IEEE Fig. 7.17 Example of sample distribution in a CTB, where BO sends the offsets of four consecutive bands. Reproduced with permission from [13], © 2012 IEEE
Fig. 7.18 CTU consists of three CTBs of color components; the current CTU can reuse SAO parameters of the left or above CTU. Reproduced with permission from [13], © 2012 IEEE Fig. 7.17 Example of sample distribution in a CTB, where BO sends the offsets of four consecutive bands. Reproduced with permission from [13], © 2012 IEEE
Fig. 7.19 Illustration of CTU-level SAO information coding when the current CTU is not merged with the CTU on the left or above. Reproduced with permission from [13], © 2012 IEEE
Fig. 7.19 Illustration of CTU-level SAO information coding when the current CTU is not merged with the CTU on the left or above. Reproduced with permission from [13], © 2012 IEEE
Fig. 7.20 Illustration of picture samples, horizontal and vertical block boundaries on the 8 x 8 grid, and those non-overlapping blocks of 8 x 8 samples (marked with dotted lines), which can be deblocked in parallel. The dashed lines mark samples used in deblocking decisions (vertical and horizontal)
Fig. 7.20 Illustration of picture samples, horizontal and vertical block boundaries on the 8 x 8 grid, and those non-overlapping blocks of 8 x 8 samples (marked with dotted lines), which can be deblocked in parallel. The dashed lines mark samples used in deblocking decisions (vertical and horizontal)
Fig. 7.21 Example of luma samples line buffer. Samples below the dashed line and above the horizontal CTU boundary should be kept in the line buffer until the unavailable samples have been reconstructed
Fig. 7.21 Example of luma samples line buffer. Samples below the dashed line and above the horizontal CTU boundary should be kept in the line buffer until the unavailable samples have been reconstructed
Table 7.4 Luma BD-rates evaluating objective effects of using deblocking filtering under various coding conditions
Table 7.4 Luma BD-rates evaluating objective effects of using deblocking filtering under various coding conditions
Fig. 7.22 Sequence BasketballDrive, Random Access, QP32: (a) deblocking turned off, (b deblocking turned on
Fig. 7.22 Sequence BasketballDrive, Random Access, QP32: (a) deblocking turned off, (b deblocking turned on
Table 7.5 Luma BD-rates evaluating objective effects of using SAO under various coding conditions
Table 7.5 Luma BD-rates evaluating objective effects of using SAO under various coding conditions
Fig. 7.25 Subjective quality comparison of RaceHorses test sequence, POC = 20, QP = 32, LP condition: (a) SAO is disabled, (b) SAO is enabled, (c) original (uncoded) sequence Fig. 7.24 Example of test sequence SliceEditing in LP condition, POC = 100, QP = 32: (a) SAO is disabled, (b) SAO is enabled
Fig. 7.25 Subjective quality comparison of RaceHorses test sequence, POC = 20, QP = 32, LP condition: (a) SAO is disabled, (b) SAO is enabled, (c) original (uncoded) sequence Fig. 7.24 Example of test sequence SliceEditing in LP condition, POC = 100, QP = 32: (a) SAO is disabled, (b) SAO is enabled
Table 7.6 Luma BD-rates evaluating the objective effects of using deblocking filter and SAO under various coding conditions signal on both sides of the boundary is flat. Therefore in HEVC, the deblocking is also applied when the signal on each side of the block boundary approximates a ramp or a slope, which can happen in smooth areas with changing luma intensity. The SAO in-loop filter attenuates ringing artifacts, which can be more pro- nounced in HEVC when larger transform sizes are used by the encoder. Moreover, SAO can also be applied to the inside samples of the large blocks, which cannot be corrected by the deblocking filter. This is especially important for HEVC because of the large transform sizes allowed in the standard.
Table 7.6 Luma BD-rates evaluating the objective effects of using deblocking filter and SAO under various coding conditions signal on both sides of the boundary is flat. Therefore in HEVC, the deblocking is also applied when the signal on each side of the block boundary approximates a ramp or a slope, which can happen in smooth areas with changing luma intensity. The SAO in-loop filter attenuates ringing artifacts, which can be more pro- nounced in HEVC when larger transform sizes are used by the encoder. Moreover, SAO can also be applied to the inside samples of the large blocks, which cannot be corrected by the deblocking filter. This is especially important for HEVC because of the large transform sizes allowed in the standard.
Table 8.1 CABAC coded syntax elements in HEVC and H.264/AVC
Table 8.1 CABAC coded syntax elements in HEVC and H.264/AVC
Fig. 8.1 CABAC block diagram (from the encoder perspective): Binarization, context modeling (including probability estimation and assignment), and binary arithmetic coding. In red: Potential throughput bottlenecks, as further discussed from the decoder perspective in Sect. 8.3.2
Fig. 8.1 CABAC block diagram (from the encoder perspective): Binarization, context modeling (including probability estimation and assignment), and binary arithmetic coding. In red: Potential throughput bottlenecks, as further discussed from the decoder perspective in Sect. 8.3.2
Table 8.2. Examples of different binarizations
Table 8.2. Examples of different binarizations
Fig. 8.2. Three key operations in CABAC (from a decoder perspective): Binarization, Context Modeling/Selection and (Binary) Arithmetic Coding. Feedback loops in the decoder are high- lighted with dashed lines
Fig. 8.2. Three key operations in CABAC (from a decoder perspective): Binarization, Context Modeling/Selection and (Binary) Arithmetic Coding. Feedback loops in the decoder are high- lighted with dashed lines
Fig. 8.3 Context speculation required to achieve 5x parallelism when processing the sig- nificance map in H.264/AVC. Notation: i = coefficient position; il = MaxNumCoeff (BlockType)—1; EOB = end of block; SIG = significant_coeff_flag; LAST = last_ significant_coeff flag Note that the feedback loops have different degrees of impact on throughput. The range update (1) and context update (2) feedback loops are simpler than the context modeling loops (3, 4) and thus do not affect throughput as severely. If the context of a bin depends on the value of another bin being decoded in parallel, then speculative computations are required, which increases area cost and critical path delay [94]. The amount of speculation can grow exponentially with the number of parallel bins, which limits the throughput that can be achieved [80]. Figure 8.3 shows an example of the speculation tree for significance map in H.264/AVC. Thus the throughput bottleneck is primarily due to the context modeling dependencies.
Fig. 8.3 Context speculation required to achieve 5x parallelism when processing the sig- nificance map in H.264/AVC. Notation: i = coefficient position; il = MaxNumCoeff (BlockType)—1; EOB = end of block; SIG = significant_coeff_flag; LAST = last_ significant_coeff flag Note that the feedback loops have different degrees of impact on throughput. The range update (1) and context update (2) feedback loops are simpler than the context modeling loops (3, 4) and thus do not affect throughput as severely. If the context of a bin depends on the value of another bin being decoded in parallel, then speculative computations are required, which increases area cost and critical path delay [94]. The amount of speculation can grow exponentially with the number of parallel bins, which limits the throughput that can be achieved [80]. Figure 8.3 shows an example of the speculation tree for significance map in H.264/AVC. Thus the throughput bottleneck is primarily due to the context modeling dependencies.
Fig. 8.4 These two examples illustrate which CTUs are terminated when slice segments are divided into substreams using tiles and wavefront parallel processing. Values of (end_of_slice_segment_flag, end_of_sub_stream_one_bit) are given for each configuration. (a) Tiles: CTU 12, 24, and 36 have (0, 1); CTU 48 (1, not signaled); and the rest of the CTUs have (0,0). (b) Wavefront parallel processing: CTU 8, 16, 24, 32 and 40 have (0, 1); CTU 48 (1, not signaled); and the rest of the CTUs have (0,0)
Fig. 8.4 These two examples illustrate which CTUs are terminated when slice segments are divided into substreams using tiles and wavefront parallel processing. Values of (end_of_slice_segment_flag, end_of_sub_stream_one_bit) are given for each configuration. (a) Tiles: CTU 12, 24, and 36 have (0, 1); CTU 48 (1, not signaled); and the rest of the CTUs have (0,0). (b) Wavefront parallel processing: CTU 8, 16, 24, 32 and 40 have (0, 1); CTU 48 (1, not signaled); and the rest of the CTUs have (0,0)
Fig. 8.6 A coding tree unit is subdivided into CUs along the associated coding quadtree. Each resulting CU may be further subdivided into PUs. The intra-coded CUs are in blue while inter- coded CUs are in orange. Note that the figure only shows the corresponding CTB, CBs, and PBs of the luma component
Fig. 8.6 A coding tree unit is subdivided into CUs along the associated coding quadtree. Each resulting CU may be further subdivided into PUs. The intra-coded CUs are in blue while inter- coded CUs are in orange. Note that the figure only shows the corresponding CTB, CBs, and PBs of the luma component
!0Note that the minimum allowed inter-coded PART_NXN size is 8 x 8, so for CU size equal to 8 x 8, the only allowed partitions are PART_2NxN and PART_Nx2N, and cMax of 2 is used for truncated unary.
!0Note that the minimum allowed inter-coded PART_NXN size is 8 x 8, so for CU size equal to 8 x 8, the only allowed partitions are PART_2NxN and PART_Nx2N, and cMax of 2 is used for truncated unary.
Table 8.4 Differences in signaling between CTU/CU layer in HEVC and MB layer in H.264/AVC
Table 8.4 Differences in signaling between CTU/CU layer in HEVC and MB layer in H.264/AVC
Table 8.5 Differences between prediction unit coding in HEVC and H.264/AVC The number of regular coded bins was reduced for intra mode coding in HEVC relative to the corresponding part in H.264/AVC, where both the flag and the 3 fixed- length bins of the remainder mode are regular coded using two separate context models. In HEVC, the flag is regular coded as well, but the remainder mode is a fixed-length 5-bin value that is entirely bypass coded. The most probable mode index (mpm_idx) is also entirely bypass coded. The number of contexts used to code intra_chroma_pred_mode is reduced from 4 to 1 for HEVC relative to H.264/AVC. To maximize the impact of fast bypass coding, the bypass coded bins for luma intra prediction mode coding within a CU are grouped together in HEVC [19]. This is beneficial when the partition mode is PART_NxN, and there are four sets of prediction modes.
Table 8.5 Differences between prediction unit coding in HEVC and H.264/AVC The number of regular coded bins was reduced for intra mode coding in HEVC relative to the corresponding part in H.264/AVC, where both the flag and the 3 fixed- length bins of the remainder mode are regular coded using two separate context models. In HEVC, the flag is regular coded as well, but the remainder mode is a fixed-length 5-bin value that is entirely bypass coded. The most probable mode index (mpm_idx) is also entirely bypass coded. The number of contexts used to code intra_chroma_pred_mode is reduced from 4 to 1 for HEVC relative to H.264/AVC. To maximize the impact of fast bypass coding, the bypass coded bins for luma intra prediction mode coding within a CU are grouped together in HEVC [19]. This is beneficial when the partition mode is PART_NxN, and there are four sets of prediction modes.
Generated bins are discriminated along the HEVC categories CTU/CU, PU, and TU as well as their corresponding counterparts in H.264/AVC Table 8.6 Distribution of bins in CABAC for HEVC and H.264/AVC under common test conditions [6, 101] and for the worst case
Generated bins are discriminated along the HEVC categories CTU/CU, PU, and TU as well as their corresponding counterparts in H.264/AVC Table 8.6 Distribution of bins in CABAC for HEVC and H.264/AVC under common test conditions [6, 101] and for the worst case
Fig. 8.8 Example of CABAC-based transform coefficient coding for a 4 x 4 transform block in H.264/AVC. Note, however, that the corresponding bins for signaling of the absolute level (in yellow) are not explicitly shown
Fig. 8.8 Example of CABAC-based transform coefficient coding for a 4 x 4 transform block in H.264/AVC. Note, however, that the corresponding bins for signaling of the absolute level (in yellow) are not explicitly shown
Fig. 8.9 Example of transform coefficient coding for a 4 x 4 transform block in HEVC. Note, however, that the corresponding bins for signaling of the “last” information (in red) and absolute level remaining (in yellow) are not explicitly shown subdivision is given for a luma CB size of 64 at RQT depth equal to 0. Table 8.7 and Fig. 8.11 illustrate an example of this configuration. Therefore, even if up to five different RQT levels are permitted, only up to three different context models are required for coding of split_transform_flag. Note that the signaling of split transform _flag at the RQT root is omitted if the quantized residual of the corresponding CU contains no non-zero transform coefficient at all, i.e., if the corresponding coded block flag at the RQOT root (see Sect. 8.6.3) is equal to 0.
Fig. 8.9 Example of transform coefficient coding for a 4 x 4 transform block in HEVC. Note, however, that the corresponding bins for signaling of the “last” information (in red) and absolute level remaining (in yellow) are not explicitly shown subdivision is given for a luma CB size of 64 at RQT depth equal to 0. Table 8.7 and Fig. 8.11 illustrate an example of this configuration. Therefore, even if up to five different RQT levels are permitted, only up to three different context models are required for coding of split_transform_flag. Note that the signaling of split transform _flag at the RQT root is omitted if the quantized residual of the corresponding CU contains no non-zero transform coefficient at all, i.e., if the corresponding coded block flag at the RQOT root (see Sect. 8.6.3) is equal to 0.
Fig. 8.10 [Illustration of residual quadtrees (one for each CU) used to signal transform units for residual coding of CUs. Note that the same relationships and comments as given in Fig. 8.6 apply here as well
Fig. 8.10 [Illustration of residual quadtrees (one for each CU) used to signal transform units for residual coding of CUs. Note that the same relationships and comments as given in Fig. 8.6 apply here as well
Fig. 8.11 Illustration of signaling of split_transform_flag, cbf_luma, cbf_cb, and cbf_cr for an RQT with depth 4. Note that at RQT depth = 0, no split_transform_flag is signaled since an implicit transform split occurs for CU of 64 as Nmax = 32. cb£_1uma is only signaled for leaf transform blocks (highlighted in red). cb£_cb and cbf_cr are signaled for the root node and all nodes where the corresponding CBF at the parent node is non-zero, except for the nodes related to TrafoSize = 4
Fig. 8.11 Illustration of signaling of split_transform_flag, cbf_luma, cbf_cb, and cbf_cr for an RQT with depth 4. Note that at RQT depth = 0, no split_transform_flag is signaled since an implicit transform split occurs for CU of 64 as Nmax = 32. cb£_1uma is only signaled for leaf transform blocks (highlighted in red). cb£_cb and cbf_cr are signaled for the root node and all nodes where the corresponding CBF at the parent node is non-zero, except for the nodes related to TrafoSize = 4
Table 8.7 Derivation of context increment (ctxInc) for split_transform_flag, cbf_luma, cbf_cb, and cbf_cr for the example in Fig. 8.11
Table 8.7 Derivation of context increment (ctxInc) for split_transform_flag, cbf_luma, cbf_cb, and cbf_cr for the example in Fig. 8.11
coding the SIG of 16 x 16 and 32 x 32 TBs are shared [81,99]. For improved coding efficiency for intra predicted CUs, so-called mode depen- dent coefficient scanning (MDCS) is used to select between vertical, horizontal, and diagonal scans based on the chosen intra prediction mode [106], as illustrated in Fig. 8.16. Table 8.8 shows how the scans are assigned based on intra prediction mode, TB size, and component. As mentioned in Sect. 8.5.2, this requires the intra mode to be decoded before decoding the corresponding transform coefficients. MDCS is only used for 4 x 4 and 8 x 8 TBs and provides coding gains of up to 1.2 %. Note that for TBs larger than 8 x 8, and for TBs of inter predicted CUs, only the diagonal scan is used.
coding the SIG of 16 x 16 and 32 x 32 TBs are shared [81,99]. For improved coding efficiency for intra predicted CUs, so-called mode depen- dent coefficient scanning (MDCS) is used to select between vertical, horizontal, and diagonal scans based on the chosen intra prediction mode [106], as illustrated in Fig. 8.16. Table 8.8 shows how the scans are assigned based on intra prediction mode, TB size, and component. As mentioned in Sect. 8.5.2, this requires the intra mode to be decoded before decoding the corresponding transform coefficients. MDCS is only used for 4 x 4 and 8 x 8 TBs and provides coding gains of up to 1.2 %. Note that for TBs larger than 8 x 8, and for TBs of inter predicted CUs, only the diagonal scan is used.
Fig. 8.16 Diagonal, vertical, and horizontal scans for 4 x 4 TBs Table 8.8 Mode dependent coefficient scanning: Mapping of intra prediction mode to scan: (0 = Diagonal, 1 = Horizontal, 2 = Vertical) for different TB sizes and components
Fig. 8.16 Diagonal, vertical, and horizontal scans for 4 x 4 TBs Table 8.8 Mode dependent coefficient scanning: Mapping of intra prediction mode to scan: (0 = Diagonal, 1 = Horizontal, 2 = Vertical) for different TB sizes and components
Table 8.9 Binarization of coordinate values of the last position Bins belonging to the bypass coded suffixes are underlined
Table 8.9 Binarization of coordinate values of the last position Bins belonging to the bypass coded suffixes are underlined
Fig. 8.17 Example of the hierarchical signaling of an 8 x 8 significance map
Fig. 8.17 Example of the hierarchical signaling of an 8 x 8 significance map
scheme, the CSBF of the neighboring right and bottom subblocks (CSBF,ign;, CSBFoottom) are used to select one of four patterns shown in Fig. 8.19: (0,0) maps to pattern 1, (1,0) to pattern 2, (0,1) to pattern 3 and (1,1) to pattern 4. The pattern maps each position within the 4 x 4 subblock to one of three contexts. As a result, there are no intrinsic dependencies for context selection of SIG within each 4 x 4 subblock.
scheme, the CSBF of the neighboring right and bottom subblocks (CSBF,ign;, CSBFoottom) are used to select one of four patterns shown in Fig. 8.19: (0,0) maps to pattern 1, (1,0) to pattern 2, (0,1) to pattern 3 and (1,1) to pattern 4. The pattern maps each position within the 4 x 4 subblock to one of three contexts. As a result, there are no intrinsic dependencies for context selection of SIG within each 4 x 4 subblock.
Table 8.11 Context selection of sig _coeff_flag based on component, TB size, scan order (Table 8.8), position of subblock within the TB (Fig. 8.15), and position based context index within 4 x 4TB or subblock (SubIdx) (Fig. 8.12 or 8.19, resp.)
Table 8.11 Context selection of sig _coeff_flag based on component, TB size, scan order (Table 8.8), position of subblock within the TB (Fig. 8.15), and position based context index within 4 x 4TB or subblock (SubIdx) (Fig. 8.12 or 8.19, resp.)
the subblock. Note that context 0 is used to code the sig_coeff_ flag ofthe DC position of all luma TBs, and context 27 is used for the DC position of all chrom: IBs. 5.5 Absolute Coefficient Level and Coefficient Sign in HEVC, parsing of transform coefficient level information is performed subblock- py-subblock using up to five scan passes for each subblock. The first scan pass is Jevoted to the SIG flags, as already explained in Sects. 8.6.4.1 and 8.6.4.3. In the second and third pass, the two additional flags coef£f_abs_level_greater1l _flag (ALGI) and coeff_abs_ level _greater2_ flag (ALG2) are con- jitionally parsed, indicating for each relevant scan position if the corresponding absolute value of the coefficient level, i.e., the absolute level (AL) is greater than | and 2, respectively. However, only up to 8 ALGI flags and one ALG2 flag are rransmitted for each subblock, as will be explained in more detail below. In the third scan pass, the sign of each significant level is signaled with the possible exception
the subblock. Note that context 0 is used to code the sig_coeff_ flag ofthe DC position of all luma TBs, and context 27 is used for the DC position of all chrom: IBs. 5.5 Absolute Coefficient Level and Coefficient Sign in HEVC, parsing of transform coefficient level information is performed subblock- py-subblock using up to five scan passes for each subblock. The first scan pass is Jevoted to the SIG flags, as already explained in Sects. 8.6.4.1 and 8.6.4.3. In the second and third pass, the two additional flags coef£f_abs_level_greater1l _flag (ALGI) and coeff_abs_ level _greater2_ flag (ALG2) are con- jitionally parsed, indicating for each relevant scan position if the corresponding absolute value of the coefficient level, i.e., the absolute level (AL) is greater than | and 2, respectively. However, only up to 8 ALGI flags and one ALG2 flag are rransmitted for each subblock, as will be explained in more detail below. In the third scan pass, the sign of each significant level is signaled with the possible exception
Fig. 8.22 Illustration of the adaptive binarization scheme for absolute levels in HEVC consisting of a concatenation of the three elementary binarizations TrU, TRk, and EGk, the latter two with varying order k and k + 1, respectively (0 < k < 4). The two variable thresholds Bo and B, specify the (variable) transition points between them Fig. 8.22 Illustration of the adaptive binarization scheme for absolute levels in HEVC consisting
Fig. 8.22 Illustration of the adaptive binarization scheme for absolute levels in HEVC consisting of a concatenation of the three elementary binarizations TrU, TRk, and EGk, the latter two with varying order k and k + 1, respectively (0 < k < 4). The two variable thresholds Bo and B, specify the (variable) transition points between them Fig. 8.22 Illustration of the adaptive binarization scheme for absolute levels in HEVC consisting
Table 8.12 Binarization of the absolute level z for the choice of parameters By = 2, Bj = 6, and k = 0, corresponding to a concatenation of TrU with cMax = 3, zero-order Truncated Rice (TRk) with cMax = 4, and first-order Exp-Golomb (EGk)
Table 8.12 Binarization of the absolute level z for the choice of parameters By = 2, Bj = 6, and k = 0, corresponding to a concatenation of TrU with cMax = 3, zero-order Truncated Rice (TRk) with cMax = 4, and first-order Exp-Golomb (EGk)
The suffix has a length of k bins when the value NV of ALRem is less than (3 << k); otherwise, it has a length of [log,(((N — (3 << k)) >> k) + 1)] +k bins. In this table, the suffix bins are shown in terms of x and C, where each x represents a bin, and C represents a fixed length bin string of length k Table 8.14 An alternative representation of coeff_abs_level_remaining (ALRem) bina- rization as a concatenation of a unary prefix and fixed length suffix
The suffix has a length of k bins when the value NV of ALRem is less than (3 << k); otherwise, it has a length of [log,(((N — (3 << k)) >> k) + 1)] +k bins. In this table, the suffix bins are shown in terms of x and C, where each x represents a bin, and C represents a fixed length bin string of length k Table 8.14 An alternative representation of coeff_abs_level_remaining (ALRem) bina- rization as a concatenation of a unary prefix and fixed length suffix
Table 8.13 Binarization of the absolute level z for the choice of parameters By = 1, B) = 9, and k = 1, corresponding to a concatenation of TrU with cMax = 2, first-order Truncated Rice (TRk) with cMax = 8, and second-order Exp-Golomb (EGk)
Table 8.13 Binarization of the absolute level z for the choice of parameters By = 1, B) = 9, and k = 1, corresponding to a concatenation of TrU with cMax = 2, first-order Truncated Rice (TRk) with cMax = 8, and second-order Exp-Golomb (EGk)
where NumT1 denotes the accumulated number of encoded/decoded trailing 1’s .e., absolute levels equal to 1, and NumG1 denotes the accumulated number o sncoded/decoded levels with absolute value greater than 1, both computed alon; he reverse scanning pattern of the subblock up to (but not including) the curren scan position. Note that both NumT1 and NumG1 are initialized with the value of ( it the beginning of the subblock scan of ALG1 flags. After each encoded/decode: ALGI flag with the value of 0, NumT1 is incremented by 1, while after eacl encoded/decoded ALGI1 flag with the value of 1, NumGI1 is incremented by 1 “igure 8.24 shows the flow chart for context increment computation of ALG1. Since the statistics of trailing 1’s may differ from subblock to subblock as wel Figure 6.24 snows the flow chart ror context increment computation of ALG!I. Since the statistics of trailing 1’s may differ from subblock to subblock as well as for subblocks belonging to different components or different locations within the TB, different sets of context models are provided, both for ALG1 and ALG2, as already mentioned above. For subblocks belonging to the luma component, 2 separate sets are used for subblocks containing the DC of the TB, i.e., for the top left subblocks in a TB. Another two sets are given for luma subblocks containing no DC as well as two additional sets for chroma subblocks. Depending on the value of ctxInc(ALG1) for the last decoded ALG! flag in the preceding subblock, the two members of each of the relevant sets related to luma DC, luma non-DC, and chroma are selected: One for the case of ctxInc(ALG1) = 0 and the other for the case of ctxInc(ALG1) > 0. Thus, a total number of 30 context models are used for coding of ALGI and ALG2: 6-4 = 24 for ALGI and 6 for ALG2. Interestingly enough, there was a 4x reduction (from 120 to 30) in the total number of contexts used for coding of the ALG1 and ALG2 flags during the development from HM3.0 to HM6.0 at virtually no loss in coding efficiency.
where NumT1 denotes the accumulated number of encoded/decoded trailing 1’s .e., absolute levels equal to 1, and NumG1 denotes the accumulated number o sncoded/decoded levels with absolute value greater than 1, both computed alon; he reverse scanning pattern of the subblock up to (but not including) the curren scan position. Note that both NumT1 and NumG1 are initialized with the value of ( it the beginning of the subblock scan of ALG1 flags. After each encoded/decode: ALGI flag with the value of 0, NumT1 is incremented by 1, while after eacl encoded/decoded ALGI1 flag with the value of 1, NumGI1 is incremented by 1 “igure 8.24 shows the flow chart for context increment computation of ALG1. Since the statistics of trailing 1’s may differ from subblock to subblock as wel Figure 6.24 snows the flow chart ror context increment computation of ALG!I. Since the statistics of trailing 1’s may differ from subblock to subblock as well as for subblocks belonging to different components or different locations within the TB, different sets of context models are provided, both for ALG1 and ALG2, as already mentioned above. For subblocks belonging to the luma component, 2 separate sets are used for subblocks containing the DC of the TB, i.e., for the top left subblocks in a TB. Another two sets are given for luma subblocks containing no DC as well as two additional sets for chroma subblocks. Depending on the value of ctxInc(ALG1) for the last decoded ALG! flag in the preceding subblock, the two members of each of the relevant sets related to luma DC, luma non-DC, and chroma are selected: One for the case of ctxInc(ALG1) = 0 and the other for the case of ctxInc(ALG1) > 0. Thus, a total number of 30 context models are used for coding of ALGI and ALG2: 6-4 = 24 for ALGI and 6 for ALG2. Interestingly enough, there was a 4x reduction (from 120 to 30) in the total number of contexts used for coding of the ALG1 and ALG2 flags during the development from HM3.0 to HM6.0 at virtually no loss in coding efficiency.
Fig. 8.25 Flow chart for coding the syntax elements of absolute level minus | and sign fora 4 Xx 4 subblock in HEVC
Fig. 8.25 Flow chart for coding the syntax elements of absolute level minus | and sign fora 4 Xx 4 subblock in HEVC
Table 8.15 Differences between CABAC for different TB sizes in HEVC and H.264/AVC umber of contexts for H.264/AVC includes separate models for both SIG and LAST in frame and field coding mode (denoted by “x2” in the two shtmost columns for the corresponding syntax elements)
Table 8.15 Differences between CABAC for different TB sizes in HEVC and H.264/AVC umber of contexts for H.264/AVC includes separate models for both SIG and LAST in frame and field coding mode (denoted by “x2” in the two shtmost columns for the corresponding syntax elements)
Table 8.16 BD-rate performance of CABAC transform coefficient coding in HEVC compared tc the extended CABAC transform coefficient coding of H.264/AVC Table 8.17 Coding efficiency impact of adopted TU coding tools
Table 8.16 BD-rate performance of CABAC transform coefficient coding in HEVC compared tc the extended CABAC transform coefficient coding of H.264/AVC Table 8.17 Coding efficiency impact of adopted TU coding tools
Note that positive BD-rate values indicate coding loss and negative BD-rate values indicate coding gain
Note that positive BD-rate values indicate coding loss and negative BD-rate values indicate coding gain
Table 8.18 Distribution of regular coded, bypass and termination bins for CABAC in H.264/AVC (JM-16.2) and HEVC (HM8.0) under common test conditions [6, 101]
Table 8.18 Distribution of regular coded, bypass and termination bins for CABAC in H.264/AVC (JM-16.2) and HEVC (HM8.0) under common test conditions [6, 101]
Note max total bins includes termination mode bins, but does not include impact of bit limit per CTU or macroblock
Note max total bins includes termination mode bins, but does not include impact of bit limit per CTU or macroblock
Table 8.20 Summary of throughput improvement techniques with references to related standard contributions 8.8.2.5 Reduce Total Number of Bins
Table 8.20 Summary of throughput improvement techniques with references to related standard contributions 8.8.2.5 Reduce Total Number of Bins
Fig. 9.2. The prediction structure of the random access configuration Fig. 9.1 The prediction structure of the intra-only configuration
Fig. 9.2. The prediction structure of the random access configuration Fig. 9.1 The prediction structure of the intra-only configuration
Fig. 9.3. The prediction structure of low-delay P and B configurations Test sequences are defined according to the picture size and applications and they are classified into six classes (class A to class F). Class A is the set of sequences with higher resolution than 1080p HDTV. The sequences are used to evaluate the coding performance of 4K/8K video. To reduce computation time, picture sizes are cropped to 2,560 x 1,600 pixels. Class B is for coding performance evaluation of 1080p HDTV and the set contains HDTV sequences, with a picture size of 1,920 x 1,080 pixels. Classes C and D are the set of test sequences with picture sizes of 832 x 480 pixels and 416 x 240 pixels, respectively. Test sequences in these two classes are for coding performance evaluation of mobile applications. Class E is the set of test sequences with a picture size of 1,280 x 720 pixels. It is used to evaluate coding performance of low-latency applications such as visual communications. CTC, in addition, defines class F sequences for coding performance evaluation of non-camera captured content such as video screen content, containing, for example, text and computer graphic. The test sequences are listed in Table 9.1. In addition to the test sequences defined in CTC, 4K test sequences listed in
Fig. 9.3. The prediction structure of low-delay P and B configurations Test sequences are defined according to the picture size and applications and they are classified into six classes (class A to class F). Class A is the set of sequences with higher resolution than 1080p HDTV. The sequences are used to evaluate the coding performance of 4K/8K video. To reduce computation time, picture sizes are cropped to 2,560 x 1,600 pixels. Class B is for coding performance evaluation of 1080p HDTV and the set contains HDTV sequences, with a picture size of 1,920 x 1,080 pixels. Classes C and D are the set of test sequences with picture sizes of 832 x 480 pixels and 416 x 240 pixels, respectively. Test sequences in these two classes are for coding performance evaluation of mobile applications. Class E is the set of test sequences with a picture size of 1,280 x 720 pixels. It is used to evaluate coding performance of low-latency applications such as visual communications. CTC, in addition, defines class F sequences for coding performance evaluation of non-camera captured content such as video screen content, containing, for example, text and computer graphic. The test sequences are listed in Table 9.1. In addition to the test sequences defined in CTC, 4K test sequences listed in
Table 9.1 Test sequences Table 9.2 4K Test sequences
Table 9.1 Test sequences Table 9.2 4K Test sequences
Table 9.4 Test conditions
Table 9.4 Test conditions
Fig. 9.7 R—D curve of U (Four People, RA-Main)
Fig. 9.7 R—D curve of U (Four People, RA-Main)
results demonstrate that HEVC intra coding outperforms encoders for still images with an average bit rate reduction ranging from 16 % (compared to JPEG 2000 4:4:4) up to 43 % (compared to JPEG).
results demonstrate that HEVC intra coding outperforms encoders for still images with an average bit rate reduction ranging from 16 % (compared to JPEG 2000 4:4:4) up to 43 % (compared to JPEG).
As shown in Fig. 9.10, the BTC structure of the DSCQS method contains twc consecutive pairs of presentations. At first, a mid-grey screen with the letter “A” in the middle, is displayed for a second followed by a 10-s presentation of video clip—either original or impaired. Then, a mid-grey screen with the letter “B” appears followed by a 10-s presentation of the second video clip. Similar process is repeated during the second round of presentation by changing letters A and B to A* and B*, instead. Finally, a message is displayed for 5 s instructing the viewers tc vote.
As shown in Fig. 9.10, the BTC structure of the DSCQS method contains twc consecutive pairs of presentations. At first, a mid-grey screen with the letter “A” in the middle, is displayed for a second followed by a 10-s presentation of video clip—either original or impaired. Then, a mid-grey screen with the letter “B” appears followed by a 10-s presentation of the second video clip. Similar process is repeated during the second round of presentation by changing letters A and B to A* and B*, instead. Finally, a message is displayed for 5 s instructing the viewers tc vote.
9.4.2.7 Rate Distortion Curves Results 9.4.2.8 Average Bit Rate Difference
9.4.2.7 Rate Distortion Curves Results 9.4.2.8 Average Bit Rate Difference
Table 9.10 Bit rate differences of tested bitstreams
Table 9.10 Bit rate differences of tested bitstreams
9.5.1.1 Encoder Settings
9.5.1.1 Encoder Settings
Table 9.12 Subjective viewing comparison results for sequences encoded using the HEVC and AVC encoders (B better, SB slightly better, VP no preference)
Table 9.12 Subjective viewing comparison results for sequences encoded using the HEVC and AVC encoders (B better, SB slightly better, VP no preference)
luma pixels and two rows of chroma pixels (per chroma component) due to the deblocking filter’s support. The size of these buffers is proportional to the widtl of the picture. Further, if the picture is split into multiple tile rows, each tile rov needs a separate line buffer if the rows are to be processed in parallel. Tiles alsc need column buffers to handle data dependencies between them in the horizonta direction. Traditionally, line buffers have been implemented using on-chip SRAM However, for very large picture sizes, it may be necessary to store them in the dense: off-chip DRAM. This results in an area and power trade-off as communicating t the off-chip | DRAM takes much more power.
luma pixels and two rows of chroma pixels (per chroma component) due to the deblocking filter’s support. The size of these buffers is proportional to the widtl of the picture. Further, if the picture is split into multiple tile rows, each tile rov needs a separate line buffer if the rows are to be processed in parallel. Tiles alsc need column buffers to handle data dependencies between them in the horizonta direction. Traditionally, line buffers have been implemented using on-chip SRAM However, for very large picture sizes, it may be necessary to store them in the dense: off-chip DRAM. This results in an area and power trade-off as communicating t the off-chip | DRAM takes much more power.
Fig. 10.1 System pipelining for HEVC decoder. Coeff buffer saves 20kB of SRAM by TU pipelining. Connections to Line Buffers are omitted in the figure for clarity (see Fig. 10.3 for details)
Fig. 10.1 System pipelining for HEVC decoder. Coeff buffer saves 20kB of SRAM by TU pipelining. Connections to Line Buffers are omitted in the figure for clarity (see Fig. 10.3 for details)
Fig. 10.2 Split system pipeline to address variable DRAM latency. Within each group, variable- sized pipeline block-level pipelining is used
Fig. 10.2 Split system pipeline to address variable DRAM latency. Within each group, variable- sized pipeline block-level pipelining is used
Fig. 10.3 Memory management in second pipeline group. A 2-VPB ping-pong and a 3-VPB rotating buffer are used as pipeline buffers. A single-port SRAM is used for pixel linebuffer to save area and access to it is arbitrated. Marked bus widths denote SRAM data width in pixels
Fig. 10.3 Memory management in second pipeline group. A 2-VPB ping-pong and a 3-VPB rotating buffer are used as pipeline buffers. A single-port SRAM is used for pixel linebuffer to save area and access to it is arbitrated. Marked bus widths denote SRAM data width in pixels
Fig. 10.4 Top-level architecture for CABAC. Memories are shown with grey boxes probability should be read from the context memory based on the type of the syntax element being processed, as well as the bin index, neighboring information (top neighbor is read from a line buffer), and component (i.e., luma or chroma). When the probability used to decode a bin is read from the context memory, it is referred to as a regular coded bin; otherwise, a probability of 0.5 is assumed and the bin is referred to as bypass coded. Bypass coded bins can be decoded much faster than regular coded bins. After each regular coded bin is decoded, an updated context with the updated probability estimate is sent back to the context memory. Finally, the debinarization module maps the sequence of bins to a syntax element. The CABAC in HEVC was redesigned for higher throughput [17]. Specifically,
Fig. 10.4 Top-level architecture for CABAC. Memories are shown with grey boxes probability should be read from the context memory based on the type of the syntax element being processed, as well as the bin index, neighboring information (top neighbor is read from a line buffer), and component (i.e., luma or chroma). When the probability used to decode a bin is read from the context memory, it is referred to as a regular coded bin; otherwise, a probability of 0.5 is assumed and the bin is referred to as bypass coded. Bypass coded bins can be decoded much faster than regular coded bins. After each regular coded bin is decoded, an updated context with the updated probability estimate is sent back to the context memory. Finally, the debinarization module maps the sequence of bins to a syntax element. The CABAC in HEVC was redesigned for higher throughput [17]. Specifically,
Fig. 10.5 Possible high-level architectures for inverse transform with 2 pixel/cycle throughput. Bus-widths are in pixels. (a) Separate row and column transform stages. (b) 1-D transform stage shared by row and column transform
Fig. 10.5 Possible high-level architectures for inverse transform with 2 pixel/cycle throughput. Bus-widths are in pixels. (a) Separate row and column transform stages. (b) 1-D transform stage shared by row and column transform
Fig. 10.6 Mapping a 32 x 32 TU to four SRAM banks for transpose operation. The color of each pixel denotes the bank and the number denotes the bank address
Fig. 10.6 Mapping a 32 x 32 TU to four SRAM banks for transpose operation. The color of each pixel denotes the bank and the number denotes the bank address
processed, the transform module must wait for it to be written to the SRAM before it can begin processing the row. This results in a delay of nine cycles for 32 x 32 TU. In general, for an NXN TU, this delay is equal to N/4 + 1 cycles. This results in a pipeline stall of 1.75—25 % cycles depending on the TU size. This stall can be avoided through the use of a row cache that stores the first N + 4 pixels in registers. As shown in Fig. 10.7b, the row cache is read for the first nine cycles of the row transforms while the last solumn.t is s being stored in the SRAM. Ey So a en oe
processed, the transform module must wait for it to be written to the SRAM before it can begin processing the row. This results in a delay of nine cycles for 32 x 32 TU. In general, for an NXN TU, this delay is equal to N/4 + 1 cycles. This results in a pipeline stall of 1.75—25 % cycles depending on the TU size. This stall can be avoided through the use of a row cache that stores the first N + 4 pixels in registers. As shown in Fig. 10.7b, the row cache is read for the first nine cycles of the row transforms while the last solumn.t is s being stored in the SRAM. Ey So a en oe
Table 10.2 Area reduction by exploiting unique operations case, only two of these inputs are available per cycle. So, it is enough to perform a partial 2 x 16 matrix multiplication every cycle and accumulate the outputs over eight cycles. In general, this would require 32 full multipliers and 32 lookup tables to store the matrix. However, knowing that the matrix has only 15 unique numbers, we can simply instantiate 15 constant multipliers with some negators and multiplexers to implement the matrix multiplication. This is shown for the 4 x 4 odd matrix multiplication (Eq.(10.1)) of the 8-pt IDCT in Fig. 10.8b. The area savings are shown in Table 10.2.
Table 10.2 Area reduction by exploiting unique operations case, only two of these inputs are available per cycle. So, it is enough to perform a partial 2 x 16 matrix multiplication every cycle and accumulate the outputs over eight cycles. In general, this would require 32 full multipliers and 32 lookup tables to store the matrix. However, knowing that the matrix has only 15 unique numbers, we can simply instantiate 15 constant multipliers with some negators and multiplexers to implement the matrix multiplication. This is shown for the 4 x 4 odd matrix multiplication (Eq.(10.1)) of the 8-pt IDCT in Fig. 10.8b. The area savings are shown in Table 10.2.
Fig. 10.8 4x4 matrix multiplication in Eq. (10.1) without and with unique operations. (a) Generic implementation. (b) Exploiting unique operations
Fig. 10.8 4x4 matrix multiplication in Eq. (10.1) without and with unique operations. (a) Generic implementation. (b) Exploiting unique operations
Values in brackets denote overhead over the block size. Costs are for uni-prediction only. For bi-prediction, all the costs are doubled
Values in brackets denote overhead over the block size. Costs are for uni-prediction only. For bi-prediction, all the costs are doubled
Fig. 10.9 Interpolation process for a pixel at a fractional location x = 1/4, y = 3/4 Table 10.5 Example costs for interpolating a block of pixels
Fig. 10.9 Interpolation process for a pixel at a fractional location x = 1/4, y = 3/4 Table 10.5 Example costs for interpolating a block of pixels
Some PU types are restricted to uni-prediction while other types can use either Table 10.6 Number of horizontal interpolations for each PU type
Some PU types are restricted to uni-prediction while other types can use either Table 10.6 Number of horizontal interpolations for each PU type
Fig. 10.11 Unified luma and chroma interpolation filters with inputs reordered. The coefficients for x3 (in dashed box) can be implemented with two adders and three multiplexers as shown ir Fig. 10.12 A reorder of the filter inputs is first applied to reduce complexity based on symmetry as shown in Fig. 10.11. Note that two sets of the chroma filter coefficients are placed in x; and x¢, instead of x2 and xs, due to the similarity with the luma coefficients 4 and 1. There are only seven cases left. The design principle adopted here is to optimize TMMCM coefficients for each filter input. As an example, the design for x3 is shown in Fig. 10.12. View: itlese caiewmersnn |) 2a slliadd smmwommebntianwe Chea aotiistwdia Tease oct weed thew
Fig. 10.11 Unified luma and chroma interpolation filters with inputs reordered. The coefficients for x3 (in dashed box) can be implemented with two adders and three multiplexers as shown ir Fig. 10.12 A reorder of the filter inputs is first applied to reduce complexity based on symmetry as shown in Fig. 10.11. Note that two sets of the chroma filter coefficients are placed in x; and x¢, instead of x2 and xs, due to the similarity with the luma coefficients 4 and 1. There are only seven cases left. The design principle adopted here is to optimize TMMCM coefficients for each filter input. As an example, the design for x3 is shown in Fig. 10.12. View: itlese caiewmersnn |) 2a slliadd smmwommebntianwe Chea aotiistwdia Tease oct weed thew
Table 10.7 Gate counts of the described and reference designs for the x3, x4, and x2|x5 TMMCM in the vertical filter based on 40 nm process synthesis results
Table 10.7 Gate counts of the described and reference designs for the x3, x4, and x2|x5 TMMCM in the vertical filter based on 40 nm process synthesis results
The reference designs for x3 and x4 are generated by [15], and the reference for x2|xs5 is designing x2 and x5 separately
The reference designs for x3 and x4 are generated by [15], and the reference for x2|xs5 is designing x2 and x5 separately
Since most of the gates are for the 2-D filter, its gate count is given in more detail in Table 10.9. For the 2-D filter, the horizontal and vertical filters occupy the most, and the area of horizontal ones is nearly one half of that of vertical ones due to their smaller internal bitwidth. This implementation does not include all PU Types used in the HEVC standard (Asymmetric Motion Partitions 32 x 8, 8 x 32, 16x 4,4 16 are not implemented), and so, uses only eight horizontal and eight vertical filters.
Since most of the gates are for the 2-D filter, its gate count is given in more detail in Table 10.9. For the 2-D filter, the horizontal and vertical filters occupy the most, and the area of horizontal ones is nearly one half of that of vertical ones due to their smaller internal bitwidth. This implementation does not include all PU Types used in the HEVC standard (Asymmetric Motion Partitions 32 x 8, 8 x 32, 16x 4,4 16 are not implemented), and so, uses only eight horizontal and eight vertical filters.
Fig. 10.14 Latency Aware DRAM mapping. 128 8 x 4 MAUs arranged in raster scan order make up one block. The twisted structure increases the horizontal distance between two rows in the same bank. Note how the MAU columns are partitioned into four datapaths (based on the last 2 bits of column address) for the four-parallel cache architecture
Fig. 10.14 Latency Aware DRAM mapping. 128 8 x 4 MAUs arranged in raster scan order make up one block. The twisted structure increases the horizontal distance between two rows in the same bank. Note how the MAU columns are partitioned into four datapaths (based on the last 2 bits of column address) for the four-parallel cache architecture
Fig. 10.15 Cache hit rate as a function of CTU size, (a) cache line geometry, (b) cache-size and (c) associativity. Experiments averaged over six sequences—Basketball Drive, Park Scene, Tennis, Crowd Run, Old Town Cross and Park Joy. The first are Full HD (240 pictures each) and the last three are 4K Ultra HD (120 pictures each). CTU size of 64 is used for the cache-size and associativity experiments
Fig. 10.15 Cache hit rate as a function of CTU size, (a) cache line geometry, (b) cache-size and (c) associativity. Experiments averaged over six sequences—Basketball Drive, Park Scene, Tennis, Crowd Run, Old Town Cross and Park Joy. The first are Full HD (240 pictures each) and the last three are 4K Ultra HD (120 pictures each). CTU size of 64 is used for the cache-size and associativity experiments
Table 10.10 Comparison of twisted 2D mapping and direct 2D mapping
Table 10.10 Comparison of twisted 2D mapping and direct 2D mapping
Fig. 10.16 Example of MC cache dispatch for a 23 x 23 reference region of a 16 X 16 PU. Seven cycles are required to fetch the 28 MAU at 4 MAU per cycle. Note that the dispatch region and the four parallel cache datapaths may be misaligned, thus requiring a reordering. For example, the region in this figure starts from datapath #1
Fig. 10.16 Example of MC cache dispatch for a 23 x 23 reference region of a 16 X 16 PU. Seven cycles are required to fetch the 28 MAU at 4 MAU per cycle. Note that the dispatch region and the four parallel cache datapaths may be misaligned, thus requiring a reordering. For example, the region in this figure starts from datapath #1
Fig. 10.17 Proposed four-parallel MC cache architecture with four independent datapaths. The hazard detection circuit is shown in detail
Fig. 10.17 Proposed four-parallel MC cache architecture with four independent datapaths. The hazard detection circuit is shown in detail
Fig. 10.18 Comparison of DDR3 bandwidth and power consumption across three scenarios. RS mapping maps all the MAUs in a raster scan order. ACT corresponds to the power and band- width induced by DRAM Precharge/Activate operations. (a) Bandwidth comparison. (b) Power comparison. (c) BW across sequences
Fig. 10.18 Comparison of DDR3 bandwidth and power consumption across three scenarios. RS mapping maps all the MAUs in a raster scan order. ACT corresponds to the power and band- width induced by DRAM Precharge/Activate operations. (a) Bandwidth comparison. (b) Power comparison. (c) BW across sequences
Fig. 10.21 Top-level architecture of deblocking filter The top-level architecture of the deblocking filter is shown in Fig. 10.21. The transpose memory needs to be only 8 x 8 pixels (as compared to 32 x 32 pixels for inverse transform). Hence it is possible to implement it using registers. For a very high throughput design which filters an entire 8 x 8 block in one cycle [30], it is possible to eliminate the transpose memory completely and have a purely combinational design.
Fig. 10.21 Top-level architecture of deblocking filter The top-level architecture of the deblocking filter is shown in Fig. 10.21. The transpose memory needs to be only 8 x 8 pixels (as compared to 32 x 32 pixels for inverse transform). Hence it is possible to implement it using registers. For a very high throughput design which filters an entire 8 x 8 block in one cycle [30], it is possible to eliminate the transpose memory completely and have a purely combinational design.
A decoder test chip was implemented in [9] with a core size of 1.77 mm? in 40nm CMOS, comprising 715K logic gates and 124kB of on-chip SRAM. Figure 10.22 shows the micrograph of the test chip. It is compliant to HEVC Test Model (HM) 4.0, and the supported decoding tools in HEVC Working Draft (WD) 4 are listed in Table 10.14 along with the main specs. The main differences from the final version of HEVC are that SAO is absent and Context-Adaptive Variable
A decoder test chip was implemented in [9] with a core size of 1.77 mm? in 40nm CMOS, comprising 715K logic gates and 124kB of on-chip SRAM. Figure 10.22 shows the micrograph of the test chip. It is compliant to HEVC Test Model (HM) 4.0, and the supported decoding tools in HEVC Working Draft (WD) 4 are listed in Table 10.14 along with the main specs. The main differences from the final version of HEVC are that SAO is absent and Context-Adaptive Variable
Fig. 10.23 Core power is measured for six different combinations—Random Access and Loy Delay encoder configurations each with all three sizes of coding tree units. The core power is mort or less constant due to our unified design Table 10.14 Chip specifications
Fig. 10.23 Core power is measured for six different combinations—Random Access and Loy Delay encoder configurations each with all three sizes of coding tree units. The core power is mort or less constant due to our unified design Table 10.14 Chip specifications
Fig. 10.24 Logic and SRAM utilization for each processing engine. (a) Logic utilization in kgates (total 715 kgate). (b) SRAM utilization in kbits (total 1,018 kbit)
Fig. 10.24 Logic and SRAM utilization for each processing engine. (a) Logic utilization in kgates (total 715 kgate). (b) SRAM utilization in kbits (total 1,018 kbit)
Fig. 10.25 Relative power consumption of processing engines and SRAMs from post-layout simulation with bi-prediction
Fig. 10.25 Relative power consumption of processing engines and SRAMs from post-layout simulation with bi-prediction
Table 10.15 Comparison with state-of-the-art video decoders
Table 10.15 Comparison with state-of-the-art video decoders
Fig. 11.1 Encoder system block diagram 11.2.1 Top Level System Diagram
Fig. 11.1 Encoder system block diagram 11.2.1 Top Level System Diagram
In this work, the IME architecture uses a parallel-PU IME architecture based on 2D SAD adder tree to meet the high throughput requirement [10], as illustrated in Fig. 11.3. In parallel-PU IME, all of the PU blocks inside current CTU are done in parallel. Instead of looping over all the pixels inside the PU block, the cost for bigger PU can be simply derived from the cost of the sub-divided PU (i.e. a bottom up approach). For example, retrieving the SAD cost of 16 x 8 PU can be simply done by adding the two co-located 8 x 8 PUs. By utilizing the 2D SAD adder tree, we may retrieve all PU costs for certain motion vectors at once. However, there are dependencies among PUs, such as reference motion vectors from the neighboring PUs used to calculate the predictor motion vector to use in motion vector cost calculation. To enable parallelism, we estimate the motion vector predictors from the nearest available vectors just outside of the current CTU. For the IME search algorithm, we need to select one of the two advanced motion vectors predictors
In this work, the IME architecture uses a parallel-PU IME architecture based on 2D SAD adder tree to meet the high throughput requirement [10], as illustrated in Fig. 11.3. In parallel-PU IME, all of the PU blocks inside current CTU are done in parallel. Instead of looping over all the pixels inside the PU block, the cost for bigger PU can be simply derived from the cost of the sub-divided PU (i.e. a bottom up approach). For example, retrieving the SAD cost of 16 x 8 PU can be simply done by adding the two co-located 8 x 8 PUs. By utilizing the 2D SAD adder tree, we may retrieve all PU costs for certain motion vectors at once. However, there are dependencies among PUs, such as reference motion vectors from the neighboring PUs used to calculate the predictor motion vector to use in motion vector cost calculation. To enable parallelism, we estimate the motion vector predictors from the nearest available vectors just outside of the current CTU. For the IME search algorithm, we need to select one of the two advanced motion vectors predictors
Fig. 11.2. HEVC inter prediction architecture
Fig. 11.2. HEVC inter prediction architecture
Fig. 11.4 The schedule of interleaving interpolation in FME
Fig. 11.4 The schedule of interleaving interpolation in FME
Fig. 11.5 Illustration of central-quarter fractional motion estimation algorithm. The white, light- gray, dark-gray circles are the best integer-pixel candidates, half-pixel candidates, and quarter- pixel candidates, respectively. The numbers represent the passes. (a) Two-step FME; (b) Central- quarter one-pass FME. The 25 candidates insides the dark square are processed in parallel
Fig. 11.5 Illustration of central-quarter fractional motion estimation algorithm. The white, light- gray, dark-gray circles are the best integer-pixel candidates, half-pixel candidates, and quarter- pixel candidates, respectively. The numbers represent the passes. (a) Two-step FME; (b) Central- quarter one-pass FME. The 25 candidates insides the dark square are processed in parallel
Fig. 11.6 The schemes of searching range data reuse (a) Level C (b) Level D
Fig. 11.6 The schemes of searching range data reuse (a) Level C (b) Level D
Fig. 11.7 Level C+ searching range data reuse (a) HF2V3 scan (b) HF2V4 scan
Fig. 11.7 Level C+ searching range data reuse (a) HF2V3 scan (b) HF2V4 scan
EMB: External Memory Bandwidth of reference frame SRy: horizontal search range SRy: vertical search range N: current CTU size n: zigzag stitch number W: frame widtl Table 11.1 The comparison of different data reuse schemes for ME [8]
EMB: External Memory Bandwidth of reference frame SRy: horizontal search range SRy: vertical search range N: current CTU size n: zigzag stitch number W: frame widtl Table 11.1 The comparison of different data reuse schemes for ME [8]
Fig. 11.9 Reference memory hierarchy subsystem architecture
Fig. 11.9 Reference memory hierarchy subsystem architecture
bandwidth and internal bandwidth. We can see the data characteristics at various levels of data granularity as illustrated in Fig. 11.10. For a given CU, IME and FME that use fast algorithms may not access the whole search window memory. Instead, only small portions of the search range are accessed. At the module level, IME and FME do not require data outside the real search region. Larger memory results in larger area, higher power, and higher area, hence it is not efficient to store the whole search range for IME and FME use. For this reason, we use a multi-level and multiple reference memory with each level optimally resized for the best efficiency. A large L3 reference SRAM is used to enable level C+ /level D style buffering for lowest bandwidth overhead. For every pixel, the reference memory access reaches one read per frame if deep level C+ or level D is used. To support high concurrent access on the memory ports at the module level, we use L2 and LI! SRAM. The ME reference prefetch unit would fill the L2 SRAM for IME usage, and the L2 buffer for FME reference broadcasting unit. For IME that uses subsampling, the SRAM must be stored in subsampled pattern. The SRAM for IME is filled according to subsampling order for storing reference pixels. The FME reference broadcasting unit fills fully sampled LI SRAM with data from the L2 buffer. With this scheme, all the concurrent access requirements from IME and FME are supported. The memory bandwidth is also minimized. yr. 2.44%. gto. ge gg pm
bandwidth and internal bandwidth. We can see the data characteristics at various levels of data granularity as illustrated in Fig. 11.10. For a given CU, IME and FME that use fast algorithms may not access the whole search window memory. Instead, only small portions of the search range are accessed. At the module level, IME and FME do not require data outside the real search region. Larger memory results in larger area, higher power, and higher area, hence it is not efficient to store the whole search range for IME and FME use. For this reason, we use a multi-level and multiple reference memory with each level optimally resized for the best efficiency. A large L3 reference SRAM is used to enable level C+ /level D style buffering for lowest bandwidth overhead. For every pixel, the reference memory access reaches one read per frame if deep level C+ or level D is used. To support high concurrent access on the memory ports at the module level, we use L2 and LI! SRAM. The ME reference prefetch unit would fill the L2 SRAM for IME usage, and the L2 buffer for FME reference broadcasting unit. For IME that uses subsampling, the SRAM must be stored in subsampled pattern. The SRAM for IME is filled according to subsampling order for storing reference pixels. The FME reference broadcasting unit fills fully sampled LI SRAM with data from the L2 buffer. With this scheme, all the concurrent access requirements from IME and FME are supported. The memory bandwidth is also minimized. yr. 2.44%. gto. ge gg pm
For the top L3 level memory, we did an analysis on the strategy for level C, level C+, and level D. The analysis result is shown in Fig. 11.12. Note that the level C+ strategy uses a non-raster scan CU order and requires frame-level pipeline at entropy stage. As can be seen in Fig. 11.12, the bandwidth of level C+ strategy is bigger initially, and drops down quickly below that of Level C strategy as the SRAM size increases. That is because frame-level pipelining is a must for level C+ strategy, so frame-level I/O overhead dominates at the beginning. However, the bandwidth gain for level C+ strategy quickly overrides the penalty from frame-level entropy pipelining. Depending on the requirement, level C+ or level D strategy is more suitable for L3 level memory.
For the top L3 level memory, we did an analysis on the strategy for level C, level C+, and level D. The analysis result is shown in Fig. 11.12. Note that the level C+ strategy uses a non-raster scan CU order and requires frame-level pipeline at entropy stage. As can be seen in Fig. 11.12, the bandwidth of level C+ strategy is bigger initially, and drops down quickly below that of Level C strategy as the SRAM size increases. That is because frame-level pipelining is a must for level C+ strategy, so frame-level I/O overhead dominates at the beginning. However, the bandwidth gain for level C+ strategy quickly overrides the penalty from frame-level entropy pipelining. Depending on the requirement, level C+ or level D strategy is more suitable for L3 level memory.
Data scheduling for each stage is explained in the following. Before start, reference frame data within the maximum supported search range are loaded into L3 SRAM through system bus. In stage 1, the ME prefetching unit reads the actually used data from the L3 SRAM and writes them to the L2 SRAMs. There are two kinds of L2 SRAMs with different data ordering. The ME prefetching unit prepares the data in both kinds of order and writes them to the corresponding L2 SRAMs. After that, the connections to the L2 SRAM and the L1 SRAM are exchanged by altering the multiplexer configuration. The L2 SRAM in stage 1 will act as the L1 SRAM and be connected to the IME engine in stage 2. Thus, the written data in stage 1 can be accessed by the IME engine in stage 2. In stage 2, the IME engine reads out the data in the LI SRAM with subsampled pattern. The FME reference broadcasting unit reads from the first L1 SRAM in fully sampled pattern and broadcasts the data to the other 11 L1 SRAMs. In stage 3, all the 12 L1 SRAMs are filled with reference frame data and are able to support three sets of FME engines with full sampled access.
Data scheduling for each stage is explained in the following. Before start, reference frame data within the maximum supported search range are loaded into L3 SRAM through system bus. In stage 1, the ME prefetching unit reads the actually used data from the L3 SRAM and writes them to the L2 SRAMs. There are two kinds of L2 SRAMs with different data ordering. The ME prefetching unit prepares the data in both kinds of order and writes them to the corresponding L2 SRAMs. After that, the connections to the L2 SRAM and the L1 SRAM are exchanged by altering the multiplexer configuration. The L2 SRAM in stage 1 will act as the L1 SRAM and be connected to the IME engine in stage 2. Thus, the written data in stage 1 can be accessed by the IME engine in stage 2. In stage 2, the IME engine reads out the data in the LI SRAM with subsampled pattern. The FME reference broadcasting unit reads from the first L1 SRAM in fully sampled pattern and broadcasts the data to the other 11 L1 SRAMs. In stage 3, all the 12 L1 SRAMs are filled with reference frame data and are able to support three sets of FME engines with full sampled access.
Table 11.2 Fast intra prediction results. All intra configuration is used modes for each PU size. The PU size may range from 4 x 4 to 64 x 64 depending on encoder configuration (for the 64 x 64 PU, the intra prediction actually happens at a 32 x 32 level since the maximum transform size supported is 32 x 32). The complexity is much higher and results in considerable hardware costs. In addition, HEVC allows the use of intra/inter CU, and consequently PU, inside the same CTU. The prediction dependency among PU is quite complex and limits the hardware parallelism. For hardware implementation of intra prediction, an architecture for intra 4 x 4 prediction with flexible reference sample selection is proposed in [28]. A full intra architecture is also proposed in [33]. In high throughput applications, intra prediction performance still needs to be improved. Here we apply two methods. Firstly, we use reduced mode search to lower the required computation at algorithm level. Next, hybrid open-closed loop intra prediction removes the dependency of reconstructed pixels and enables PU-level parallelism.
Table 11.2 Fast intra prediction results. All intra configuration is used modes for each PU size. The PU size may range from 4 x 4 to 64 x 64 depending on encoder configuration (for the 64 x 64 PU, the intra prediction actually happens at a 32 x 32 level since the maximum transform size supported is 32 x 32). The complexity is much higher and results in considerable hardware costs. In addition, HEVC allows the use of intra/inter CU, and consequently PU, inside the same CTU. The prediction dependency among PU is quite complex and limits the hardware parallelism. For hardware implementation of intra prediction, an architecture for intra 4 x 4 prediction with flexible reference sample selection is proposed in [28]. A full intra architecture is also proposed in [33]. In high throughput applications, intra prediction performance still needs to be improved. Here we apply two methods. Firstly, we use reduced mode search to lower the required computation at algorithm level. Next, hybrid open-closed loop intra prediction removes the dependency of reconstructed pixels and enables PU-level parallelism.
Firstly, fast RDO is done. To save computation overhead, fast RDO selects several candidates among intra prediction directions and inter prediction motion vectors and modes for each depth level. In fast RDO, the rate is determined by mode bits and motion vectors, and the distortion is calculated by the sum of absolute difference (SAD) or the sum of absolute transformed difference (SATD). Although fast RDO is not accurate, it is still able to prune the less probable cases. Fast RDO selects totally more than 30 modes depending on encoder configurations. Then, the full RDO costs are estimated and compared for these modes. In full RDO process, all the residues are transformed, quantized, inverse quantized, and inverse transformed to produce the reconstructed differences. Distortion is then calculated by the sum of squared difference (SSD). The prediction information and residue coefficients will go through CABAC bit estimator to obtain bit rate if estimated mode is selected. After that, final decision between the modes is made by Lagrangian cost with SSD distortion and estimated CABAC bit rate to optimize the trade-off.
Firstly, fast RDO is done. To save computation overhead, fast RDO selects several candidates among intra prediction directions and inter prediction motion vectors and modes for each depth level. In fast RDO, the rate is determined by mode bits and motion vectors, and the distortion is calculated by the sum of absolute difference (SAD) or the sum of absolute transformed difference (SATD). Although fast RDO is not accurate, it is still able to prune the less probable cases. Fast RDO selects totally more than 30 modes depending on encoder configurations. Then, the full RDO costs are estimated and compared for these modes. In full RDO process, all the residues are transformed, quantized, inverse quantized, and inverse transformed to produce the reconstructed differences. Distortion is then calculated by the sum of squared difference (SSD). The prediction information and residue coefficients will go through CABAC bit estimator to obtain bit rate if estimated mode is selected. After that, final decision between the modes is made by Lagrangian cost with SSD distortion and estimated CABAC bit rate to optimize the trade-off.
also used a mode pre-decision scheme at IME to reduce the computation for FME. However, the BD rate increase for the fast RDO only method is quite high in HEVC. If we cancel all the full RDO and use only fast RDO in HM, the BD rate increases 10—15 % in intra frames and may even increase to more than 40 % in inter frames. Therefore, it seems that it is quite harmful to eliminate the full RDO completely due to inaccurate prediction of rate-distortion cost. To keep the coding quality while reducing cost for RDO process, hardware encoder should have a more limited number of full RDO, but still keep the important decisions to full RDO. Ficure 11.16 shows the proposed two-step RDO algorithm flow. Since it is
also used a mode pre-decision scheme at IME to reduce the computation for FME. However, the BD rate increase for the fast RDO only method is quite high in HEVC. If we cancel all the full RDO and use only fast RDO in HM, the BD rate increases 10—15 % in intra frames and may even increase to more than 40 % in inter frames. Therefore, it seems that it is quite harmful to eliminate the full RDO completely due to inaccurate prediction of rate-distortion cost. To keep the coding quality while reducing cost for RDO process, hardware encoder should have a more limited number of full RDO, but still keep the important decisions to full RDO. Ficure 11.16 shows the proposed two-step RDO algorithm flow. Since it is
Table 11.3 Bit estimation algorithm comparison state memory between modes. The issue for sharing state memory is that states are updated in arithmetic coding process. To eliminate this issue, we use fixed state memory that is not updated during bit estimation. However, if the states used in the bit estimation are too different from the actual states, the bit estimation will not be accurate enough and cause low quality decision in HCMD. Hence, we keep the context fixed at a CTU-level. bit rate increase for this scheme is limited [17]. The states are the same in CABAC at the beginning of the CTU. During the bit estimation process inside CTU, the states are not updated. After the final mode decision is made, the bits for the selected mode are traversed and the final states are updated. For more simplification, we also uses bit estimation table in JCTVC-G763 [1] for bits look-up instead of arithmetic encoder in context-fixed scheme. For quality comparison, the BD-rate differences vs. CABAC-based bit estimation
Table 11.3 Bit estimation algorithm comparison state memory between modes. The issue for sharing state memory is that states are updated in arithmetic coding process. To eliminate this issue, we use fixed state memory that is not updated during bit estimation. However, if the states used in the bit estimation are too different from the actual states, the bit estimation will not be accurate enough and cause low quality decision in HCMD. Hence, we keep the context fixed at a CTU-level. bit rate increase for this scheme is limited [17]. The states are the same in CABAC at the beginning of the CTU. During the bit estimation process inside CTU, the states are not updated. After the final mode decision is made, the bits for the selected mode are traversed and the final states are updated. For more simplification, we also uses bit estimation table in JCTVC-G763 [1] for bits look-up instead of arithmetic encoder in context-fixed scheme. For quality comparison, the BD-rate differences vs. CABAC-based bit estimation
hroughput CABAC design and parallelism design of CABAC are discussed. For CABAC hardware design, there are two-stage [3], three-stage [32], and ‘our-stage [12, 19, 20, 34] pipelines. Four-stage is mostly used in recent high hroughput designs. Figure 11.18 shows the four-stage overall CABAC architecture ‘or H.264/AVC. Syntax finite-state machine (FSM) controls the coding order of the syntax elements. The data access module prepares the required data for binarizer and ontext modeler. Binarizer will convert original syntax elements to binary streams. Context modeler determines the next context states to be used. After that, binary irithmetic coding is applied.
hroughput CABAC design and parallelism design of CABAC are discussed. For CABAC hardware design, there are two-stage [3], three-stage [32], and ‘our-stage [12, 19, 20, 34] pipelines. Four-stage is mostly used in recent high hroughput designs. Figure 11.18 shows the four-stage overall CABAC architecture ‘or H.264/AVC. Syntax finite-state machine (FSM) controls the coding order of the syntax elements. The data access module prepares the required data for binarizer and ontext modeler. Binarizer will convert original syntax elements to binary streams. Context modeler determines the next context states to be used. After that, binary irithmetic coding is applied.
Typical arithmetic coder can be further partitioned into four main stages: State, Range, Low and Output. State stage will update MPS and state as shown in Fig. 11.19. Range and Low stage will update the range and low values for arithmetic encoding. Output stage will be in charge of outputting the bitstream. Normalization is performed on range and low after encoding each bin so that they can be represented with a fixed 9-bit precision. We can see data dependencies between the four blocks. With this architecture, one bin per cycle is achieved. To achieve better throughput, pre-normalization circuit may be used to reduce normalization critical path [44]. In HEVC, more than 20 % of the bins are bypass bins. Since the range update circuit and the context model are not affected in bypass coding, a bypass bin spitting (BPBS) scheme can be applied to split the process from the bin stream and remerge into the bitstream before the low update stage [44]. Bast Khieh se2cAlnBAn annlesatiaAme tha “heaviohent AL Anatwes CCA BAL’ 46 HAF
Typical arithmetic coder can be further partitioned into four main stages: State, Range, Low and Output. State stage will update MPS and state as shown in Fig. 11.19. Range and Low stage will update the range and low values for arithmetic encoding. Output stage will be in charge of outputting the bitstream. Normalization is performed on range and low after encoding each bin so that they can be represented with a fixed 9-bit precision. We can see data dependencies between the four blocks. With this architecture, one bin per cycle is achieved. To achieve better throughput, pre-normalization circuit may be used to reduce normalization critical path [44]. In HEVC, more than 20 % of the bins are bypass bins. Since the range update circuit and the context model are not affected in bypass coding, a bypass bin spitting (BPBS) scheme can be applied to split the process from the bin stream and remerge into the bitstream before the low update stage [44]. Bast Khieh se2cAlnBAn annlesatiaAme tha “heaviohent AL Anatwes CCA BAL’ 46 HAF
Fig. 11.20 Multi-bin (a) overall block diagram (b) state forwarding circuit
Fig. 11.20 Multi-bin (a) overall block diagram (b) state forwarding circuit
Fig. 11.22. Branch imbalance of four-stage pipeline architecture in H.264/AVC [11] Fig. 11.21 Simplified architecture of multi-bin binary arithmetic encoder
Fig. 11.22. Branch imbalance of four-stage pipeline architecture in H.264/AVC [11] Fig. 11.21 Simplified architecture of multi-bin binary arithmetic encoder
For one MPS encoder and one LPS encoder, the throughput per cycle in traditional architecture is always one bin only. But it is possible to fully utilize both encoders to code two bins in one cycle. The corresponding architecture can be easily configured as in Fig. 11.23a and in a more simplified form as in Fig. 11.23b. Now, two bins are simultaneously checked in one cycle. If the two bins are MPS and LPS in order, both M coder and L coder are active. If the two bins are both MPS, only the M coder is active to encode the first bin, and the second bin will be coded in the next cvcle.
For one MPS encoder and one LPS encoder, the throughput per cycle in traditional architecture is always one bin only. But it is possible to fully utilize both encoders to code two bins in one cycle. The corresponding architecture can be easily configured as in Fig. 11.23a and in a more simplified form as in Fig. 11.23b. Now, two bins are simultaneously checked in one cycle. If the two bins are MPS and LPS in order, both M coder and L coder are active. If the two bins are both MPS, only the M coder is active to encode the first bin, and the second bin will be coded in the next cvcle.
Table 11.4 Comparison of H.264/AVC and HEVC encoders Table 11.5 Summary of modifications
Table 11.4 Comparison of H.264/AVC and HEVC encoders Table 11.5 Summary of modifications
Fig. 11.25 RD curve comparison for 8K sequence. Both sequences are cropped to 2,560 x 1,600 and converted to 8 bit per channel. JM results are also included to show the coding gain over the previous H.264/AVC encoders, where LC stands for low complexity mode decision (i.e. fast RDO only), and I stands for intra mode only. (a) Steam locomotive train. (b) Nebuta festival * Including HM4 frame-level loop filters, which is removed from final standards
Fig. 11.25 RD curve comparison for 8K sequence. Both sequences are cropped to 2,560 x 1,600 and converted to 8 bit per channel. JM results are also included to show the coding gain over the previous H.264/AVC encoders, where LC stands for low complexity mode decision (i.e. fast RDO only), and I stands for intra mode only. (a) Steam locomotive train. (b) Nebuta festival * Including HM4 frame-level loop filters, which is removed from final standards
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved