lvpengwei’s Blog

学习历程，生活点滴。

如何获取VideoToolbox的reorder Size

| Comments

Decoder 的区别

FFmpeg 和 MediaCodec 解码的时候，送数据的顺序是 dts，出数据的顺序是 pts，而 VideoToolbox 是送一个出一个，没有按照 pts 来出数据，需要我们自己排序。

去网上查资料的时候，发现有很多不同的方式

  1. sps.max_num_ref_frames
  2. sps.vui.max_num_reorder_frames
  3. 通过 sps.level 计算
  4. 直接设置为4

通过测试几个文件的 sps 发现 max_num_ref_frames 不是很准

  1. max_num_ref_frames=0; max_num_reorder_frames=2
  2. max_num_ref_frames=9; max_num_reorder_frames=2

sps.max_num_ref_frames

取 max_num_ref_frames 的有两个播放器，ijkplayer 和 ThumbPlayer

ijkplayer

ijkplayer 的逻辑是先取 sps.max_num_ref_frames，然后再取最小值2，最大值5。

1
2
3

fmt_desc->max_ref_frames = FFMAX(fmt_desc->max_ref_frames, 2);

fmt_desc->max_ref_frames = FFMIN(fmt_desc->max_ref_frames, 5);

主要代码在下面两个文件。
IJKVideoToolBoxAsync.m
h264_sps_parser.h

ThumbPlayer

ThumbPlayer 的逻辑是取 sps.max_num_ref_frames，如果没有设置为 10。

sps.vui.max_num_reorder_frames

取 max_num_reorder_frames 的有三个

  1. Chrome
  2. vlc
  3. MediaCodec

Chrome

Chrome 的主要代码如下，代码文件在vt_video_decode_accelerator_mac.cc 。

  1. 先判断 pocType，为 2 直接返回不需要排序
  2. 再判断是否有 vuiParameters，取 max_num_reorder_frames
  3. 然后是特定的 profile，不需要排序
  4. 最后返回 max_dpb_frames 的 默认值16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

int32_t ComputeReorderWindow(const H264SPS* sps) {
  // When |pic_order_cnt_type| == 2, decode order always matches presentation
  // order.
  // TODO(sandersd): For |pic_order_cnt_type| == 1, analyze the delta cycle to
  // find the minimum required reorder window.
  if (sps->pic_order_cnt_type == 2)
    return 0;

  // TODO(sandersd): Compute MaxDpbFrames.
  int32_t max_dpb_frames = 16;

  // See AVC spec section E.2.1 definition of |max_num_reorder_frames|.
  if (sps->vui_parameters_present_flag && sps->bitstream_restriction_flag) {
    return std::min(sps->max_num_reorder_frames, max_dpb_frames);
  } else if (sps->constraint_set3_flag) {
    if (sps->profile_idc == 44 || sps->profile_idc == 86 ||
        sps->profile_idc == 100 || sps->profile_idc == 110 ||
        sps->profile_idc == 122 || sps->profile_idc == 244) {
      return 0;
    }
  }
  return max_dpb_frames;
}

vlc

vlc 的逻辑和 chrome 类似，多了一个根据 level 计算 max_dpb_frames

  1. 判断是否有 vuiParameters，取 max_num_reorder_frames
  2. 然后是特定的 profile，不需要排序
  3. 最后计算 max_dpb_frames

代码文件在h264_nal.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

static uint8_t h264_get_max_dpb_frames( const h264_sequence_parameter_set_t *p_sps )
{
    const h264_level_limits_t *limits = h264_get_level_limits( p_sps );
    if( limits )
    {
        unsigned i_frame_height_in_mbs = ( p_sps->pic_height_in_map_units_minus1 + 1 ) *
                                         ( 2 - p_sps->frame_mbs_only_flag );
        unsigned i_den = ( p_sps->pic_width_in_mbs_minus1 + 1 ) * i_frame_height_in_mbs;
        uint8_t i_max_dpb_frames = limits->i_max_dpb_mbs / i_den;
        if( i_max_dpb_frames < 16 )
            return i_max_dpb_frames;
    }
    return 16;
}

bool h264_get_dpb_values( const h264_sequence_parameter_set_t *p_sps,
                          uint8_t *pi_depth, unsigned *pi_delay )
{
    uint8_t i_max_num_reorder_frames = p_sps->vui.i_max_num_reorder_frames;
    if( !p_sps->vui.b_bitstream_restriction_flag )
    {
        switch( p_sps->i_profile ) /* E-2.1 */
        {
            case PROFILE_H264_BASELINE:
                i_max_num_reorder_frames = 0; /* only I & P */
                break;
            case PROFILE_H264_CAVLC_INTRA:
            case PROFILE_H264_SVC_HIGH:
            case PROFILE_H264_HIGH:
            case PROFILE_H264_HIGH_10:
            case PROFILE_H264_HIGH_422:
            case PROFILE_H264_HIGH_444_PREDICTIVE:
                if( p_sps->i_constraint_set_flags & H264_CONSTRAINT_SET_FLAG(3) )
                {
                    i_max_num_reorder_frames = 0; /* all IDR */
                    break;
                }
                /* fallthrough */
            default:
                i_max_num_reorder_frames = h264_get_max_dpb_frames( p_sps );
                break;
        }
    }

    *pi_depth = i_max_num_reorder_frames;
    *pi_delay = 0;

    return true;
}

MediaCodec

MediaCodec 和 vlc/Chrome也差不多，计算max_dpb_frames的时候考虑了max_num_ref_frames和max_dec_frame_buffering

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114

bool H264Decoder::ProcessSPS(int sps_id, bool* need_new_buffers) {
  DVLOG(4) << "Processing SPS id:" << sps_id;

  const H264SPS* sps = parser_.GetSPS(sps_id);
  if (!sps)
    return false;

  *need_new_buffers = false;

  if (sps->frame_mbs_only_flag == 0) {
    DVLOG(1) << "frame_mbs_only_flag != 1 not supported";
    return false;
  }

  Size new_pic_size = sps->GetCodedSize().value_or(Size());
  if (new_pic_size.IsEmpty()) {
    DVLOG(1) << "Invalid picture size";
    return false;
  }

  int width_mb = new_pic_size.width() / 16;
  int height_mb = new_pic_size.height() / 16;

  // Verify that the values are not too large before multiplying.
  if (std::numeric_limits<int>::max() / width_mb < height_mb) {
    DVLOG(1) << "Picture size is too big: " << new_pic_size.ToString();
    return false;
  }

  int level = sps->level_idc;
  int max_dpb_mbs = LevelToMaxDpbMbs(level);
  if (max_dpb_mbs == 0)
    return false;

  // MaxDpbFrames from level limits per spec.
  size_t max_dpb_frames = std::min(max_dpb_mbs / (width_mb * height_mb),
                                   static_cast<int>(H264DPB::kDPBMaxSize));
  DVLOG(1) << "MaxDpbFrames: " << max_dpb_frames
           << ", max_num_ref_frames: " << sps->max_num_ref_frames
           << ", max_dec_frame_buffering: " << sps->max_dec_frame_buffering;

  // Set DPB size to at least the level limit, or what the stream requires.
  size_t max_dpb_size =
      std::max(static_cast<int>(max_dpb_frames),
               std::max(sps->max_num_ref_frames, sps->max_dec_frame_buffering));
  // Some non-conforming streams specify more frames are needed than the current
  // level limit. Allow this, but only up to the maximum number of reference
  // frames allowed per spec.
  DVLOG_IF(1, max_dpb_size > max_dpb_frames)
      << "Invalid stream, DPB size > MaxDpbFrames";
  if (max_dpb_size == 0 || max_dpb_size > H264DPB::kDPBMaxSize) {
    DVLOG(1) << "Invalid DPB size: " << max_dpb_size;
    return false;
  }

  if ((pic_size_ != new_pic_size) || (dpb_.max_num_pics() != max_dpb_size)) {
    if (!Flush())
      return false;
    DVLOG(1) << "Codec level: " << level << ", DPB size: " << max_dpb_size
             << ", Picture size: " << new_pic_size.ToString();
    *need_new_buffers = true;
    pic_size_ = new_pic_size;
    dpb_.set_max_num_pics(max_dpb_size);
  }

  Rect new_visible_rect = sps->GetVisibleRect().value_or(Rect());
  if (visible_rect_ != new_visible_rect) {
    DVLOG(2) << "New visible rect: " << new_visible_rect.ToString();
    visible_rect_ = new_visible_rect;
  }

  if (!UpdateMaxNumReorderFrames(sps))
    return false;
  DVLOG(1) << "max_num_reorder_frames: " << max_num_reorder_frames_;

  return true;
}

bool H264Decoder::UpdateMaxNumReorderFrames(const H264SPS* sps) {
  if (sps->vui_parameters_present_flag && sps->bitstream_restriction_flag) {
    max_num_reorder_frames_ =
        base::checked_cast<size_t>(sps->max_num_reorder_frames);
    if (max_num_reorder_frames_ > dpb_.max_num_pics()) {
      DVLOG(1)
          << "max_num_reorder_frames present, but larger than MaxDpbFrames ("
          << max_num_reorder_frames_ << " > " << dpb_.max_num_pics() << ")";
      max_num_reorder_frames_ = 0;
      return false;
    }
    return true;
  }

  // max_num_reorder_frames not present, infer from profile/constraints
  // (see VUI semantics in spec).
  if (sps->constraint_set3_flag) {
    switch (sps->profile_idc) {
      case 44:
      case 86:
      case 100:
      case 110:
      case 122:
      case 244:
        max_num_reorder_frames_ = 0;
        break;
      default:
        max_num_reorder_frames_ = dpb_.max_num_pics();
        break;
    }
  } else {
    max_num_reorder_frames_ = dpb_.max_num_pics();
  }

  return true;
}

sps.level 计算

vlc和MediaCodec 都计算得出dpb.max_num_pics，拿这个值保底
gst-plugins-bad 只通过 level 计算，计算部分和 MediaCodec一样。

设置为4

iOS解码关于视频中带B帧排序问题

HEVC

vlc 中还有 HEVC(H265) 视频获取 max_num_reorder 的方式，代码文件在hevc_nal.c

FFmpeg

h264
h265

总结

  • Chrome，vlc，MediaCodec的策略几乎一致，MediaCodec逻辑最完整。
  • vlc还处理了hevc的max_num_reorder

Link

ijkplayer
Chrome
vlc
Android
FFmpeg
iOS解码关于视频中带B帧排序问题

Comments