webRTC中timing信息的使用

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"webRTC解決方案實現了P2P的音視頻通信，其中有關timing的幾個問題值得歸納總結。開始本文之前建議先行閱讀"},{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/738b8293dce86f7c8748e2629","title":null},"content":[{"type":"text","text":"https://xie.infoq.cn/article/738b8293dce86f7c8748e2629"}]},{"type":"text","text":" 瞭解視頻傳輸的關鍵路徑。webRTC是一個異步系統，通信的雙方無需做時間同步。本文主要探討webRTC是怎樣解決下面兩個跟時間有關的問題：1. 音視頻同步 2. 基於延時的帶寬評估。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"音視頻同步（Lip-sync）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發送端採集-編碼-發送，接收端解碼-渲染，音頻流和視頻流的處理和網絡傳輸是互相獨立的，而且各自的採樣/播放頻率也是不同的。如何在接收端還原採集端的真實場景，從來都不是一件容易的事情。好在人的聽覺/視覺系統本來就有一定容忍能力，ITU（國際電信聯盟）給了一個建議：音頻之於視頻在這個範圍內[-125ms,45ms]，也就是落後125ms或早於45ms，人類感覺上是可以接受的，我們認爲這是音視頻處於同步狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"webRTC解決這個問題的原理也比較簡單，發送端給音頻流和視頻流的數據包都打上時間戳，這些時間戳都可以跟同一個時間基準對齊，接收端利用時間戳和緩存就可以調整每個流上音頻/視頻幀渲染時間，最終達到同步的效果。我們可以進一步瞭解一下實現的細節，以視頻流爲例。下圖爲視頻處理的流水線，每個矩形框是一個線程實例。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/23/23ad66b6490698c663004dc013ffa89b.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實現上有三種時間信息："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.本地系統時間：從操作系統啓動計時至當前的時間差值"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.NTP（"},{"type":"text","marks":[{"type":"italic"}],"text":"Network Time Protocol"},{"type":"text","text":"）時間：全局時間信息，從1/1/1900-00:00h計時到當前的時間差值"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.RTP時間：幀時間戳，以視頻採樣90k頻率爲例，rtp_timestamp=ntp_timestamp*90"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這三種時間座標都是對時間的度量，只是描述時間的方式不同。比如當前絕對時間2020-08-05T06:08:52+00:00, 它們是這樣表達的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本地時間1919620051：表示開機計數起，過去1919620051ms了，大概22.2days。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NTP時間3805596543795：表示距離1/1/1900-00:00h，過去3805596543795ms了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RTP時間：RTP時間由NTP時間計算而來，時間單位1/90000s，u32存儲，計算過程會發送溢出，((u32)3805596543795)*90=1521922030。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"發送端"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一幀視頻畫面在caputer線程就記錄下了，這一幀對應的三個時間信息，尤其重要的是RTP時間。這個rtp_timestamp在Packet pacer模塊會加一個提前設定的偏移量，作爲最終的rtp時間發出去。這個偏移量加在了整個rtp時間座標系內，所有的對外的RTP時間都加了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"視頻流按照自己的RTP時間對每一個包做了標記，音頻流也類似的根據自己的RTP時間對每一個音頻包做了標記，但這兩條流裏的時間都是按照自己的步調在走，是獨立的。如果要求接收端使這兩條流同步渲染，就要想辦法讓這些時間統一跟同一個時間基準對齊。如下圖示，邏輯上舉例描述了兩條流如何同步，其中的時間數字只做參考，非真實數據。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bf/bf865fcae54e4856ee86b478fdf72334.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RTCP SR(sender report)的作用之一就是做時間對齊的，將該流中的RTP時間於NTP時間對齊。所有的流都對齊發送端的NTP時間，這樣接收端就有了統一時間基準。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RTCP SR format 如下："}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" 0 1 2 3\n 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\nheader |V=2|P| RC | PT=SR=200 | length |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | SSRC of sender |\n +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+\nsender | NTP timestamp, most significant word |\ninfo +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | NTP timestamp, least significant word |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | RTP timestamp |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | sender's packet count |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | sender's octet count |\n +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+\nreport | SSRC_1 (SSRC of first source) |\nblock +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n 1 | fraction lost | cumulative number of packets lost |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | extended highest sequence number received |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | interarrival jitter |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | last SR (LSR) |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | delay since last SR (DLSR) |\n +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+\nreport | SSRC_2 (SSRC of second source) |\nblock +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n 2 : ... :\n +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+\n | profile-specific extensions |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n\n The sender report packet consists of three sections, possibly\n followed by a fourth profile-specific extension section if defined.\n The first section, the header, is 8 octets long. The fields have the\n following meaning:\n\n version (V): 2 bits\n Identifies the version of RTP, which is the same in RTCP packets\n as in RTP data packets. The version defined by this specification\n is two (2)."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"接收端"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖中可以看到經過網絡傳輸後，到達接收端的幀數據可能經過了jitter（抖動），亂序，比如stream1 的幀2/3/4。接收端通過RTCP SR和buffer的設計，採用pull的模式，以渲染作爲終點倒推從frame queue中取幀的延遲。從單條流處理過程中可以看到該延遲包含渲染+解碼+抖動延遲，而多流之間的同步還需要考慮流之間的相對傳輸延遲（參考RtpStreamsSynchronizer），最終得到每條流的取幀延遲。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"基於延時的帶寬估計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"WebRTC的成功之一在於其設計一套擁塞控制算法，基礎數據來自於發送端的丟包統計和包接收時間的統計。這裏只講一下有關timing的RTP包接收時間的統計和反饋，不對擁塞算法展開講述。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e3/e3843eeef1f866b03b0ecb082c076e27.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"擁塞控制的邏輯現在默認都在發送端執行，有關時間延遲的計算包括髮送時間T和接收時間t，發送端自己可以保存每個包T，接收端只需要反饋t即可。算法的邏輯是每個包的接收時間都要反饋，這涉及到交互數據和頻次就會比較多，webRTC對此也有精心設計。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"webRTC默認在發送端做傳輸帶寬的估計，媒體流走的RTP/UDP協議棧，UDP層沒有帶寬估計的功能，webRTC通過擴展RTP/RTCP的傳輸格式使得可以在發送端做傳輸層的帶寬估計。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RTP format"}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" 0 1 2 3\n 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n |V=2|P|X| CC |M| PT | sequence number |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | timestamp |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | synchronization source (SSRC) identifier |\n +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+\n | contributing source (CSRC) identifiers(if mixed) |\n | .... |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | header extension (optional) |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | payload header (format depended) |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | payload data |\n +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在header extension域組織如下類型的擴展內容"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://tools.ietf.org/html/draft-holmer-rmcat-transport-wide-cc-extensions-01","title":null},"content":[{"type":"text","text":"https://tools.ietf.org/html/draft-holmer-rmcat-transport-wide-cc-extensions-01"}]}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" 0 1 2 3\n 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | 0xBE | 0xDE | length=1 |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | ID | L=1 |transport-wide sequence number | zero padding |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tips: 可以注意到RTP包裏有兩種sequence number。"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"equence number：是RTP層的概念，用於RTP stream的重組解複用。比如多條流複用的場景，每條流有各自的自增序列。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"transport-wide sequence number：是傳輸層概念，傳輸層包的標識，用於傳輸層的碼率統計。該序列自增不受多流複用的影響，因爲複用發生在RTP層。"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發送端在發送的時候對每一個RTP packet都打上transport-wide sequence number的序號（PacketRouter::SendPacket），比如發送seq=53，54，55。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接收端收到該包後，把該包的到達時間記錄下來，記錄時間爲本地內部時間戳（單位ms），即開機多久了。"}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"packet_arrival_times_[53]=1819746010\npacket_arrival_times_[54]=1819746020\npacket_arrival_times_[55]=1819746026 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接收端RemoteEstimatorProxy模塊負責傳輸層統計的反饋，週期性的把包接收的時間信息回饋到發送端。transport feedback的格式有詳細的規則，定義如下 "},{"type":"link","attrs":{"href":"https://tools.ietf.org/id/draft-dt-rmcat-feedback-message-04.html#rfc.section.3.1","title":null},"content":[{"type":"text","text":"https://tools.ietf.org/id/draft-dt-rmcat-feedback-message-04.html#rfc.section.3.1"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏有一篇寫的不錯的註解可以參考 "},{"type":"link","attrs":{"href":"https://blog.jianchihu.net/webrtc-research-transport-cc-rtp-rtcp.html","title":null},"content":[{"type":"text","text":"https://blog.jianchihu.net/webrtc-research-transport-cc-rtp-rtcp.html"}]}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" 0 1 2 3\n 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n |V=2|P| FMT=CCFB | PT = 205 | length |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | SSRC of packet sender |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | SSRC of 1st media source |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | begin_seq | end_seq |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n |L|ECN| Arrival time offset | ... .\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n . .\n . .\n . .\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | SSRC of nth media source |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | begin_seq | end_seq |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n |L|ECN| Arrival time offset | ... |\n . .\n . .\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\n | Report Timestamp (32bits) |\n +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RTCP transport feedback一般是RTCP通道上最頻繁的傳遞內容，webRTC對其傳輸也有特別的設計。關注以下幾個參數"}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"max_intervel = 250ms //feedback 最大週期\nmin_intervel =50ms //feedback 最小週期\nrtcp_ratio = 5% //feedback佔用帶寬比例\nAvg_feedback_size = 68bytes //平均一個feedback包的大小"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發送RTCP transport feedback的時間週期控制在[50ms,250ms]內，在這個範圍內根據當前帶寬動態調整，儘量把RTCP transport feedback的傳輸佔用帶寬比例控制在5%。可以計算得到邊界，單單傳輸feedback佔用的帶寬範圍[2176bps,10880bps]，也是一筆不小的開銷了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文總結了webRTC中三種timing類型，本地時間、NTP時間、RTP時間，同時分析了音視頻同步和基於延遲帶寬評估兩個專題對時間信息的使用。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"附錄"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"附上webRTC工程上有關timing的幾個關鍵數據結構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Capturer"}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"class webrtc::VideoFrame{\n...\n uint16_t id_; //picture id\n uint32_t timestamp_rtp_; //rtp timestamp, (u32)ntp_time_ms_ *90\n int64_t ntp_time_ms_; //ntp timestamp, capture time since 1/1/1900-00:00h\n int64_t timestamp_us_; //internal timestamp, capture time since system started, round at 49.71days\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"VideoStreamEncoder::OnFrame // caluclate capture timing"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"VideoStreamEncoder::OnEncodedImage // fill capture timing "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RtpVideoSender::OnEncodedImage // timestamp_rtp_+random value"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"class webrtc::EncodedImage{\n...\n //RTP Video Timing extension\n //https://webrtc.googlesource.com/src/+/refs/heads/master/docs/native-code/rtp-hdrext/video-timing\n struct Timing {\n uint8_t flags = VideoSendTiming::kInvalid;\n int64_t encode_start_ms = 0; //frame encoding start time, base on ntp_time_ms_\n int64_t encode_finish_ms = 0; //frame encoding end time, base on ntp_time_ms_\n int64_t packetization_finish_ms = 0; //encoded frame packetization time, base on ntp_time_ms_\n int64_t pacer_exit_ms = 0; //packet sent time when leaving pacer, base on ntp_time_ms_\n int64_t network_timestamp_ms = 0; //reseved for network node\n int64_t network2_timestamp_ms = 0; //reseved for network node\n int64_t receive_start_ms = 0;\n int64_t receive_finish_ms = 0;\n } timing_;\n uint32_t timestamp_rtp_;\t\t\t\t\t\t\t\t\t\t\t//same as caputrer.timestamp_rtp_\n int64_t ntp_time_ms_;\t\t\t\t\t\t\t\t\t\t //same as caputrer.ntp_time_ms_\n int64_t capture_time_ms_;\t\t\t\t\t\t\t\t\t\t //same as caputrer.capture_time_ms_\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RTPSenderVideo::SendVideo"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"class webrtc::RtpPacketToSend{\n...\n // RTP Header.\n bool marker_; //frame end marker\n uint16_t sequence_number_; //RTP sequence number, start at random(1,32767)\n uint32_t timestamp_; //capturer timestamp_rtp_ + u32.random()\n uint32_t ssrc_; //Synchronization Source, specify media source\n \n int64_t capture_time_ms_; //same as capturer.capture_time_ms_\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"==="}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"receiver side"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RtpTransport::DemuxPacket"}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"class webrtc::RtpPacketReceived{\n...\n NtpTime capture_time_;\n int64_t arrival_time_ms_; //RTP packet arrival time, local internal timestamp\n // RTP Header.\n bool marker_; //frame end marker\n uint16_t sequence_number_; //RTP sequence number, start at random(1,32767)\n uint32_t timestamp_; //sender's rtp timestamp maintained by RTPSenderVideo \n uint32_t ssrc_; //Synchronization Source, specify media source\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RtpVideoStreamReceiver::ReceivePacket /OnReceivedPayloadData"}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"struct webrtc::RTPHeader{\n ...\n bool markerBit;\n uint16_t sequenceNumber; //RTP sequence, set by sender per RTP packet\n uint32_t timestamp; //sender's RTP timestamp \n uint32_t ssrc;\n RTPHeaderExtension extension; //contains PlayoutDelay&VideoSendTiming if has\n}\nclass webrtc::RtpDepacketizer::ParsedPayload{\n RTPVideoHeader video;\n\n const uint8_t* payload;\n size_t payload_length;\n}\n\nclass webrtc::RTPVideoHeader{\n ...\n bool is_first_packet_in_frame;\n bool is_last_packet_in_frame;\n PlayoutDelay playout_delay; //playout delay extension\n VideoSendTiming video_timing; //Video Timing extension, align with sender's webrtc::EncodedImage::timing\n}\n\nclass webrtc::VCMPacket{\n...\n uint32_t timestamp; //sender's RTP timestamp\n int64_t ntp_time_ms_;\n uint16_t seqNum;\n RTPVideoHeader video_header;\n RtpPacketInfo packet_info;\n}\nclass webrtc::RtpPacketInfo{\n ...\n uint32_t ssrc_;\n uint32_t rtp_timestamp_; //sender's rtp timestamp\n //https://webrtc.googlesource.com/src/+/refs/heads/master/docs/native-code/rtp-hdrext/abs-capture-time\n absl::optional absolute_capture_time_; //\n int64_t receive_time_ms_; //packet receive time, local internal timestamp \n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PacketBuffer::InsertPacket"}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"class webrtc::video_coding::RtpFrameObject: public EncodedImage{\n...\n RTPVideoHeader rtp_video_header_;\n uint16_t first_seq_num_;\n uint16_t last_seq_num_;\n int64_t last_packet_received_time_;\n int64_t _renderTimeMs;\n //inherit from webrtc::EncodedImage\n uint32_t timestamp_rtp_;\t\t\t\t\t\t\t\t\t\t\t\n int64_t ntp_time_ms_;\t\t\t\t\t\t\t\t\t\t \n int64_t capture_time_ms_;\t\t\t\t\t\t\t\t\t\t \n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}