Implement audio video synchronization

[nanangizz]

How it works in general:

1. App creates AV sync using pjmedia_av_sync_create().
2. App adds all media to be synchronized using pjmedia_av_sync_add_media().
3. Each time a media receives an RTCP-SR, update AV sync usingpjmedia_av_sync_update_ref().
4. Each time a media returns a frame to be rendered, e.g: via port.get_frame(), update AV sync using pjmedia_av_sync_update_pts(), the function will return a number:
   • zero : nothing to do,
   • positive: increase delay as many as the returned number,
   • negative: decrease delay as many as the returned number.

Logic in pjmedia_av_sync_update_pts():

• Calculate the absolute/NTP timestamp of the frame playback based on the reference NTP and the supplied frame RTP imestamp. This timestamp is usually called presentation time (pts).
• If pts is the largest or most recent, set it as AV sync's max_ntp. Otherwise, calculate the difference/delay of this pts to the max_ntp, then request the media to speed up as much as the delay.
• If after some requests the delay is still relatively high, request the fastest media to slow down instead.

Changes needed in stream:

1. Update reference time (NTP & RTP timestamps) whenever receive RTCP-SR.
2. Update presentation time whenever returns a frame (to play/render).
3. Add mechanism to calculate current delay & optimal delay from jitter buffer size.
4. When AV sync request to speed up, as long as the end delay is not lower than the optimal delay, do it.
5. When AV sync request to slow down, do it, i.e: increase and maintain the jitter buffer size.

[sauwming]

An early comment since it's still a draft.
The term 'av' seems to imply the sync will only apply to audio and video only, even though it looks like the description can cover general cases.
Especially with the upcoming text media, which may also need to be sync-ed (although the sync may not be available in the early version, but perhaps in the future).

Perhaps removing the av and call it pjmedia_sync should be sufficient?

[nanangizz]

An early comment since it's still a draft.

Sure, actually early feedbacks is one of the goal of creating this draft.

The term 'av' seems to imply the sync will only apply to audio and video only, even though it looks like the description can cover general cases. Especially with the upcoming text media, which may also need to be sync-ed (although the sync may not be available in the early version, but perhaps in the future).

Perhaps removing the av and call it pjmedia_sync should be sufficient?

Yes, the sync module is generic for any media, especially delivered via RTP/RTCP and using a shared/synchronized NTP source.

IMO the 'av_sync' prefix will enhance intuitiveness, i.e: it sounds related to presentation time synchronization. While 'sync' prefix may be somewhat ambiguous, e.g: what to sync?. I recall how 'ssl_sock' term was chosen over tls_sock for catchy/intuitiveness reason, when it was being developed, TLS was already around replacing SSL (even PJSIP transport was already using 'TLS', see also #957). What do you think?

[sauwming]

The term should be inter-media synchronization, but yes, it's not quite as catchy, so I suppose it's okay if the 'av' stays.