1056187 - Frame drops observed during RTSP streaming

Status: RESOLVED FIXED

(Firefox OS Graveyard :: RTSP, defect)

Description

[bhargavg1]

+++ This bug was initially created as a clone of Bug #1050351 +++

Compared to other platforms (like LA) see lot of frame drops while doing RTSP streaming on FFOS. 

Do we have any competitive comparison on the overall quality for RTSP streaming compared to other platforms

Comment 1

[bhargavg1]

[Blocking Requested - why for this release]:

Comment 2

[Ethan Tseng [:ethan]]

(In reply to bhargavg1 from comment #0)
> Compared to other platforms (like LA) see lot of frame drops while doing
> RTSP streaming on FFOS. 
Excuse me. What does "LA" stand for?

Comment 3

[Inder]

LA - Linux Android. Sorry for not being clear, this is an internal term we use for Android.

Comment 4

[Ethan Tseng [:ethan]]

(In reply to Inder from comment #3)
> LA - Linux Android. Sorry for not being clear, this is an internal term we
> use for Android.
Thanks, Inder.

Comment 5

[Ethan Tseng [:ethan]]

(In reply to bhargavg1 from comment #0)
> Do we have any competitive comparison on the overall quality for RTSP
> streaming compared to other platforms

Yes. Mostly we compare our RTSP streaming to Android and sometimes to VLC player.
We had setup Darwin Streaming Server ourselves and compare the clients' performance in our LAN.
We had also compare their performance by some public RTSP links.

Could you describe more specifically about the frame drop condition you encounter?
1) Do you see many broken blocks (in green or grey color) in the video? Or the playback is not smooth?
2) What kind of media codecs do you observe frame drops?
In our experience, the performance could be quite variant for different codecs. While many other factors could also impact the client's performance.
It would be helpful if we have more clues. :)

Comment 6

[Ethan Tseng [:ethan]]

Excuse me. I have several more questions here.
1. Did you compare FFOS and LA using the same device or chipset?
2. Do you use any tool to observe/analyze frame drops?
   If so, what is the frame drop rate you observed?

Comment 7

[howie [:howie]]

Hi Inder & Bhargavg, can you provide feedback to Comment 5 and Comment 6? Thanks.

Comment 8

[bhargavg1]

(In reply to Ethan Tseng [:ethan] from comment #5)
> (In reply to bhargavg1 from comment #0) 

Info from our test teams,

> Could you describe more specifically about the frame drop condition you
> encounter?
> 1) Do you see many broken blocks (in green or grey color) in the video? Or
> the playback is not smooth?
Even though the playback looks smooth, we see framedrops in the logs. We did see cases where the playback pauses sometimes and expect it to be because of buffering and broken blocks

> 2) What kind of media codecs do you observe frame drops?
We are seeing frame drops for all the codecs( H264, H263 & MPEG4)

> 3)Did you compare FFOS and LA using the same device or chipset?
This info is when compared against the same chipset to conclude if the issue is there on the base platform or not

> 4) Do you use any tool to observe/analyze frame drops?
Purely rely on the adb logs to see the frame drops. Ideally the frame drops shouldn’t be greater than 1% ( Total dropped frames/Total decoded frames) during End to End playback and 3%during seek. We are seeing frame drops beyond this level.

We rely on shared/js/media/video_stats.js on video element to find the total dropped frames and decoded frames.

Comment 9

[howie [:howie]]

Hi Ethan, can you try shared/js/media/video_stats.js, this path is under Gaia repository.

Comment 10

[Ethan Tseng [:ethan]]

(In reply to howie [:howie] from comment #9)
> Hi Ethan, can you try shared/js/media/video_stats.js, this path is under
> Gaia repository.

Thanks for information! I'll try it.

Comment 11

[Ethan Tseng [:ethan]]

(In reply to bhargavg1 from comment #8)
> > 2) What kind of media codecs do you observe frame drops?
> We are seeing frame drops for all the codecs( H264, H263 & MPEG4)
Hm... It seems all codecs suffer.
I will try to observe the frame drop rate in the way you did.

> > 4) Do you use any tool to observe/analyze frame drops?
> Purely rely on the adb logs to see the frame drops. Ideally the frame drops
> shouldn’t be greater than 1% ( Total dropped frames/Total decoded frames)
> during End to End playback and 3%during seek. We are seeing frame drops
> beyond this level. 
> We rely on shared/js/media/video_stats.js on video element to find the total
> dropped frames and decoded frames.
Could you share how did you use the tool "video_stats.js"?
I see videoStats.start() requires an HTMLVideoElement as parameter.
So did you write an app on your own and create HTMLVideoElement in your JS?
Or you still play RTSP by the browser app? If so, how did you utilize video_stat.js?

Comment 12

[bhargavg1]

(In reply to Ethan Tseng [:ethan] from comment #11)
> (In reply to bhargavg1 from comment #8)
> > > 4) Do you use any tool to observe/analyze frame drops?
> > Purely rely on the adb logs to see the frame drops. Ideally the frame drops
> > shouldn’t be greater than 1% ( Total dropped frames/Total decoded frames)
> > during End to End playback and 3%during seek. We are seeing frame drops
> > beyond this level. 
> > We rely on shared/js/media/video_stats.js on video element to find the total
> > dropped frames and decoded frames.
> Could you share how did you use the tool "video_stats.js"?
> I see videoStats.start() requires an HTMLVideoElement as parameter.
> So did you write an app on your own and create HTMLVideoElement in your JS?
> Or you still play RTSP by the browser app? If so, how did you utilize
> video_stat.js?

we have a small internal app forked off the mainline video app to take an
option to play either a local file or an URL (http/rtsp streams). This app uses the video_stats.js funcs to pull in the necessary info

Comment 13

[Ethan Tseng [:ethan]]

(In reply to bhargavg1 from comment #12)
> we have a small internal app forked off the mainline video app to take an
> option to play either a local file or an URL (http/rtsp streams). This app
> uses the video_stats.js funcs to pull in the necessary info

I got it. Thanks for your information.
I'll try to observer dropped frame rate on our side.

Comment 14

[bhavana bajaj [:bajaj]]

(In reply to Ethan Tseng [:ethan] from comment #13)
> (In reply to bhargavg1 from comment #12)
> > we have a small internal app forked off the mainline video app to take an
> > option to play either a local file or an URL (http/rtsp streams). This app
> > uses the video_stats.js funcs to pull in the necessary info
> 
> I got it. Thanks for your information.
> I'll try to observer dropped frame rate on our side.

Blocking you could observe the frame drop, if this should not be blocking, please re-nom.

Comment 15

[Ethan Tseng [:ethan]]

Attachment: RTP Receiver Block Diagram.png

A block diagram of an RTP receiver.

Comment 16

[Ethan Tseng [:ethan]]

I plan to do two tasks to analyze the root cause of frame drop.

1) Observe the frame drop rate using video_stats.js.
   Purpose: To reproduce the reported issue, that is, total frame drop rate > 1%.
   How    : Launch video app for RTSP links (we did this before in v1.3 and v1.4).
   Comment: video_stats.js uses the Web API getVideoPlaybackQuality() to retrieve frame statistics.

2) Observe the counter of input and output access-units of RTP assembler.
   Purpose: To exclude RTP assembler being the factor of frame drop.
   How    : Add logs in the input and output of XXXAssembler.
   Comment: If we see frame drops from video_stats.js but no drop in RTP assembler,
            we can be sure the performance bottleneck is not in RTP assembler.
            It could be HW decoder or other components.

Refer to the diagram in attachment 8481227 [details].

Comment 17

[Ethan Tseng [:ethan]]

Attachment: How does VideoPlaybackQuality calcuate counters

A block diagram illustrating how the VideoPlaybackQuality calculates the counters of video statistics.

Comment 18

[Ethan Tseng [:ethan]]

See attachment 8482199 [details].

The VideoPlaybackQuality interface provides three counters about video frames.
They are calculated in HTMLVideoElement::GetVideoPlaybackQuality():
* totalVideoFrames     = parsedFrames
* droppedVideoFrames   = totalVideoFrames - presentedFrames
* corruptedVideoFrames = totalVideoFrames - decodedFrames

"corruptedVideoFrames" is caused by HW decoder.
Another counter "droppedByAVSync", which is not shown by VideoPlaybackQuality, is caused by
AV Sync Buffer (in MediaDecoderStateMachine)

They both contribute to "droppedVideoFrames".

Comment 19

[Ethan Tseng [:ethan]]

Benjamin, please provide your input to comment 18, thanks!

Comment 20

[Ethan Tseng [:ethan]]

(In reply to Ethan Tseng [:ethan] from comment #16)
> 1) Observe the frame drop rate using video_stats.js.
I used a patch to launch the video app for RTSP scheme.
We do see a significant frame drop rate reporting from video_stats.js.
The formula is listed in comment 18.

> 2) Observe the counter of input and output access-units of RTP assembler.
>    Purpose: To exclude RTP assembler being the factor of frame drop.
This step is not required since we already observe frame drops in step 1.

Comment 21

[Benjamin Chen [:bechen]]

(In reply to Ethan Tseng [:ethan] from comment #18)
> See attachment 8482199 [details].
> 
> The VideoPlaybackQuality interface provides three counters about video
> frames.
> They are calculated in HTMLVideoElement::GetVideoPlaybackQuality():
> * totalVideoFrames     = parsedFrames
> * droppedVideoFrames   = totalVideoFrames - presentedFrames
> * corruptedVideoFrames = totalVideoFrames - decodedFrames
> 
> "corruptedVideoFrames" is caused by HW decoder.
> Another counter "droppedByAVSync", which is not shown by
> VideoPlaybackQuality, is caused by
> AV Sync Buffer (in MediaDecoderStateMachine)
> 
> They both contribute to "droppedVideoFrames".

There are two kinds of frame drops:
1. corruptedVideoFrames:
When our statemachine feels the decoding speed is too slow, it will trigger "skip to next I-frame", but unfortunately, our RTSP doesn't implement this kind of mechanism. That make the number of corruptedVideoFrames huge.
http://dxr.mozilla.org/mozilla-central/source/content/media/MediaDecoderStateMachine.cpp#612

For example: statemachine wants the frame after 5s, but we still input the 4s, 4.5s 4.8s data into HW decoder, these 3 frames are treated as corruptedVideoFrames.
If we really want to solve this, we can do at Bug 897486.

2. droppedByAVSync:
This seems an systematic problem, we need more analysis on it.
Ex: If there are 1s 2s 3s three video frames in video queue, but now the audio clock had played at 4s, the 1s 2s frames are treated as droppedByAVSync frames. Only the 3s frame will be displayed.
Means we need a faster HW video decoder or we need go into "buffer state".

In Overall, I guess our HW might waste some decoding power for the corruptedVideoFrames, make the droppedByAVSync worst.

Comment 22

[Ethan Tseng [:ethan]]

(In reply to Benjamin Chen [:bechen] from comment #21)
> In Overall, I guess our HW might waste some decoding power for the
> corruptedVideoFrames, make the droppedByAVSync worst.
Benjamin, thanks for your input!
As discussed offline today, we can trace the problem starting here:
http://dxr.mozilla.org/mozilla-central/source/content/media/MediaDecoderStateMachine.cpp#612

Compared to HTTP streaming, RTSP streaming enters this block quite often.
I'll look into those variables to find out why.

Comment 23

[Benjamin Chen [:bechen]]

Also, we need to check the network status here. If the network speed is lower than HW decoder, the thread will be block here:
http://dxr.mozilla.org/mozilla-central/source/content/media/RtspMediaResource.cpp?from=RtspMediaResource.cpp#222

For HTTP streaming:
http://dxr.mozilla.org/mozilla-central/source/content/media/MediaCache.cpp?from=MediaCache.cpp#2231

Comment 24

[Benjamin Chen [:bechen]]

Attachment: WIP-rtsp_byteconsume.patch

wip patch to testing.

Comment 25

[Ethan Tseng [:ethan]]

*** Progress Update ***
We made some measurement and experiments but are still working to identify the root cause.

Observations:
1. Frame drop rate between HTTP and RTSP streaming.
   Server   : Apache server and DarwinStreamingServer on the same PC (Ubuntu Linux)
   Video    : H264, AAC, 720p, 30fps (unhinted for HTTP and hinted for RTSP)
   Duration : 45 sec
   Player   : Gaia video app (Flame v2.1)
   Statistic: VideoPlaybackQuality from video_stats.js
                   TotalFrames    DroppedFrames    DroppedFrameRate
   HTTP streaming  around 1350    1~2              < 0.15%
   RTSP streaming  around 1350    300~400          22~30%

2. Since HTTP and RTSP streaming use the same HW decoder and the same MediaDecoderStateMachine,
   we believe the video frame drops of RTSP are possibly caused by the design of playout buffer,
   which is mainly implemented in RtspMediaResource and RtspTrackBuffer.
   (Playout buffer is illustrated in attachment 8482199 [details])

3. One important purpose of the playout buffer is to compensate for network timing variability.
   We'll further analyze our playout buffer satisfy this feature or not.

4. We measured RTSP quality by playing low resolution video, such as 144p.
   The frame drop rate drops to 3~5%.
   By reviewing the current implementation, memory copies of video frames could be a burden to
   the playback performance.

Comment 26

[bhargavg1]

(In reply to Ethan Tseng [:ethan] from comment #20)
> I used a patch to launch the video app for RTSP scheme.

Ethan, can you point to the patch as to how we can modify the video app to take an RTSP stream URL

Comment 27

[StevenLee[:slee]]

Attachment: log.patch

I added some code to log the time of incoming frames. I found that
1. sometimes the parent process sends lots of frames to content process but content process gets the frames after almost 1 second
2. when I boot b2g device and do nothing, b2g process uses about 20% CPU. I will spend time on profiling it.

Comment 28

[StevenLee[:slee]]

The high CPU problem is filed on bug 1062119.

Comment 30

[Ethan Tseng [:ethan]]

Attachment: Launch Video App for RTSP (Gecko part)

This patch basically reverts the code changes of bug 1000340.

Comment 31

[Ethan Tseng [:ethan]]

Attachment: Launch Video App for RTSP (Gaia part)

How to launch video app for RTSP scheme.

Comment 32

[Ethan Tseng [:ethan]]

(In reply to bhargavg1 from comment #26)
> Ethan, can you point to the patch as to how we can modify the video app to
> take an RTSP stream URL

You can launch the video app for RTSP scheme by these steps:
1. Apply the patch of attachment 8484851 [details] [diff] [review].
2. Follow the instructions in attachment 8484853 [details] to modify your video app.
3. Flash gecko and install gaia.
4. Open the browser app. Clicking any RTSP URL would launch the video app now.

If you have any problem, please let me know. Thanks.

Comment 33

[Ethan Tseng [:ethan]]

Attachment: WIP to fix frame drop of RTSP

Add a fixed playout delay (2 sec) to reduce frame drop rate.
We see the frame drop rate become less than 2% by applying this patch.

Hi Bhargavg,
Please note this is just a experimental patch, not a final solution.
If you have time, please help to test with this patch and let us know
your result. Thanks!

Comment 34

[Ethan Tseng [:ethan]]

A meta bug 1063466 was filed to track all the issues we discovered while investigating this bug.
We plan to focus on fixing the "playout buffer delay", which is the top bottleneck for now, in this bug.

Comment 35

[bhargavg1]

(In reply to Ethan Tseng [:ethan] from comment #33)
> Created attachment 8484885 [details] [diff] [review]
> WIP to fix frame drop of RTSP
> 
> Hi Bhargavg,
> Please note this is just a experimental patch, not a final solution.
> If you have time, please help to test with this patch and let us know
> your result. Thanks!

Sure will try this out hopefully should have something by Monday

Comment 36

[Ken Chang[:kenkai]]

According to comment 34, I am thinking that having *playout buffer delay* is a big feature. I wonder if we can remove it from 2.0 CAF and add it in 2.1.

Comment 37

[bhargavg1]

(In reply to Ethan Tseng [:ethan] from comment #33)
> Created attachment 8484885 [details] [diff] [review]
> WIP to fix frame drop of RTSP
> 

Results from testing the patch. 

Listed both without and with patch results 

Clip                  frame-drops-WIP     frame-drops-without-patch
clip1_1080p.mp4           195/1204            196/1209 
clip2_720p.mp4            151/1040            102/1052

Comment 38

[Inder]

[Blocking Requested - why for this release]:

Ken: based on offline discussion, moving this out of 2.0 to 2.1. We may want to backport to 2.0 in future with proper risk assessment in case customer requests.

Comment 39

[Vincent Chang[:vchang][changyihsin]]

Attachment: Patch v1.0

Comment 40

[Benjamin Chen [:bechen]]

Comment on attachment 8487041 [details] [diff] [review]
Patch v1.0

Review of attachment 8487041 [details] [diff] [review]:
-----------------------------------------------------------------

1. The playout-delay should also apply to seek.
2. We should have a check for the occupancy of RtspTrackBuffer, and terminate the playout-delay when buffer is almost full.
3. The EOS/network error should also terminate the playout-delay.
4. Do we need to adjust the delay time dynamically?

::: content/media/RtspMediaResource.cpp
@@ +61,5 @@
>    , mSlotSize(aSlotSize)
>    , mTotalBufferSize(BUFFER_SLOT_NUM * mSlotSize)
>    , mFrameType(0)
> +  , mIsStarted(false)
> +  , mPlayoutDely(false) {

typo

@@ +389,5 @@
>    mMonitor.NotifyAll();
>  }
>  
> +void RtspTrackBuffer::FireTimerCallback(nsITimer *aTimer, void *aClosure) {
> +  RtspTrackBuffer *pRes = reinterpret_cast<RtspTrackBuffer *>(aClosure);

static_cast

Comment 41

[Ethan Tseng [:ethan]]

(In reply to Vincent Chang[:vchang] from comment #39)
> Created attachment 8487041 [details] [diff] [review]
> Patch v1.0
Thanks for providing this patch!
I will test and see how much improvement we can make from it.

Comment 42

[Vincent Chang[:vchang][changyihsin]]

Hi Benjamin, thanks for your feedback on the patch. Per offline talk with Ethan, he will provide the follow-up patch to address you comments.

Comment 43

[Ethan Tseng [:ethan]]

Attachment: framedrop-wip.patch

WIP patch: Based on Vincent's patch. Add playout delay mechanism, in order to compensate timing variation caused by network.

Comment 44

[StevenLee[:slee]]

Attachment: 0001-add-log.patch

This patch is to dump some time stamp when a "whole" video frame is received on
1. b2g onTrackDataAvailable
2. b2g send meta data to content process
3. content process receives the frame on RecvOnMediaDataAvailable
4. content process handles the frame on RtspMediaResource::OnMediaDataAvailable
5. content process writes the frame to buffer(WriteBuffer)
6. content process reads the frame from buffer(ReadBuffer)
7. content process decodes the frame 
8. content process render the frame

Comment 45

[Benjamin Chen [:bechen]]

After apply the playout-delay and slee's patch, now we suspect the bottleneck is the |mDecodeTaskQueue| in the StateMachine.

The TaskQueue executes the "DecodeAudio" and "DecodeVideo" sequentially, once the "playout-delay buffer" is exhausted, the "DecodeAudio" might be blocked then the "DecodeVideo" is blocked too.

Comment 46

[Benjamin Chen [:bechen]]

Attachment: playoutdelay-wip.patch

Comment 47

[Ethan Tseng [:ethan]]

Attachment: bug-1056187-part1.patch

WIP patch: Add 3-sec playout delay to RtspTrackBuffer.

Benjamin, this patch does reduce the frame drop rate, but the playback
is not smooth in the first few seconds.
From logs I can see several pause-play command pairs in the beginning.
Let's discuss this tomorrow.

Comment 48

[Ethan Tseng [:ethan]]

We filed two bugs to unravel the causes and solutions of this bug.

Comment 49

[Ken Chang[:kenkai]]

Ethan, it seems that we can not have a fix for this bug before 10/7. Is it possible to fix it before 10/10?

Comment 50

[Ethan Tseng [:ethan]]

(In reply to Ken Chang[:ken] from comment #49)
> Ethan, it seems that we can not have a fix for this bug before 10/7. Is it
> possible to fix it before 10/10?

Bug 1075400 already got r+ and is in landing process now.
We also have an idea how to fix bug 1074791 and I'm working on the patch.
I think we can fix bug 1074791 in 2-3 days, that is, before 10/10.
I'll provide a patch for this bug soon.

In conclusion, it's possible to fix the whole frame drop issue by 10/10.
I'll keep updating the status.

Comment 51

[Ethan Tseng [:ethan]]

Attachment: Add playout delay to RtspTrackBuffer

Hi Benjamin,
Please help to review this patch. :)

Comment 52

[Benjamin Chen [:bechen]]

Comment on attachment 8500910 [details] [diff] [review]
Add playout delay to RtspTrackBuffer

Review of attachment 8500910 [details] [diff] [review]:
-----------------------------------------------------------------

Overall looks good to me. Please add comments as possible as you can.

::: content/media/RtspMediaResource.cpp
@@ +461,5 @@
> +
> +// static
> +void
> +RtspTrackBuffer::PlayoutDelayTimerCallback(nsITimer *aTimer,
> +                                           void *aClosure)

Since the aClosure is a raw pointer, we need to cancel this timer before ~RtspTrackBuffer() .

::: content/media/omx/RtspOmxReader.cpp
@@ +91,5 @@
> +
> +  // FIXME: Calling SetIdle() impacts performance, but not calling it might be
> +  // problematic when auto-play is disabled.
> +  // Stop the network traffic because the stream is not really played.
> +  // SetIdle();

If we don't call SetIdle() function here, the network data won't stop if we are go into "waiting resource state".

@@ +94,5 @@
> +  // Stop the network traffic because the stream is not really played.
> +  // SetIdle();
> +  if (rv == NS_OK && !IsWaitingMediaResources()) {
> +    mRtspResource->EnablePlayoutDelay();
> +  }

Call SetIdle() function if we are waiting resources.
//
else if (IsWaitingMediaResources  ) {
SetIdle();
}

Comment 53

[Ethan Tseng [:ethan]]

Comment on attachment 8500910 [details] [diff] [review]
Add playout delay to RtspTrackBuffer

Review of attachment 8500910 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/media/RtspMediaResource.cpp
@@ +461,5 @@
> +
> +// static
> +void
> +RtspTrackBuffer::PlayoutDelayTimerCallback(nsITimer *aTimer,
> +                                           void *aClosure)

The timer is canceled by StopPlayoutDelay().
And StopPlayoutDelay() is called in RtspTrackBuffer::Stop() and Reset().
Is there any other case we should take care of?

::: content/media/omx/RtspOmxReader.cpp
@@ +94,5 @@
> +  // Stop the network traffic because the stream is not really played.
> +  // SetIdle();
> +  if (rv == NS_OK && !IsWaitingMediaResources()) {
> +    mRtspResource->EnablePlayoutDelay();
> +  }

Okay, I'll add this.

Comment 54

[Ethan Tseng [:ethan]]

Attachment: Add playout delay to RtspTrackBuffer

Addresses issues in comment 52.

Comment 55

[Ethan Tseng [:ethan]]

Attachment: Add playout delay to RtspTrackBuffer

1. Refresh comment "r=bechen".
2. Fix a compile warning of comparison between signed and unsigned integer expressions

Comment 56

[Ethan Tseng [:ethan]]

Attachment: Add playout delay to RtspTrackBuffer

Truncate white spaces.

Comment 57

[Ethan Tseng [:ethan]]

After applying the fixes of bug 1075400, bug 1074791 and this bug, the frame drop rate could be less 
than 1%.

The video statistics dumped on my Flame 2.2 today:

I/GeckoDump(31210): Video statistics start
I/GeckoDump(31210): Dimensions: 1280x720
I/GeckoDump(31210): Complete duration: 00:45
I/GeckoDump(31210): Start time: 00:00
I/GeckoDump(31210): End time: 00:00
I/GeckoDump(31210): Total frames: 1340
I/GeckoDump(31210): Decoded frames: 1340
I/GeckoDump(31210): Corrupted frames: 0
I/GeckoDump(31210): Rendered frames: 1339
I/GeckoDump(31210): Dropped frames: 1
I/GeckoDump(31210): Average rendered FPS: 28.45

Comment 58

[Ethan Tseng [:ethan]]

Attachment: bug-1056187-fix.patch

Fixed compile error revealed by Treeherder.

Comment 59

[Ethan Tseng [:ethan]]

Attachment: Add playout delay to RtspTrackBuffer

Comment 60

[Ethan Tseng [:ethan]]

The result of Treeherder:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=8e4d441013bd

Request check-in of the patch of attachment 8501847 [details] [diff] [review].

Comment 61

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c5a41dac5729

Comment 62

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/c5a41dac5729

Comment 63

[Ryan VanderMeulen [:RyanVM]]

Please request b2g34 approval on this patch when you get a chance.

Comment 64

[Ethan Tseng [:ethan]]

Comment on attachment 8501847 [details] [diff] [review]
Add playout delay to RtspTrackBuffer

Approval Request Comment
[Feature/regressing bug #]: Reduce frame drop rate over RTSP
[User impact if declined]: RTSP streaming playback is not smooth
[Describe test coverage new/current, TBPL]: On m-c
[Risks and why]: None
[String/UUID change made/needed]: None

Comment 65

[Ethan Tseng [:ethan]]

Comment on attachment 8501847 [details] [diff] [review]
Add playout delay to RtspTrackBuffer

Correct the approval flag.

Comment 66

[Ethan Tseng [:ethan]]

Hi Ryan,
We intend to land this patch to v2.1.
Should I also request aurora approval on the patch?

Comment 67

[Ryan VanderMeulen [:RyanVM]]

v2.1 is moving to b2g34 on Monday :)

Comment 68

[Ethan Tseng [:ethan]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #67)
> v2.1 is moving to b2g34 on Monday :)
Got it. Thanks!

Comment 69

[Ethan Tseng [:ethan]]

(In reply to Ethan Tseng [:ethan] from comment #57)
> After applying the fixes of bug 1075400, bug 1074791 and this bug, the frame
> drop rate could be less than 1%.

Please notice that although the frame drop rate is low after these 3 fixes, we can still see a obvious
lagging near the beginning of playback (3~4 sec).
The root cause was found and fixed in bug 1080470.

Comment 70

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/9bc37dee3984

Comment 71

[Ethan Tseng [:ethan]]

Bhargavg,

Please note our solution made a tradeoff between quality and promptness.
After the fix, we have to wait for around 3 seconds in the initial playback.
The media player is buffering enough stream data for smoothing playback during this period.

We think this behavior is reasonable and acceptable for streaming.
Android phone has the same buffering behavior (it waits for 4~5 sec).

Comment 72

[Pooja]

HI Ethan,
 We have tried again with the fix and observed that frame drop of H264 test clips is reduced to 1% but we are still seeing huge frame drop for MPEG4 test clip.

Have sent an email with attached MPEG4 test clip where we are seeing huge frame drop and also while RTSP playback video stops and audio continues.

Video Stats:
10-28 18:15:21.516  1516  1516 I GeckoDump: Total frames: 140
10-28 18:15:21.516  1516  1516 I GeckoDump: Decoded frames: 66
10-28 18:15:21.516  1516  1516 I GeckoDump: Corrupted frames: 74
10-28 18:15:21.516  1516  1516 I GeckoDump: Rendered frames: 56
10-28 18:15:21.516  1516  1516 I GeckoDump: Dropped frames: 84
10-28 18:15:21.516  1516  1516 I GeckoDump: Average rendered FPS: 0.82

Comment 73

[Pooja]

HI Ethan,

As you are already working on this (as per your mail) I am re-opening this bug, would be easy to track.

Comment 74

[Ethan Tseng [:ethan]]

(In reply to Pooja from comment #73)
> HI Ethan,
> As you are already working on this (as per your mail) I am re-opening this
> bug, would be easy to track.

Hi Pooja,

Sorry for my late response.

I hinted the clip and played it over RTSP streaming by different players.
1) FXOS (Flame v2.2)    : Most video frames cannot be rendered, the screen is frozen most of the time.
                          Audio frames are played but quality is poor.
2) VLC (desktop v2.1.5) : Better than FXOS. But the overall quality is still poor.
3) Android (Nexus5 v4.4): Better than FXOS. But the overall quality is still poor.

It seems the MPEG4 format in this clip is not well supported by most streaming players.
Based on this result, I doubt whether this clip should be the quality criteria for us.

Comment 75

[Ethan Tseng [:ethan]]

In my opinion, this is kind of a format issue but not performance issue.
I suggest we file a new bug to trace it.

Comment 76

[Ethan Tseng [:ethan]]

I filed bug 1093548 - [RTSP] Poor quality for certain MPEG4-AAC clip.

Comment 77

[howie [:howie]]

Hi Pooja, do you have any feedback for comment 74 ?

Comment 78

[Pooja]

Agree with comment #74.
Because we are not seeing same huge frame drop for all 720p/1080p clips but only with some specific clips.

Comment 79

[Ethan Tseng [:ethan]]

(In reply to Pooja from comment #78)
> Agree with comment #74.
> Because we are not seeing same huge frame drop for all 720p/1080p clips but
> only with some specific clips.

Okay. Then I'll set this bug as resolved again.
Let's trace the specific clip issue in bug 1093548.

Comment 80

[Yeojin Chung [:YeojinC]]

Unable to verify this bug and dependencies as it is a back-end issue.

Comment 81

[Yeojin Chung [:YeojinC]]

(In reply to Yeojin Chung [:YeojinC] from comment #80)
> Unable to verify this bug and dependencies as it is a back-end issue.

Correction: We do not have the necessary tools to verify this issue.