Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The myth of HTMLVideoElement.readyState and HTMLVideoElement.requestVideoFrameCallback #6577

Closed
hujiajie opened this issue Jun 29, 2022 · 2 comments · Fixed by #6626
Closed

The myth of HTMLVideoElement.readyState and HTMLVideoElement.requestVideoFrameCallback #6577

hujiajie opened this issue Jun 29, 2022 · 2 comments · Fixed by #6626
Assignees
@hujiajie
Labels
comp:webgpu

Comments

@hujiajie
Copy link
Contributor

I'm seeing this error when running latest TFJS unit tests with the WebGPU backend:

Chrome 105.0.0.0 (Windows 10) webgpu backend webgpu webgpu {"WEBGPU_CPU_FORWARD":false} should behave well if WEBGPU_USER_IMPORT is true or false FAILED
 Error: The video element has not loaded data yet. Please wait for `loadeddata` event on the <video> element.
     at fromPixels_ (tfjs-core/src/ops/browser.ts:115:13 <- tfjs/tfjs-backend-webgpu/src/tfjs-backend-webgpu_test_bundle.js:5941:15)
     at Object.<anonymous> (tfjs-core/src/ops/browser.ts:279:10 <- tfjs/tfjs-backend-webgpu/src/tfjs-backend-webgpu_test_bundle.js:6021:14)
     at <Jasmine>
     at tfjs/tfjs-backend-webgpu/src/tfjs-backend-webgpu_test_bundle.js:52:64
     at <Jasmine>
     at __async (tfjs-backend-webgpu/src/tfjs-backend-webgpu_test_bundle.js:36:12)
     at Object.fromPixelsAsync (tfjs-backend-webgpu/src/tfjs-backend-webgpu_test_bundle.js:6004:12)
     at ../../tfjs-backend-webgpu/src/backend_webgpu_test.ts:248:36 <- tfjs/tfjs-backend-webgpu/src/tfjs-backend-webgpu_test_bundle.js:84990:47
     at <Jasmine>
     at fulfilled (tfjs-backend-webgpu/src/tfjs-backend-webgpu_test_bundle.js:39:27)

OS: Windows 10 (19044.1766)
GPU: Intel UHD 620 (driver version: 30.0.101.1994)
Browser: Chrome Canary (version: 105.0.5148.0)

What I don't understand is the readiness of the frame data has already been gated by HTMLVideoElement.requestVideoFrameCallback:

tfjs/tfjs-backend-webgpu/src/backend_webgpu_test.ts

Line 243 in ca9799d

await new Promise(go => (video as any).requestVideoFrameCallback(go));

How can the ready state be 1 (HAVE_METADATA) instead of 2 (HAVE_CURRENT_DATA) or higher here:

tfjs/tfjs-core/src/ops/browser.ts

Lines 112 to 118 in ca9799d

if (isVideo &&
(pixels as HTMLVideoElement).readyState <
HAVE_CURRENT_DATA_READY_STATE) {
throw new Error(
'The video element has not loaded data yet. Please wait for ' +
'`loadeddata` event on the <video> element.');
}

Am I misunderstanding anything? Thanks

cc @gyagp @qjia7 @shaoboyan
@qjia7
Copy link
Contributor

qjia7 commented Jun 29, 2022

@hujiajie Please file a chromium issue to see whether it's a chrome bug about the readyState and lint it here. Thanks.

@hujiajie
Copy link
Contributor Author

https://bugs.chromium.org/p/chromium/issues/detail?id=1340433

hujiajie added a commit to hujiajie/tfjs that referenced this issue Jul 15, 2022
@hujiajie
De-flake fromPixels test failure
1bec9ec 
`HTMLVideoElement.requestVideoFrameCallback()` provides a realiable way
to guarantee the readiness of a video frame, whereas the browser may not
update the ready state until a few frames later. In such cases, false
alarms will be raised by the validation in `fromPixels()`.

For where rVFC is unavailable, fallback to the old path based on the
`loadeddata` event. This at least seems better than waiting on nothing.

Fixes tensorflow#6577
@hujiajie hujiajie mentioned this issue Jul 15, 2022
Merged
hujiajie added a commit to hujiajie/tfjs that referenced this issue Aug 1, 2022
@hujiajie
De-flake fromPixels test failure
2723787 
`HTMLVideoElement.requestVideoFrameCallback()` provides a realiable way
to guarantee the readiness of a video frame, whereas the browser may not
update the ready state until a few frames later. In such cases, false
alarms will be raised by the validation in `fromPixels()`.

Fixes tensorflow#6577
gyagp pushed a commit that referenced this issue Aug 5, 2022
@hujiajie
De-flake fromPixels test failure (#6626)
a96fb56 
`HTMLVideoElement.requestVideoFrameCallback()` provides a realiable way
to guarantee the readiness of a video frame, whereas the browser may not
update the ready state until a few frames later. In such cases, false
alarms will be raised by the validation in `fromPixels()`.

Fixes #6577

Co-authored-by: Yang Gu <yang.gu@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hujiajie hujiajie

Labels
comp:webgpu
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

De-flake fromPixels test failure hujiajie/tfjs
3 participants
@hujiajie @qjia7 @rthadur