Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No video played when setVideoEffects called on NVIDIA Shield #1620

Open
1 task done
damontecres opened this issue Aug 14, 2024 · 12 comments
Open
1 task done

No video played when setVideoEffects called on NVIDIA Shield #1620

damontecres opened this issue Aug 14, 2024 · 12 comments
Assignees

Comments

@damontecres
Copy link

Version

Media3 1.4.0

More version details

Calling player.setVideoEffects() with any effects or even an empty list results in no video (just black screen).

I reviewed the requirements for this to work and they all seem met:

  • Not passing a surface manually, using a androidx. media3.ui.PlayerView
  • media-effect lib is added
  • Checking player.getRenderer and the only video one is MediaCodecVideoRenderer
  • Effects do not edit timestamps
  • No DRM
  • Function is called before prepare()

I guess it must be a hardware decoding issue, but I'm not sure how to validate. There's nothing obviously erroring in the logs.

Bare-bones code that causes the issue:

val player = ExoPlayer.Builder(this).build()
playerView.player = player
player.setVideoEffects(listOf()) // Commenting this out works
player.setMediaItem(MediaItem.fromUri("https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4"))
player.prepare()
player.play()

Devices that reproduce the issue

  • NVIDIA Shield (2017 Pro & 2019 Pro models)

Devices that do not reproduce the issue

  • Emulator running Android TV API 34
  • Fire TV Stick 4k Max 2nd Gen

Reproducible in the demo app?

Not tested

Reproduction steps

  1. Call setVideoEffects on the ExoPlayer object (even with an empty list)
  2. Play a video

Expected result

Video plays successfully with effects applied or if no effects added at least the video plays normally

Actual result

Video playback is a black screen.

Audio plays as expected.

Media

https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4

Bug Report

  • You will email the zip file produced by adb bugreport to [email protected] after filing this issue.
@damontecres
Copy link
Author

I didn't see an example in the demo apps that used setVideoEffects, so I made a test one https://github.com/damontecres/media3-issue if that helps. Thanks!

@claincly
Copy link
Contributor

I was wondering what output is connected to this? Is it a PlayerView?

Can you help me double check a few things, adding logs to show the variables related to resolutions at

videoSink.setOutputSurfaceInfo(checkStateNotNull(displaySurface), outputResolution);

and

inputListener.onInputFrameProcessed(inputTexture);

And log if this line is executed:

EGL14.eglSwapBuffers(eglDisplay, outputEglSurface);

@damontecres
Copy link
Author

damontecres commented Aug 14, 2024

Hi @claincly!

I was wondering what output is connected to this? Is it a PlayerView?

Yes, defined very simply:

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
  android:layout_width="match_parent"
  android:layout_height="match_parent">
  <androidx.media3.ui.PlayerView
    android:id="@+id/player_view"
    android:layout_width="match_parent"
    android:layout_height="match_parent"/>
</FrameLayout>

Can you help me double check a few things, adding logs to show the variables related to resolutions at

Here are the logs when executing with setVideoEffects called:

2024-08-14 11:53:55.657  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=0x0
2024-08-14 11:53:55.657  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:53:55.657  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = false
2024-08-14 11:53:55.771  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=1920x984
2024-08-14 11:53:55.771  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:53:55.771  2520-2563  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = true
2024-08-14 11:53:56.514  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-14 11:53:56.514  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-14 11:53:56.567  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface EGL14.eglSwapBuffers was called
2024-08-14 11:53:56.592  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-14 11:53:56.592  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-14 11:53:56.603  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface EGL14.eglSwapBuffers was called
2024-08-14 11:53:56.613  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-14 11:53:56.613  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-14 11:53:56.639  2520-2577  issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface EGL14.eglSwapBuffers was called

Here are the logs when executing without setVideoEffects called:

2024-08-14 11:54:51.660  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=0x0
2024-08-14 11:54:51.660  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:54:51.661  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = false
2024-08-14 11:54:51.772  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=1920x984
2024-08-14 11:54:51.772  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:54:51.772  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = true
2024-08-14 11:54:52.389  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=1749x984
2024-08-14 11:54:52.389  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-14 11:54:52.389  2703-2743  issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = true

@damontecres
Copy link
Author

@claincly, is there any other information I can provide to help debug this?

Any ideas on if there is a work around? I tried sending the MSG_SET_VIDEO_OUTPUT_RESOLUTION message manually a few different ways without success.

@claincly
Copy link
Contributor

I wonder if you could log on this line?

EGL14.eglSwapBuffers(eglDisplay, outputEglSurface);

The resolution logs look correct. You might noticed some discrepancies between with/without video effects, but that's expected.

@damontecres
Copy link
Author

Hi @claincly, here are some logs from in that method. Thoughts?

2024-08-27 19:56:19.353 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=0x0
2024-08-27 19:56:19.353 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-27 19:56:19.353 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = false
2024-08-27 19:56:19.452 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION Size=1920x1080
2024-08-27 19:56:19.452 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION videoSink!=null = false
2024-08-27 19:56:19.452 23576-23612 issue1620               com...hub.damontecres.myapplication  D  MSG_SET_VIDEO_OUTPUT_RESOLUTION displaySurface!=null = true
2024-08-27 19:56:20.181 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.181 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-27 19:56:20.202 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: inputTexture=androidx.media3.common.GlTextureInfo@51e72ea
2024-08-27 19:56:20.202 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: presentationTimeUs=1000000000000
2024-08-27 19:56:20.202 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: renderTimeNs=-1
2024-08-27 19:56:20.217 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: eglDisplay=android.opengl.EGLDisplay@210
2024-08-27 19:56:20.217 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputEglSurface=android.opengl.EGLSurface@4edfb0a5
2024-08-27 19:56:20.217 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.width=1920
2024-08-27 19:56:20.217 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.height=1080
2024-08-27 19:56:20.218 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface after EGL14.eglSwapBuffers was called
2024-08-27 19:56:20.244 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.244 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-27 19:56:20.249 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: inputTexture=androidx.media3.common.GlTextureInfo@51e72ea
2024-08-27 19:56:20.249 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: presentationTimeUs=1000000041666
2024-08-27 19:56:20.249 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: renderTimeNs=440391116378964
2024-08-27 19:56:20.253 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: eglDisplay=android.opengl.EGLDisplay@210
2024-08-27 19:56:20.253 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputEglSurface=android.opengl.EGLSurface@4edfb0a5
2024-08-27 19:56:20.253 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.width=1920
2024-08-27 19:56:20.253 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.height=1080
2024-08-27 19:56:20.254 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface after EGL14.eglSwapBuffers was called
2024-08-27 19:56:20.263 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.263 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-27 19:56:20.265 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: inputTexture=androidx.media3.common.GlTextureInfo@51e72ea
2024-08-27 19:56:20.265 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: presentationTimeUs=1000000083333
2024-08-27 19:56:20.265 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: renderTimeNs=440391166428963
2024-08-27 19:56:20.268 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: eglDisplay=android.opengl.EGLDisplay@210
2024-08-27 19:56:20.268 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputEglSurface=android.opengl.EGLSurface@4edfb0a5
2024-08-27 19:56:20.268 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.width=1920
2024-08-27 19:56:20.268 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.height=1080
2024-08-27 19:56:20.269 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface after EGL14.eglSwapBuffers was called
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: inputTexture=androidx.media3.common.GlTextureInfo@51e72ea
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: presentationTimeUs=1000000125000
2024-08-27 19:56:20.294 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: renderTimeNs=440391199795629
2024-08-27 19:56:20.296 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: eglDisplay=android.opengl.EGLDisplay@210
2024-08-27 19:56:20.296 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputEglSurface=android.opengl.EGLSurface@4edfb0a5
2024-08-27 19:56:20.296 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.width=1920
2024-08-27 19:56:20.296 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface: outputSurfaceInfo.height=1080
2024-08-27 19:56:20.297 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrameToOutputSurface after EGL14.eglSwapBuffers was called
2024-08-27 19:56:20.345 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.width=1280
2024-08-27 19:56:20.345 23576-23620 issue1620               com...hub.damontecres.myapplication  D  FinalShaderProgramWrapper.renderFrame inputTexture.height=720

@claincly
Copy link
Contributor

Hmmm that is indeed very strange - I mean if eglSwapBuffers() is called, then we have already send the frame to display on the screen.

I also noticed you mentioned you didn't try reproducing using the demo app, could you try using the demo app and see it reproduces? I just wanted to rule out any unrelated reasons.

@damontecres
Copy link
Author

The no video issue is reproducible in the basic demo app with a change to import the lib-effect module (it crashes when calling setVideoEffects otherwise).

I tried on another NVIDIA shield device (2017 model) and the issue is reproducible there as well.

I also tried the demo-transformer app on the same device and set some random RGB video effects. It produced a correctly transformed video including playback during export in the "debug preview". It looks like this debug preview is a Surface but not from/connected to a PlayerView if I'm reading the code correctly. Maybe that is a clue?

Here is the patch just in case:

diff --git a/demos/main/build.gradle b/demos/main/build.gradle
index 6a49adf169..78a43f180b 100644
--- a/demos/main/build.gradle
+++ b/demos/main/build.gradle
@@ -85,6 +85,7 @@ dependencies {
     implementation project(modulePrefix + 'lib-ui')
     implementation project(modulePrefix + 'lib-datasource-cronet')
     implementation project(modulePrefix + 'lib-exoplayer-ima')
+    implementation project(modulePrefix + 'lib-effect')
     withDecoderExtensionsImplementation project(modulePrefix + 'lib-decoder-av1')
     withDecoderExtensionsImplementation project(modulePrefix + 'lib-decoder-ffmpeg')
     withDecoderExtensionsImplementation project(modulePrefix + 'lib-decoder-flac')
diff --git a/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java b/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java
index 6be37323bd..2b3c11977a 100644
--- a/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java
+++ b/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java
@@ -258,7 +258,7 @@ public class PlayerActivity extends AppCompatActivity
   /**
    * @return Whether initialization was successful.
    */
-  protected boolean initializePlayer() {
+  @OptIn(markerClass = UnstableApi.class) protected boolean initializePlayer() {
     Intent intent = getIntent();
     if (player == null) {
 
@@ -279,6 +279,7 @@ public class PlayerActivity extends AppCompatActivity
       player.addAnalyticsListener(new EventLogger());
       player.setAudioAttributes(AudioAttributes.DEFAULT, /* handleAudioFocus= */ true);
       player.setPlayWhenReady(startAutoPlay);
+      player.setVideoEffects(new ArrayList<>());
       playerView.setPlayer(player);
       configurePlayerWithServerSideAdsLoader();
       debugViewHelper = new DebugTextViewHelper(player, debugTextView);

@claincly
Copy link
Contributor

It looks like this debug preview is a Surface but not from/connected to a PlayerView if I'm reading the code correctly. Maybe that is a clue?

Interesting find. You are correct the debug preview is not a PlayerView (I think it's just a SurfaceView). The only difference I can think of is we call eglPresentationTimeANDROID() when rendering to a PlayerView. Now that method is enabled by an OpenGL extension which might now work reliably on all devices.

Although I don't recommend it, could you try commenting that line out and see if the video plays?

EGLExt.eglPresentationTimeANDROID(
eglDisplay,
outputEglSurface,
renderTimeNs == VideoFrameProcessor.RENDER_OUTPUT_FRAME_IMMEDIATELY
? System.nanoTime()
: renderTimeNs);

@damontecres
Copy link
Author

Although I don't recommend it, could you try commenting that line out and see if the video plays?

Commenting out the EGLExt.eglPresentationTimeANDROID method call does not resolve the issue, still just a black screen with audio.

I also tried updating to 1.4.1 just in case, but no joy.

@claincly
Copy link
Contributor

claincly commented Aug 29, 2024

One last thing to try - Nvidia might have very customized OpenGL stack),

Adding glFinish() after calling eglSwapBuffers(), and

change the color to something like red in clearFocusedBuffers(), here to something not black, maybe set red to 1

public static void clearFocusedBuffers() throws GlException {
GLES20.glClearColor(/* red= */ 0, /* green= */ 0, /* blue= */ 0, /* alpha= */ 0);

@damontecres
Copy link
Author

Adding glFinish() after calling eglSwapBuffers(), and

No change

change the color to something like red in clearFocusedBuffers(), here to something not black, maybe set red to 1

Screen is still black!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants