Camera Screen goes black when using Skia Frame Processors with VisionCamera (doesn't happen with normal frame processors)

[DragonDare]

Description

I'm trying to implement pose detection by drawing stick-figure of the pose on my app. But with Skia Frame processor, the camera screen goes black with only the stick figure being visible. However, if i use a normal frame processor, the camera is visible and not black.

Packages:

"react-native-fast-tflite": "^1.5.1",
"@shopify/react-native-skia": "1.5.0",
"react-native-worklets-core": "^1.5.0",
"vision-camera-resize-plugin": "^3.2.0"
"react-native-vision-camera": "^4.6.3"

React Native Skia Version

1.5.0

React Native Version

0.76.6

Using New Architecture

• Enabled

Steps to Reproduce

1. Reproduce Mrousavy's SkiaCameraPoseDetection example
2. Remove the black paint
3. Still, camera isn't visible, but Pose Skeleton is.
4. If we use normal frame processor instead of skia, camera is visible.

Snack, Code Example, Screenshot, or Link to Repository

Here is my camera screen code which is creating the problem.

import TopAppBar from '@/components/TopAppBar';
import { SlamdunkTheme } from '@/constants/Theme';
import { scaleHeight, scaleWidth } from '@/utils/scaling';
import React, { useEffect, useMemo } from 'react';
import { StyleSheet, View, Text, Platform } from 'react-native';
import { SafeAreaView } from 'react-native-safe-area-context';
import { Camera, useCameraDevice, useCameraPermission, useSkiaFrameProcessor } from 'react-native-vision-camera';
import { useResizePlugin } from 'vision-camera-resize-plugin';
import { TensorflowModel, useTensorflowModel } from 'react-native-fast-tflite';
import { PaintStyle, Skia } from '@shopify/react-native-skia';
import getBestFormat from './formatFilter';

function tensorToString(tensor: TensorflowModel['inputs'][number]): string {
return ${tensor.dataType} [${tensor.shape}];
}

const LINE_WIDTH = 1;
const MIN_CONFIDENCE = 0.2;

const VIEW_WIDTH = scaleWidth(375);

const CameraScreen = () => {
const device = useCameraDevice('back');

const { hasPermission, requestPermission } = useCameraPermission();

useEffect(() => {
if (!hasPermission) {
requestPermission().catch((err) => {
console.error('Camera permission request failed:', err);
});
}
}, [hasPermission, requestPermission]);

if (!device) {
return Loading camera...;
}

const { resize } = useResizePlugin();

const delegate = Platform.OS === 'ios' ? 'core-ml' : undefined;
const plugin = useTensorflowModel(
require('../assets/models/singlepose-lightning-tflite-float16.tflite'),
delegate,
);
const format = useMemo(
() => (device != null ? getBestFormat(device, 720, 1000) : undefined),
[device],
);
console.log(format?.videoWidth, format?.videoHeight);

const pixelFormat = Platform.OS === 'ios' ? 'rgb' : 'yuv';

useEffect(() => {
const model = plugin.model;
if (model == null) {
return;
}
console.log(
Model: ${model.inputs.map(tensorToString)} -> ${model.outputs.map( tensorToString, )},
);
}, [plugin]);

const inputTensor = plugin.model?.inputs[0];
const inputWidth = inputTensor?.shape[1] ?? 0;
const inputHeight = inputTensor?.shape[2] ?? 0;
if (inputTensor != null) {
console.log(
Input: ${inputTensor.dataType} ${inputWidth} x ${inputHeight},
);
}

// to get from px -> dp since we draw in the camera coordinate system
const SCALE = (format?.videoWidth ?? VIEW_WIDTH) / VIEW_WIDTH;

const paint = Skia.Paint();
paint.setStyle(PaintStyle.Fill);
paint.setStrokeWidth(LINE_WIDTH * SCALE);
paint.setColor(Skia.Color('yellow'));

const lines = [
// left shoulder -> elbow
5, 7,
// right shoulder -> elbow
6, 8,
// left elbow -> wrist
7, 9,
// right elbow -> wrist
8, 10,
// left hip -> knee
11, 13,
// right hip -> knee
12, 14,
// left knee -> ankle
13, 15,
// right knee -> ankle
14, 16,

// left hip -> right hip
11, 12,
// left shoulder -> right shoulder
5, 6,
// left shoulder -> left hip
5, 11,
// right shoulder -> right hip
6, 12,

];

const rotation = Platform.OS === 'ios' ? '0deg' : '90deg'; // hack to get android oriented properly

const frameProcessor = useSkiaFrameProcessor(
frame => {
'worklet';

  if (plugin.model != null) {
    const smaller = resize(frame, {
      scale: {
        width: inputWidth,
        height: inputHeight,
      },
      pixelFormat: 'rgb',
      dataType: 'uint8',
      rotation: rotation,
    });
    const outputs = plugin.model.runSync([smaller]);

    const output = outputs[0];
    const frameWidth = frame.width;
    const frameHeight = frame.height;

    for (let i = 0; i < lines.length; i += 2) {
      const from = lines[i];
      const to = lines[i + 1];

      const confidence = output[from * 3 + 2];
      if (confidence > MIN_CONFIDENCE) {
        frame.drawLine(
          Number(output[from * 3 + 1]) * Number(frameWidth),
          Number(output[from * 3]) * Number(frameHeight),
          Number(output[to * 3 + 1]) * Number(frameWidth),
          Number(output[to * 3]) * Number(frameHeight),
          paint,
        );
      }
    }
  }
},
[plugin, paint],

);

return (




Make sure your body is positioned correctly


{hasPermission && (

)}
{/*

*/}

);
};

export default CameraScreen;

const styles = StyleSheet.create({
overlay: {
...StyleSheet.absoluteFillObject,
top: scaleHeight(161),
alignItems: 'center',
},
textBox: {
width: scaleWidth(290),
alignSelf: 'center',
},
camera: {
height: scaleHeight(638),
},
instructionText: {
fontFamily: SlamdunkTheme.fonts.poppinsRegular,
fontSize: 22,
fontWeight: 400,
lineHeight: 33,
color: SlamdunkTheme.colors.backgroundColor,
textAlign: 'center',
},
dottedBox: {
width: scaleWidth(319),
height: scaleHeight(514),
borderWidth: 2,
borderColor: SlamdunkTheme.colors.backgroundVariant3,
borderStyle: 'dashed',
backgroundColor: SlamdunkTheme.colors.backgroundVariant3,
borderRadius: 8,
},
});

[maxximee]

Same issue here.
Can be reproduced with a very simple usecase:

 const defaultPaint = Skia.Paint()
 const frameProcessor = useSkiaFrameProcessor(frame => {
        'worklet'
        frame.render(defaultPaint)
    }, [defaultPaint])

-> in component:
frameProcessor={frameProcessor}

This will show black screen on some devices.

I think it's linked to devices who are using Vulkan and Angle for OpenGL support, including a lot of samsung devices as wcandillon stated.

@DragonDare are you on a samsung device/emulator?

Also I think this issue should be addressed in react-native-vision-camera

[wcandillon]

yes I would definitely recommend to try on 1.7.7 and above.

[DragonDare]

Same issue here. Can be reproduced with a very simple usecase:

 const defaultPaint = Skia.Paint()
 const frameProcessor = useSkiaFrameProcessor(frame => {
        'worklet'
        frame.render(defaultPaint)
    }, [defaultPaint])

-> in component: frameProcessor={frameProcessor}

This will show black screen on some devices.

I think it's linked to devices who are using Vulkan and Angle for OpenGL support, including a lot of samsung devices as wcandillon stated.

@DragonDare are you on a samsung device/emulator?

Also I think this issue should be addressed in react-native-vision-camera

Im using a Motorola Edge 40 physical device.

Also I did mention this issue in vision camera github, no comments so far. Could you bump up the issue there? Will get some recognition.

[DragonDare]

yes I would definitely recommend to try on 1.7.7 and above.

Tried with skia version 1.7.7 right now, same problem.

[DragonDare]

@wcandillon any updates on this issue? If possible, could we atleast debug if this is a vision camera issue or a skia issue? Thanks

[wcandillon]

@DragonDare Yes I think it would be interesting to have proper support for VisionCamera, would you be ok to add a small example of Skia/Vision Camera to the example app: https://github.com/Shopify/react-native-skia/tree/main/apps/paper
That way we can monitor if the integration breaks or not.

@mrousavy Is that something you monitor on your side as well? Maybe we need to do some touch up on the integration part.

[DragonDare]

@wcandillon I did try the example repo https://github.com/mrousavy/VisionCameraSkiaDemo, which uses Skia Frame processors and it did give me a black screen (Yes i removed the black paint in the code)

[mrousavy]

@mrousavy Is that something you monitor on your side as well? Maybe we need to do some touch up on the integration part.

Yea I added Skia as a dependency - but I don't update it everytime a new version gets released. It definitely helps to also add VisionCamera to the Skia example app to test this integration better!