react-native-vision-camera-selfie-segmentation

A VisionCamera Frame Processor Plugin to mask selfies using MLKit Vision Selfie Segmentation.

Notice: This repo was made as a proof of concept. If you're looking to use the MLKit Selfie Segmentation on a live camera view, it doesn't appear possible with Vision Camera right now, but feel free to give it a go.

It's probably much easier to use the Selfie Segmentation on static images only.

Installation

npm install react-native-vision-camera-selfie-segmentation
cd ios && pod install

Add the plugin to your babel.config.js:

module.exports = {
  plugins: [
    [
      'react-native-reanimated/plugin',
      {
        globals: ['__getSelfieSegments'],
      },
    ],
    // ...

Note: You have to restart metro-bundler for changes in the babel.config.js file to take effect.

Usage

import { getSelfieSegments } from 'react-native-vision-camera-selfie-segmentation';

// ...

const frameProcessor = useFrameProcessor((frame) => {
  'worklet';
  const base64Image = getSelfieSegments(frame, '#FF0000', '#00FF00');
}, []);

Props

| Prop | Type | Default | Definition | |-----------------|---------|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| | frame | Frame | frame | Required - The frame as passed by the useFrameProcessor hook | | backgroundColor | string | #000000 | Required - The background colour of the background pixels of the image | | foregroundColor | string? | null | Optional - The foreground color of the image, used for the selfie pixels of the image. If none is passed, the original image will be used, showing the person. |

Response

| Response | Type | Default | Definition | |--------------|--------|-------|-------------------------------------------------------------------------------------------------| | Base64 Image | String | "" | A base 64 image, containing "data:image/jpeg;base64," and relevant base64 data of the new image |

Example

const [image, setImage] = useState<string>('');

const updateImage = (newImage: string) => {
  setImage(newImage);
};

const frameProcessor = useFrameProcessor((frame) => {
'worklet';
  const image = getSelfieSegments(frame, '#FFF000');
  runOnJS(updateImage)(image);
}, []);

return (
  <>
    {image !== '' && <Image source={{uri: image}} />}
  </>
)

Known Issues

1. Originally, I wanted to pass the selfie segments data back to the frame to mask the original camera. However, this didn't seem possible, but maybe in future.
2. Performance - Due to the intensive amount of work required to get the selfie segmentation data, you'll be lucky to get 2-3 FPS. Though, this is mainly due to my lack of knowledge in Swift/Java, so can probably be improved.
3. Android - There's a flash between images as it's updated in state.

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

License

MIT