react-native-vision-camera-mlkit

📦 A Google ML Kit frame processor plugin for react-native-vision-camera.

This plugin provides a simple way to use various ML Kit Vision APIs in your React Native App's Frame processor.

:warning: This plugin is still in development and not yet ready for iOS.

🧵 Vision APIs

Video and image analysis APIs to label images and detect barcodes, text, faces, and objects.

• [X] Barcode scanning Scan and process barcodes. Supports most standard 1D and 2D formats.
• [ ] Face detection Detect faces and facial landmarks.
• [ ] Face mesh detection Detect face mesh info on close-range images.
• [X] Text recognition v2 Recognize and extract text from images.
• [X] Image labeling Identify objects, locations, activities, animal species, products, and more. Use a general-purpose base model or tailor to your use case with a custom TensorFlow Lite model.
• [X] Object detection and tracking Localize and track in real time one or more objects in the live camera feed.
• [ ] Digital ink recognition Recognizes handwritten text and handdrawn shapes on a digital surface, such as a touch screen. Recognizes 300+ languages, emojis and basic shapes.
• [ ] Pose detection Detect the position of the human body in real time.
• [ ] Selfie segmentation Separate the background from users within a scene and focus on what matters.
• [ ] Subject segmentation Separate subjects (people, pets, or objects) from the background in a picture.
• [ ] Document scanner Digitize physical documents from pictures.

🚀 Getting Started

🚨 Required Dependencies

Ensure you have installed the required packages before installing this plugin.

| Package | Version | | - | - | | react-native-vision-camera | >=4.0.1 | | react-native-worklets-core | >=1.2.0 |

Follow the installation instructions for each package.

💻 Installation

To install the plugin, run:

npm install react-native-vision-camera-mlkit
# or
yarn add react-native-vision-camera-mlkit

🪝 Hooks

useBarcodeScanner (Barcode scanning)

  const { barcodeScanner } = useBarcodeScanner();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const barcodes = barcodeScanner(frame);
        console.log(barcodes);
      });
    },
    [barcodeScanner]
  );

useImageLabeler (Image labeling)

  const { imageLabeler } = useImageLabeler();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const labels = imageLabeler(frame);
        console.log(labels);
      });
    },
    [imageLabeler]
  );

useObjectDetector (Object detection and tracking)

  const { objectDetector } = useObjectDetector();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const objects = objectDetector(frame);
        console.log(objects);
      });
    },
    [objectDetector]
  );

useTextRecognizer (Text recognition v2)

  const { textRecognizer } = useTextRecognizer();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const text = textRecognizer(frame);
        console.log(text);
      });
    },
    [textRecognizer]
  );