Skip to main content

Drawing to a Frame (Skia)

What is Skia?

Skia is a 2D graphics library that can be used to draw shapes, images, text, color-shaders and much more. Skia is GPU-accelerated by Metal on iOS and OpenGL on Android.

To provide a powerful cross-platform API for drawing directly to Camera Frames in realtime, VisionCamera provides a first-party react-native-skia integration via Skia Frame Processors:

const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
const bananas = detectBananas()

frame.render()
for (const banana of bananas) {
const paint = Skia.Paint()
paint.setColor(Skia.Color('red'))
frame.drawRect(banana.rect, paint)
}
}, [])

Installation

Skia Frame Processors require @shopify/react-native-skia 1.2.1 or higher, and react-native-reanimated 3.0.0 or higher. Install the packages through npm and make sure you follow their installation instructions:

npm i @shopify/react-native-skia
npm i react-native-reanimated

Skia Frame Processors

A Skia Frame Processor, just like any other Frame Processor, runs synchronously for every Camera Frame. Instead of a Frame, it is called with a DrawableFrame, which extends the Frame with a drawing canvas.

To create a Skia Frame Processor, use the useSkiaFrameProcessor hook. In a Skia Frame Processor the caller is responsible for rendering the Camera Frame, so Frame.render() must always be called:

const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render()
}, [])

Skia APIs

To draw something to the Frame, use Skia's imperative APIs. For example, to draw a red rectangle in the center of the Frame, use drawRect(...):

const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render()

const centerX = frame.width / 2
const centerY = frame.height / 2
const rect = Skia.XYWHRect(centerX, centerY, 150, 150)
const paint = Skia.Paint()
paint.setColor(Skia.Color('red'))
frame.drawRect(rect, paint)
}, [])

The Camera Frame is rendered like any other SkImage. You can pass a custom SkPaint object to the render(..) function to use a shader, for example to render the Frame with inverted colors use a RuntimeEffect:

const invertColorsFilter = Skia.RuntimeEffect.Make(`
uniform shader image;
half4 main(vec2 pos) {
vec4 color = image.eval(pos);
return vec4((1.0 - color).rgb, 1.0);
}
`)
const shaderBuilder = Skia.RuntimeShaderBuilder(invertColorsFilter)
const imageFilter = Skia.ImageFilter.MakeRuntimeShader(shaderBuilder, null, null)
const paint = Skia.Paint()
paint.setImageFilter(imageFilter)

const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render(paint)
}, [paint])

Coordinate system

Each Frame, a Skia Frame Processor is rendering to an offscreen SkSurface. The Camera Frame is a GPU-texture-backed SkImage, and it's coordinate system is in Frame dimensions and orientation.

  • (0, 0) is top left
  • (frame.width, frame.height) is bottom right

Performance

Just like normal Frame Processors, Skia Frame Processors are really fast. Skia is GPU-accelerated by Metal and OpenGL, and VisionCamera Frames are streamed as efficiently as possible using GPU-buffers and textures.

A Skia Frame Processor can run and render at up to 500 FPS, depending on how complex the rendering code is.

Preview-only

info

Skia Frame Processors are currently preview-only. Any content drawn to the Frame will not be visible in captured photos, snapshots or videos.

We at Margelo have worked a lot with 2D/3D graphics and Camera realtime processing (see the Snapchat-style mask filter on our website for example - that is running in VisionCamera/React Native!), if you need to capture drawn content to photos or videos, reach out to us and we'll build a customized/tailored solution for your company! :)


🚀 Next section: Zooming (or creating a Frame Processor Plugin)