Orientation

Camera Sensor Orientation

A Camera sensor is configured to deliver buffers in a specific size, for example 4096x2160. To avoid re-allocating such large buffers every time the phone rotates, the Camera pipeline will always deliver buffers in it's native sensor orientation (see sensorOrientation) and frames need to be rotated to appear up-right.

Since actually rotating pixels in such large buffers is really expensive and causes an unnecessary performance overhead, VisionCamera applies rotations through flags or transforms:

Photo & Video Output

Photos and videos will be captured in a potentially "wrong" orientation, and VisionCamera will write an EXIF flag to the photo/video file with the correct presentation rotation.

info

This is handled automatically, and it's behaviour can be controlled via the outputOrientation property.

Preview View

The Preview output will stream in a potentially "wrong" orientation, but uses view transforms (rotate + translate matrix) to properly display the Camera stream "up-right".

info

This will always happen automatically according to the screen's rotation.

Frame Processor Output

Frame Processors will stream frames in a potentially "wrong" orientation, and the client is responsible for properly interpreting the Frame data.

info

This needs to be handled manually, see Frame.orientation. For example, in MLKit just pass the Frame's orientation to the detect(...) method.

Instead of always rotating up-right to portrait, you might also want to rotate the Frame to either preview-, or output-orientation, depending on your use-case.

Implementation

VisionCamera supports three ways to implement orientation:

• Camera UI (preview view) is locked, but the buttons can rotate to the desired photo/video output orientation (recommended)
• Camera UI (preview view) also rotates alongside with the photo/video output orientation
• Both Camera UI (preview view) and photo/video output orientation are locked to a specific orientation

The outputOrientation prop

The orientation in which photos and videos are captured can be adjusted via the outputOrientation property:

<Camera {...props} outputOrientation="device" />

• "device": With the output orientation set to device (the default), photos and videos will be captured in the phone's physical orientation, even if the screen-rotation is locked. This means, even though the preview view and other views don't rotate to landscape, holding the phone in landscape mode will still capture landscape photos. This is the same behaviour as in the iOS stock Camera.

• "preview": Similar to device, the preview orientation mode will follow the phone's physical orientation, allowing the user to capture landscape photos and videos - but will always respect screen-rotation locks. This means that the user is not able to capture landscape photos or videos if the preview view and other views stay in portrait mode (e.g. if the screen-lock is on).

Listen to orientation changes

Whenever the output orientation changes, the onOutputOrientationChanged event will be called with the new output orientation. This is a good point to rotate the buttons to the desired output orientation now.

The onPreviewOrientationChanged event will be called whenever the preview orientation changes, which might be unrelated to the output orientation. Depending on the device's natural orientation (e.g. iPads being landscape by default), you should rotate all buttons on the UI relative to the preview orientation.

As a helper method, VisionCamera fires the onUIRotationChanged event whenever the target UI rotation changes. You can directly apply this rotation to any UI elements such as buttons to rotate them to the correct orientation:

const [uiRotation, setUiRotation] = useState(0)
const uiStyle: ViewStyle = {
  transform: [{ rotate: `${uiRotation}deg` }]
}

return (
  <View>
    <Camera {...props} onUIRotationChanged={setUiRotation} />
    <FlipCameraButton style={uiStyle} />
  </View>
)

tip

For a smoother user experience, you should animate changes to the UI rotation. Use a library like react-native-reanimated to smoothly animate the rotate style.

The Frame's orientation

In a Frame Processor, frames are streamed in their native sensor orientation. This means even if the phone is rotated from portrait to landscape, the Frame's width and height stay the same.

The Frame's orientation represents the image buffer's orientation, relative to the device's native portrait mode.

For example, if the Frame's orientation is landscape-right, it is 90° rotated and needs to be counter-rotated by -90° to appear "up-right".

On an iPhone, "up-right" means portrait mode (the home-button is at the bottom). On an iPad, "up-right" might mean a landscape orientation.

Instead of actually rotating pixels in the buffers, frame processor plugins just need to interpret the frame as being rotated.

MLKit handles this via a orientation property on the MLImage/VisionImage object:

public override func callback(_ frame: Frame, withArguments _: [AnyHashable: Any]?) -> Any? {
  let mlImage = MLImage(sampleBuffer: frame.buffer)
  mlImage.orientation = frame.orientation
  // ...
}

You can then either rotate to preview-, or output-orientation, depending on your use-case.

Rotate Frame.orientation to output Orientation

If you have a Frame Processor that detects objects or faces and the user holds the phone in a landscape orientation, your algorithm might not be able to detect the object or face because it is rotated.

In this case you can just rotate the Frame.orientation by the outputOrientation (see onOutputOrientationChanged), and it will then be a landscape Frame if the user rotates the phone to landscape, or a portrait Frame if the user holds the phone in portrait.

Rotate Frame.orientation to preview Orientation

If you have a Frame Processor tht applies some drawing operations or provides visual feedback to the Preview, you don't want to use the outputOrientation as that can be different than the previewOrientation.

In this case you can follow the same idea as above, just rotate the Frame.orientation by the previewOrientation (see onPreviewOrientationChanged) to receive a Frame in the same orientation the Preview view is currently in.

Orientation in Skia Frame Processors

A Skia Frame Processor applies orientation via rotation and translation. This means the coordinate system stays the same, but output will be rotated accordingly. For a landscape-left frame, (0,0) will not be top left, but rather top right.

Mirroring (isMirrored)

The photo-, video- and snapshot- outputs can be mirrored alongside the vertical axis (left/right flipped) by setting isMirrored to true. By default, outputs are mirrored in the selfie camera, and not mirrored in any other cameras. To disable mirroring even for the selfie camera, just set isMirrored to false.

The preview view is always mirrored in the selfie camera, and never mirrored in any other cameras.

<Camera {...props} isMirrored={true} />

🚀 Next section: Exposure