Stereo camera¶

The stereo camera component contains functionality for acquiring stereo image pairs and performing planar rectification needed to use the stereo camera as a measurement device.

Image acquisition¶

Acquiring stereo image pairs is the first step toward stereo vision. Since both cameras are equipped with global shutters and their chips are hardware-synchronized, all pixels of both camera images are always exposed at exactly the same time. The time at the middle of the image exposure is attached to the images as a timestamp. This timestamp becomes important for dynamic applications in which the cameras or the scene moves.

Exposure time can be set manually to a fixed value. This is useful in an environment where lighting is controlled so that it is always at the same intensity. The camera is set to auto exposure by default. In this mode, the exposure time is chosen automatically, up to a user defined maximum. The permitted maximum is meant to limit the motion blur that occurs when taking images while the camera or the scene is moving. The maximum exposure time thus depends on the application. If the maximum exposure time is reached, the auto-exposure algorithm uses the gain to increase image brightness. However, larger gain factors also amplify image noise. Thus, the maximum exposure time trades motion blur off against image noise under weak-light conditions.

Planar rectification¶

Camera parameters such as focal length, lens distortion, and the relationship of the cameras to each other must be exactly known to use the stereo camera as a measuring instrument. The rc_visard is already calibrated at production time and normally requires no recalibration. The camera parameters describe with great precision all of the stereo-camera system’s geometric properties, but the resulting model is complex and difficult to use.

Rectification is the process of remapping the images according to an ideal stereo-camera model. Lens distortion is removed and the images are aligned so that an object point is always projected onto the same image row in both images. The cameras’ optical axes become exactly parallel. This means that points at infinite distance are projected onto the same image column in both images. The closer an object point is, the larger is the difference between its image columns in the right and left images. This difference is called disparity.

Mathematically, the object point \(P=(P_x, P_y, P_z)\) is projected onto image point \(p_l=(p_{lx}, p_{ly}, 1)\) in the left rectified image and onto \(p_r=(p_{rx}, p_{ry}, 1)\) in the right rectified image by

\[ \begin{align}\begin{aligned}\begin{split}A &= \left(\begin{array}{ccc} f & 0 & \frac{w}{2}\\ 0 & f & \frac{h}{2}\\ 0 & 0 & 1 \end{array}\right), \qquad T_s = \left(\begin{array}{c}t\\ 0\\ 0 \end{array}\right),\end{split}\\\begin{split}s_1 p_l &= AP, \\ s_2 p_r &= A(P-T_s).\end{split}\end{aligned}\end{align} \]

The focal length \(f\) is the distance between the common image plane and the optical centers of the left and right cameras. It is measured in pixels. The baseline \(t\) is the distance between the optical centers of the two cameras. The image width \(w\) and height \(h\) are measured in pixels, too. \(s_1\) and \(s_2\) are scale factors ensuring that the third coordinates of the image points \(p_l\) and \(p_r\) are equal to 1.

Note

The rc_cube reports a focal length factor via its various interfaces. It relates to the image width for supporting different image resolutions. The focal length \(f\) in pixels can be easily obtained by multiplying the focal length factor by the image width in pixels.

Accessing Images¶

The rc_cube provides the time-stamped, rectified left and right images over the GenICam interface (see Provided image streams). Live streams of the images are provided with reduced quality in the Web GUI.

The Web GUI also provides the opportunity to download a snapshot of the current scene as a .tar.gz file by clicking on the camera icon below the image live streams on the Camera tab. This snapshot contains:

the rectified left and right camera images in full resolution as .png files,
a camera parameter file containing the camera matrix, image dimensions, exposure time, gain value and the stereo baseline,
the current IMU readings as imu.csv file,
a nodes.json file containing information about all nodes running on the rc_cube,
a system_info.json file containing system information about the rc_cube.

Parameters¶

The stereo-camera software component is called rc_stereocamera and is represented by the Camera tab in the Web GUI. The user can change the camera parameters there, or directly via the REST-API (REST-API interface) or GigE Vision (GigE Vision 2.0/GenICam image interface).

Note

Camera parameters cannot be changed via the Web GUI or REST-API if rc_cube is used via GigE Vision.

Parameter overview¶

This component offers the following run-time parameters:

Table 1 The `rc_stereocamera` component’s run-time parameters¶
Name	Type	Min	Max	Default	Description
`exp_auto`	bool	false	true	true	Switching between auto and manual exposure
`exp_auto_average_max`	float64	0.0	1.0	0.75	Maximum average intensity if exp_auto is true
`exp_auto_average_min`	float64	0.0	1.0	0.25	Minimum average intensity if exp_auto is true
`exp_auto_mode`	string	-	-	Normal	Auto-exposure mode, i.e. Normal, Out1High or AdaptiveOut1
`exp_height`	int32	0	959	0	Height of auto exposure region. 0 for whole image.
`exp_max`	float64	6.6e-05	0.018	0.018	Maximum exposure time in seconds if exp_auto is true
`exp_offset_x`	int32	0	1279	0	First column of auto exposure region
`exp_offset_y`	int32	0	959	0	First row of auto exposure region
`exp_value`	float64	6.6e-05	0.018	0.005	Manual exposure time in seconds if exp_auto is false
`exp_width`	int32	0	1279	0	Width of auto exposure region. 0 for whole image.
`fps`	float64	1.0	25.0	25.0	Frames per second in Hertz
`gain_value`	float64	0.0	18.0	0.0	Manual gain value in decibel if exp_auto is false
`wb_auto`	bool	false	true	true	Switching white balance on and off (only for color camera)
`wb_ratio_blue`	float64	0.125	8.0	2.4	Blue to green balance ratio if wb_auto is false (only for color camera)
`wb_ratio_red`	float64	0.125	8.0	1.2	Red to green balance ratio if wb_auto is false (only for color camera)

Description of run-time parameters¶

Fig. 7 The Web GUI’s Camera tab

fps (FPS): This value is the cameras’ frame rate (fps, frames per second), which determines the upper frequency at which depth images can be computed. This is also the frequency at which the rc_cube delivers images via GigE Vision. Reducing this frequency also reduces the network bandwidth required to transmit the images.

exp_auto (Exposure Auto or Manual): This value can be set to true for auto-exposure mode, or to false for manual exposure mode. In manual exposure mode, the chosen exposure time is kept, even if the images are overexposed or underexposed. In auto-exposure mode, the exposure time and gain factor is chosen automatically to correctly expose the image. The last automatically determined exposure and gain values are set into exp_value and gain_value when switching auto-exposure off.

exp_auto_mode (Auto Exposure Mode)

The auto exposure mode can be set to Normal, Out1High or AdaptiveOut1. These modes are relevant when the rc_cube is used with an external light source or projector connected to the rc_visard’s GPIO Out1, which can be controlled by the optional IOControl component (IO and Projector Control).

Normal: All images are considered for exposure control, except if the IOControl mode for GPIO Out1 is ExposureAlternateActive: then only images where GPIO Out1 is HIGH will be considered, since these images may be brighter in case GPIO Out1 is used to trigger an external light source.

Out1High: This exposure mode adapts the exposure time using only images with GPIO Out1 HIGH. Images where GPIO Out1 is LOW are not considered at all, which means, that the exposure time does not change when only images with Out1 LOW are acquired. This mode is recommended for using the acquisition_mode SingleFrameOut1 in the stereo matching component as described in Stereo Matching Parameters and having an external projector connected to GPIO Out1, when changes in the brightness of the scene should only be considered when Out1 is HIGH. This is the case, for example, when a bright part of the robot moves through the field of view of the camera just before a detection is triggered, which should not affect the exposure time.

AdaptiveOut1: This exposure mode uses all camera images and tracks the exposure difference between images with GPIO Out1 LOW and HIGH. While the IOControl mode for GPIO Out1 is LOW, the images are under-exposed by this exposure difference to avoid over-exposure for when GPIO Out1 triggers an external projector. The resulting exposure difference is given as Out1 Reduction below the live images. This mode is recommended for using the acquisition_mode SingleFrameOut1 in the stereo matching component as described in Stereo Matching Parameters and having an external projector connected to GPIO Out1, when changes in the brightness of the scene should be considered at all times. This is the case, for example, in applications where the external lighting is changing.

exp_max (Max Exposure): This value is the maximal exposure time in auto-exposure mode in seconds. The actual exposure time is adjusted automatically so that the images are exposed correctly. If the maximum exposure time is reached, but the images are still underexposed, the rc_cube stepwise increases the gain to increase the images’ brightness. Limiting the exposure time is useful for avoiding or reducing motion blur during fast movements. However, higher gain introduces noise into the image. The best trade-off depends on the application.

exp_auto_average_max (Max Brightness) and exp_auto_average_min (Min Brightness)

The auto-exposure tries to set the exposure time and gain factor such that the average intensity (i.e. brightness) in the image or exposure region is between a maximum and a minimum. The maximum brightness will be used if there is no saturation, e.g. no over-exposure due to bright surfaces or reflections. In case of saturation, the exposure time and gain factor are reduced, but only down to the minimum brightness.

The maximum brightness has precendence over the minimum brightness parameter. If the minimum brightness is larger than the maximum brightness, the auto-exposure always tries to make the average intensity equal to the maximum brightness.

The current brightness is always shown in the status bar below the images.

exp_offset_x, exp_offset_y, exp_width, exp_height (Exposure Region)

These values define a rectangular region in the left rectified image for limiting the area used for computing the auto exposure. The exposure time and gain factor of both images are chosen to optimally expose the defined region. This can lead to over- or underexposure of image parts outside the defined region. If either the width or height is 0, then the whole left and right images are considered by the auto exposure function. This is the default.

The region is visualized in the Web GUI by a rectangle in the left rectified image. It can be defined using the sliders or by selecting it in the image after pressing the button Select Region in Image.

exp_value (Exposure)

This value is the exposure time in manual exposure mode in seconds. This exposure time is kept constant even if the images are underexposed.

gain_value (Gain): This value is the gain factor in decibel that can be set in manual exposure mode. Higher gain factors reduce the required exposure time but introduce noise.

wb_auto (White Balance Auto or Manual): This value can be set to true for automatic white balancing or false for manually setting the ratio between the colors using wb_ratio_red and wb_ratio_blue. The last automatically determined ratios are set into wb_ratio_red and wb_ratio_blue when switching automatic white balancing off. White balancing is without function for monochrome cameras and will not be displayed in the Web GUI in this case.

wb_ratio_blue and wb_ratio_red (Blue | Green and Red | Green): These values are used to set blue to green and red to green ratios for manual white balance. White balancing is without function for monochrome cameras and will not be displayed in the Web GUI in this case.

These parameters are also available over the GenICam interface with slightly different names and partly with different units or data types (see GigE Vision 2.0/GenICam image interface).

Status values¶

This component reports the following status values:

Table 2 The `rc_stereocamera` component’s status values¶
Name	Description
`out1_reduction`	Fraction of reduction (0.0 - 1.0) of brightness for images with GPIO Out1=LOW in exp_auto_mode=AdaptiveOut1 or exp_auto_mode=Out1High
`baseline`	Stereo baseline \(t\) in meters
`brightness`	Current brightness of the image as value between 0 and 1
`color`	0 for monochrome cameras, 1 for color cameras
`exp`	Actual exposure time in seconds. This value is shown below the image preview in the Web GUI as Exposure (ms).
`focal`	Focal length factor normalized to an image width of 1
`fps`	Actual frame rate of the camera images in Hertz. This value is shown in the Web GUI below the image preview as FPS (Hz).
`gain`	Actual gain factor in decibel. This value is shown in the Web GUI below the image preview as Gain (dB).
`height`	Height of the camera image in pixels
`temp_left`	Temperature of the left camera sensor in degrees Celsius
`temp_right`	Temperature of the right camera sensor in degrees Celsius
`test`	0 for live images and 1 for test images
`time`	Processing time for image grabbing in seconds
`width`	Width of the camera image in pixels

Services¶

The stereo camera component offers the following services for persisting and restoring parameter settings.

`save_parameters`¶

With this service call, the stereo camera component’s current parameter settings will be made persistent to the rc_cube. That means, these values are applied even after reboot.

This service has no arguments.

The definition for the response with corresponding datatypes is:
{
  "return_code": {
    "message": "string",
    "value": "int16"
  }
}

`reset_defaults`¶

Restores and applies the default values for this component’s parameters (“factory reset”).

Warning

By calling this service, the current parameter settings for the camera component are irrecoverably lost.

This service has no arguments.

The definition for the response with corresponding datatypes is:
{
  "return_code": {
    "message": "string",
    "value": "int16"
  }
}