LoadCarrier¶
Introduction¶
The LoadCarrier module allows the detection of load carriers, which is usually the first step when objects or grasp points inside a bin should be found.
The LoadCarrier module is an optional on-board module of the rc_cube and is licensed with any of the modules ItemPick and BoxPick or CADMatch and SilhouetteMatch. Otherwise it requires a separate LoadCarrier license to be purchased.
The load carrier functionality is provided by the LoadCarrier module itself, but also by the ItemPick and BoxPick and CADMatch and SilhouetteMatch modules.
| Supported load carrier types | Standard 4-sided, with solid rim or stepped rim |
| Min. load carrier dimensions | 0.1 m x 0.1 m x 0.1 m |
| Max. load carrier dimensions | 2 m x 2 m x 2 m |
| Max. number of load carriers | 50 |
| Load carriers available in | ItemPick and BoxPick and CADMatch and SilhouetteMatch |
| Supported pose types | no pose, orientation prior, exact pose |
| Supported reference frames | camera, external |
| Support for | 2D ROI, 3D ROI (see Region of interest) |
| Detection types | detect load carrier, detect filling level |
Load carrier definition¶
A load carrier (bin) is a container with four walls, a floor and a rectangular rim, which can contain objects. It can be used to limit the volume in which to search for objects or grasp points.
A load carrier is defined by its outer_dimensions and inner_dimensions.
The maximum outer_dimensions are 2.0 meters in every dimension.
The origin of the load carrier reference frame is in the center of the load carrier’s outer box and its z axis is perpendicular to the load carrier’s floor pointing outwards (see Fig. 9).
Fig. 9 Load carrier with reference frame and inner and outer dimensions
Note
Typically, outer and inner dimensions of a load carrier are available in the specifications of the load carrier manufacturer.
The inner volume of the load carrier is defined by its inner dimensions, but includes a region
of 10 cm height above the load carrier, so that also items protruding from the load carrier
are considered for detection or grasp computation. Furthermore, an additional
crop_distance is subtracted from the inner volume in every dimension, which acts as a safety margin and can
be configured as run-time parameter (see Parameters).
Fig. 10 visualizes the inner volume of a load carrier. Only
points which are inside this volume are considered for detections.
Fig. 10 Visualization of the inner volume of a load carrier. Only points which are inside this volume are considered for detections.
Since the load carrier detection is based on the detection of the load carrier’s rim,
the rim geometry must be specified if it cannot be determined from the difference between
outer and inner dimensions. For this, the rim_thickness can be explicitly set. The rim
thickness gives the thickness of the outer part of the rim in in x and y direction.
When a rim thickness is given, an optional rim_step_height can also be
specified, which gives the height of the step between the outer and the inner part of the rim.
Examples of load carriers which require the rim thickness to be set are given in Fig. 11.
Fig. 11 Examples of load carriers which require a rim thickness to be specified
Warning
The rim step height is not considered for collision checking (see CollisionCheck), but always assumed to be 0.
A load carrier can be specified with a full 3D pose consisting of a position and
an orientation quaternion, given in a pose_frame. Based on the given pose_type
this pose is either used as an orientation prior (pose_type is ORIENTATION_PRIOR or empty),
or as the exact pose of the load carrier (pose_type is EXACT_POSE).
In case the pose serves as orientation prior, the detected load carrier pose is guaranteed to have the minimum rotation with respect to the load carrier’s prior pose. This pose type is useful for detecting tilted load carriers and for resolving the orientation ambiguity in the x and y direction caused by the symmetry of the load carrier model.
In case the pose type is set to EXACT_POSE, no load carrier detection will be performed on the scene data,
but the given pose will be used in exactly the same way as if the load carrier is detected at
that pose. This pose type is especially useful in cases where load carriers do not change their positions
and/or are hard to detect (e.g. because their rim is too thin or the material is too shiny).
The rc_cube can persistently store up to 50 different
load carrier models, each one identified by a different id.
The configuration of a load carrier model is normally performed offline,
during the set up the desired application.
This can be done via the REST-API interface or in the rc_cube Web GUI.
Note
The configured load carrier models are persistent even over firmware updates and rollbacks.
Load carrier compartments¶
Some detection modules can make use of a load_carrier_compartment to further limit the volume for the detection,
for example ItemPick’s compute_grasps service.
A load carrier compartment is a box whose pose is defined as the transformation from the load carrier reference frame to the compartment
reference frame, which is located in the center of the compartment box (see Fig. 12).
Fig. 12 Sample compartment inside a load carrier. The coordinate frame shown in the image is the reference frame of the compartment.
The compartment volume is intersected with the load carrier inner volume to compute the volume for the detection. If this intersection should also contain the 10 cm region above the load carrier, the height of the compartment box must be increased accordingly.
Detection of load carriers¶
The load carrier detection algorithm is based on the detection of the load carrier’s
rectangular rim. By default, the algorithm searches for a load carrier whose rim plane
is perpendicular to the measured gravity vector. To detect tilted load carriers, its
approximate orientation must be specified as pose and the pose_type should be
set to ORIENTATION_PRIOR.
When a 3D region of interest (see Region of interest) is used to limit the volume in which a load carrier should be detected, only the load carrier’s rim must be fully included in the region of interest.
The detection algorithm returns the pose of the load carrier’s origin (see Load carrier definition) in the desired pose frame.
The detection functionality also determines if the detected load carrier is overfilled,
which means, that objects protrude from the plane defined by the load carrier’s outer part of the rim.
Fig. 13 Load carrier models and reference frames
Detection of filling level¶
The LoadCarrier module offers the detect_filling_level
service to compute the filling level of a detected load carrier.
The load carrier is subdivided in a configurable number of cells in a 2D grid. The maximum number of cells is 10x10. For each cell, the following values are reported:
level_in_percent: minimum, maximum and mean cell filling level in percent from the load carrier floor. These values can be larger than 100% if the cell is overfilled.level_free_in_meters: minimum, maximum and mean cell free level in meters from the load carrier rim. These values can be negative if the cell is overfilled.cell_size: dimensions of the 2D cell in meters.cell_position: position of the cell center in meters (either incameraorexternalframe, see Hand-eye calibration). The z-coordinate is on the level of the load carrier rim.coverage: represents the proportion of valid pixels in this cell. It varies between 0 and 1 with steps of 0.1. A low coverage indicates that the cell contains several missing data (i.e. only a few points were actually measured in this cell).
These values are also calculated for the whole load carrier itself. If no cell subdivision is specified, only the overall filling level is computed.
Fig. 14 Visualizations of the load carrier filling level: overall filling level without grid (left), 4x3 grid (center), 8x8 grid (right). The load carrier content is shown in a green gradient from white (on the load carrier floor) to dark green. The overfilled regions are visualized in red. Numbers indicate cell IDs.
Interaction with other modules¶
Internally, the LoadCarrier module depends on, and interacts with other on-board modules as listed below.
Note
All changes and configuration updates to these modules will affect the performance of the LoadCarrier module.
Stereo camera and Stereo matching¶
The LoadCarrier module makes internally use of the following data:
- Rectified images from the Stereo camera module
(
rc_stereocamera); - Disparity, error, and confidence images from the Stereo matching module
(
rc_stereomatching).
All processed images are guaranteed to be captured after the module trigger time.
Estimation of gravity vector¶
For each load carrier detection, the module estimates the gravity vector by subscribing to the rc_visard’s IMU data stream.
Note
The gravity vector is estimated from linear acceleration readings from the on-board IMU. For this reason, the LoadCarrier module requires the rc_visard to remain still while the gravity vector is being estimated.
IO and Projector Control¶
In case the rc_cube is used in conjunction with an external random dot
projector and the IO and Projector Control module (rc_iocontrol), it is recommended to connect
the projector to GPIO Out 1 and set the stereo-camera module’s
acquisition mode to SingleFrameOut1 (see Stereo matching
parameters, so that on each
image acquisition trigger an image with and without projector pattern is
acquired.
Alternatively, the output mode for the GPIO output in use should be set
to ExposureAlternateActive (see Description of run-time parameters).
In either case,
the Auto Exposure Mode exp_auto_mode should be set to AdaptiveOut1 to optimize the exposure
of both images (see Stereo camera parameters.
No additional changes are required to use the LoadCarrier module in combination with a random dot projector.
Hand-eye calibration¶
In case the camera has been calibrated to a robot, the loadcarrier module
can automatically provide poses in the robot coordinate frame.
For the loadcarrier nodes’ Services, the frame of the
output poses can be controlled with the pose_frame argument.
Two different pose_frame values can be chosen:
- Camera frame (
camera). All poses provided by the modules are in the camera frame, and no prior knowledge about the pose of the camera in the environment is required. This means that the configured load carriers move with the camera. It is the user’s responsibility to update the configured poses if the camera frame moves (e.g. with a robot-mounted camera). - External frame (
external). All poses provided by the modules are in the external frame, configured by the user during the hand-eye calibration process. The module relies on the on-board Hand-eye calibration module to retrieve the sensor mounting (static or robot mounted) and the hand-eye transformation. If the mounting is static, no further information is needed. If the sensor is robot-mounted, therobot_poseis required to transform poses to and from theexternalframe.
Note
If no hand-eye calibration is available, all pose_frame values should be set to camera.
All pose_frame values that are not camera or external are rejected.
Parameters¶
The LoadCarrier module is called rc_load_carrier
in the REST-API and is used internally by several other modules.
The user can explore and configure the LoadCarrier
module’s run-time parameter, e.g. for development and testing, using the
LoadCarrier page under the Modules tab in the
Web GUI,
or the REST-API interface.
Parameter overview¶
This module offers the following run-time parameters:
| Name | Type | Min | Max | Default | Description |
|---|---|---|---|---|---|
crop_distance |
float64 | 0.0 | 0.05 | 0.005 | Safety margin in meters by which the load carrier inner dimensions are reduced to define the region of interest for detection |
model_tolerance |
float64 | 0.003 | 0.025 | 0.008 | Indicates how much the estimated load carrier dimensions are allowed to differ from the load carrier model dimensions in meters |
Description of run-time parameters¶
Each run-time parameter is represented by a row on the LoadCarrier Settings section
of the Web GUI’s LoadCarrier page.
The name in the Web GUI is given in brackets behind the parameter name and the parameters are
listed in the order they appear in the Web GUI. The parameters are prefixed with load_carrier_
when they are used outside the rc_load_carrier module from another detection module
using the REST-API interface.
model_tolerance (Model Tolerance)¶
indicates how much the estimated load carrier dimensions are allowed to differ from the load carrier model dimensions in meters.
Via the REST-API, this parameter can be set as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/parameters?model_tolerance=<value>
crop_distance (Crop Distance)¶
sets the safety margin in meters by which the load carrier’s inner dimensions are reduced to define the region of interest for detection (ref. Fig. 10).
Via the REST-API, this parameter can be set as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/parameters?crop_distance=<value>
Services¶
The user can explore and call the LoadCarrier module’s services, e.g. for development and testing, using the REST-API interface or the rc_cube Web GUI on the LoadCarrier page under the Modules tab.
Each service response contains a return_code,
which consists of a value plus an optional message.
A successful service returns with a return_code value of 0.
Negative return_code values indicate that the service failed.
Positive return_code values indicate that the service succeeded with additional information.
The smaller value is selected in case a service has multiple return_code values,
but all messages are appended in the return_code message.
The following table contains a list of common codes:
| Code | Description |
|---|---|
| 0 | Success |
| -1 | An invalid argument was provided |
| -4 | Data acquisition took longer than the maximum allowed time of 5.0 seconds |
| -10 | New element could not be added as the maximum storage capacity of load carriers has been exceeded |
| -11 | Sensor not connected, not supported or not ready |
| -302 | More than one load carrier provided to the detect_load_carriers or detect_filling_level services, but only one is supported |
| 10 | The maximum storage capacity of load carriers has been reached |
| 11 | An existent persistent model was overwritten by the call to set_load_carrier |
| 100 | The requested load carriers were not detected in the scene |
| 102 | The detected load carrier is empty |
| 300 | A valid robot_pose was provided as argument but it is not required |
All software modules providing the load carrier functionality offer the following services.
set_load_carrier¶
Persistently stores a load carrier on the rc_cube. All configured load carriers are persistent over firmware updates and rollbacks.
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/set_load_carrierDetails for the definition of the
load_carriertype are given in Detection of load carriers.The field
typeis optional and currently only acceptsSTANDARD.The field
rim_ledgeis optional and currently only accepts 0.The definition for the request arguments with corresponding datatypes is:
{ "args": { "load_carrier": { "id": "string", "inner_dimensions": { "x": "float64", "y": "float64", "z": "float64" }, "outer_dimensions": { "x": "float64", "y": "float64", "z": "float64" }, "pose": { "orientation": { "w": "float64", "x": "float64", "y": "float64", "z": "float64" }, "position": { "x": "float64", "y": "float64", "z": "float64" } }, "pose_frame": "string", "pose_type": "string", "rim_ledge": { "x": "float64", "y": "float64" }, "rim_step_height": "float64", "rim_thickness": { "x": "float64", "y": "float64" }, "type": "string" } } }The definition for the response with corresponding datatypes is:
{ "name": "set_load_carrier", "response": { "return_code": { "message": "string", "value": "int16" } } }
get_load_carriers¶
Returns the configured load carriers with the requested
load_carrier_ids. If noload_carrier_idsare provided, all configured load carriers are returned.Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/get_load_carriersThe definition for the request arguments with corresponding datatypes is:
{ "args": { "load_carrier_ids": [ "string" ] } }The field
typewill always beSTANDARDandrim_ledgewill always be 0.The definition for the response with corresponding datatypes is:
{ "name": "get_load_carriers", "response": { "load_carriers": [ { "id": "string", "inner_dimensions": { "x": "float64", "y": "float64", "z": "float64" }, "outer_dimensions": { "x": "float64", "y": "float64", "z": "float64" }, "pose": { "orientation": { "w": "float64", "x": "float64", "y": "float64", "z": "float64" }, "position": { "x": "float64", "y": "float64", "z": "float64" } }, "pose_frame": "string", "pose_type": "string", "rim_ledge": { "x": "float64", "y": "float64" }, "rim_step_height": "float64", "rim_thickness": { "x": "float64", "y": "float64" }, "type": "string" } ], "return_code": { "message": "string", "value": "int16" } } }
delete_load_carriers¶
Deletes the configured load carriers with the requested
load_carrier_ids. All load carriers to be deleted must be explicitly stated inload_carrier_ids.Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/delete_load_carriersThe definition for the request arguments with corresponding datatypes is:
{ "args": { "load_carrier_ids": [ "string" ] } }The definition for the response with corresponding datatypes is:
{ "name": "delete_load_carriers", "response": { "return_code": { "message": "string", "value": "int16" } } }
detect_load_carriers¶
Triggers a load carrier detection as described in Detection of load carriers.
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/detect_load_carriersRequired arguments:
pose_frame: see Hand-eye calibration.
load_carrier_ids: IDs of the load carriers which should be detected.Potentially required arguments:
robot_pose: see Hand-eye calibration.Optional arguments:
region_of_interest_id: ID of the 3D region of interest where to search for the load carriers.
region_of_interest_2d_id: ID of the 2D region of interest where to search for the load carriers.Warning
Only one type of region of interest can be set.
The definition for the request arguments with corresponding datatypes is:
{ "args": { "load_carrier_ids": [ "string" ], "pose_frame": "string", "region_of_interest_2d_id": "string", "region_of_interest_id": "string", "robot_pose": { "orientation": { "w": "float64", "x": "float64", "y": "float64", "z": "float64" }, "position": { "x": "float64", "y": "float64", "z": "float64" } } } }
load_carriers: list of detected load carriers.
timestamp: timestamp of the image set the detection ran on.
return_code: holds possible warnings or error codes and messages.The definition for the response with corresponding datatypes is:
{ "name": "detect_load_carriers", "response": { "load_carriers": [ { "id": "string", "inner_dimensions": { "x": "float64", "y": "float64", "z": "float64" }, "outer_dimensions": { "x": "float64", "y": "float64", "z": "float64" }, "overfilled": "bool", "pose": { "orientation": { "w": "float64", "x": "float64", "y": "float64", "z": "float64" }, "position": { "x": "float64", "y": "float64", "z": "float64" } }, "pose_frame": "string", "rim_ledge": { "x": "float64", "y": "float64" }, "rim_step_height": "float64", "rim_thickness": { "x": "float64", "y": "float64" }, "type": "string" } ], "return_code": { "message": "string", "value": "int16" }, "timestamp": { "nsec": "int32", "sec": "int32" } } }
detect_filling_level¶
Triggers a load carrier filling level detection as described in Detection of filling level.
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/detect_filling_levelRequired arguments:
pose_frame: see Hand-eye calibration.
load_carrier_ids: IDs of the load carriers which should be detected.Potentially required arguments:
robot_pose: see Hand-eye calibration.Optional arguments:
filling_level_cell_count: Number of cells in the filling level grid.
region_of_interest_id: ID of the 3D region of interest where to search for the load carriers.
region_of_interest_2d_id: ID of the 2D region of interest where to search for the load carriers.Warning
Only one type of region of interest can be set.
The definition for the request arguments with corresponding datatypes is:
{ "args": { "filling_level_cell_count": { "x": "uint32", "y": "uint32" }, "load_carrier_ids": [ "string" ], "pose_frame": "string", "region_of_interest_2d_id": "string", "region_of_interest_id": "string", "robot_pose": { "orientation": { "w": "float64", "x": "float64", "y": "float64", "z": "float64" }, "position": { "x": "float64", "y": "float64", "z": "float64" } } } }
load_carriers: list of detected load carriers and their filling levels.
timestamp: timestamp of the image set the detection ran on.
return_code: holds possible warnings or error codes and messages.The definition for the response with corresponding datatypes is:
{ "name": "detect_filling_level", "response": { "load_carriers": [ { "cells_filling_levels": [ { "cell_position": { "x": "float64", "y": "float64", "z": "float64" }, "cell_size": { "x": "float64", "y": "float64" }, "coverage": "float64", "level_free_in_meters": { "max": "float64", "mean": "float64", "min": "float64" }, "level_in_percent": { "max": "float64", "mean": "float64", "min": "float64" } } ], "filling_level_cell_count": { "x": "uint32", "y": "uint32" }, "id": "string", "inner_dimensions": { "x": "float64", "y": "float64", "z": "float64" }, "outer_dimensions": { "x": "float64", "y": "float64", "z": "float64" }, "overall_filling_level": { "cell_position": { "x": "float64", "y": "float64", "z": "float64" }, "cell_size": { "x": "float64", "y": "float64" }, "coverage": "float64", "level_free_in_meters": { "max": "float64", "mean": "float64", "min": "float64" }, "level_in_percent": { "max": "float64", "mean": "float64", "min": "float64" } }, "overfilled": "bool", "pose": { "orientation": { "w": "float64", "x": "float64", "y": "float64", "z": "float64" }, "position": { "x": "float64", "y": "float64", "z": "float64" } }, "pose_frame": "string", "rim_ledge": { "x": "float64", "y": "float64" }, "rim_step_height": "float64", "rim_thickness": { "x": "float64", "y": "float64" }, "type": "string" } ], "return_code": { "message": "string", "value": "int16" }, "timestamp": { "nsec": "int32", "sec": "int32" } } }
set_region_of_interest¶
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/set_region_of_interest
get_regions_of_interest¶
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/get_regions_of_interest
delete_regions_of_interest¶
see delete_regions_of_interest.
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/delete_regions_of_interest
set_region_of_interest_2d¶
see set_region_of_interest_2d.
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/set_region_of_interest_2d
get_regions_of_interest_2d¶
see get_regions_of_interest_2d.
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/get_regions_of_interest_2d
delete_regions_of_interest_2d¶
see delete_regions_of_interest_2d.
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/delete_regions_of_interest_2d
save_parameters¶
Saves the currently set parameters persistently. That means, these values are applied even after reboot. The parameters are also persistent over firmware updates and rollbacks.
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/reset_defaultsThis service has no arguments.The definition for the response with corresponding datatypes is:
{ "name": "save_parameters", "response": { "return_code": { "message": "string", "value": "int16" } } }
reset_defaults¶
Restores and applies the default values for this module’s parameters (“factory reset”).
Details
This service can be called as follows.
PUT http://<host>/api/v1/nodes/rc_load_carrier/services/reset_defaultsThis service has no arguments.The definition for the response with corresponding datatypes is:
{ "name": "reset_defaults", "response": { "return_code": { "message": "string", "value": "int16" } } }