External Camera Calibration

Introduction

Image sensors inside PhoXi 3D Scanners and MotionCam-3D (non-color) are capable of providing grayscale images only. Applications that require color information along with the 3D data can be performed using the MotionCam-3D Color or by a calibration (also referred to as alignment) of an additional color camera with the PhoXi 3D Scanners and MotionCam-3D (non-color).

The calibration is achieved via PhoXi C++ API by capturing multiple images of marker pattern both from the device and the external camera. The output of this calculation is a transformation from the default device coordinate space to the coordinate space of the external camera. As a result, the API can provide a depth map from the perspective of the external camera as well as a 3D point cloud with a mapped color texture.

This manual will guide you through the use of the API example ExternalCamera in the following steps:

  1. Calibration of sample external camera with PhoXi 3D Scanner or MotionCam-3D.

  2. Calculation of the depth map from the point of view of the external camera.

  3. Application of color information on the 3D point cloud.

Provided example application has two modes of operation:

  1. Batch mode controlled by command line arguments capable of calibration of the device with an external camera, depth map calculation, and color point cloud processing using saved files.

  2. Interactive mode where each step is based on user input. This mode is capable of calibrating the external camera with the device, calculating the depth map, and processing the color point cloud while connected to the device.

Running the Example with Sample Data

Provided API application runs with sample data to demonstrate the process of calibration with an external camera, producing a depth map and colored point cloud. Implementation and use of your own external camera is described in later parts of this document.

The ExternalCamera API example can be found in

  • Windows: Program Files/PhotoneoPhoXiControl/API/examples/CPP/ExternalCamera

  • Linux: /opt/PhotoneoPhoXiControl/API/examples/CPP/ExternalCamera

It is recommended to copy the example folder into your user folder or other location where you have writing rights.

Download example data for calibration from the following link and copy the Data folder into your project folder:

http://photoneo.com/files/installer/PhoXi/api/external_camera_example_1.2.zip

Install OpenCV 3.1.0 on the following path:

  • Windows: C:/opencv

  • Linux: home/OpenCV

Calibration

The process of calibration serves to compute the transformation between the default coordinate space of the device and the coordinate space of the external camera which has origin in the external camera. The computation is achieved by processing:

  • Ten pairs of images of marker pattern taken by the device and the external camera, each taken from a different point of view. For calibration purposes, the images from the external camera have to be in greyscale. The images are located in <path>/Data directory with the following naming convention:

    • Prefix frame for images captured by the internal camera of the device.

    • Prefix image for images captured by the external camera.

  • Input parameters from files located in the <path>/Settings directory:

    • FocalLength → focal length of the external camera.

    • PixelSize → pixel size of the external camera.

    • MarkerPositions → positions of centers of circles on the marker pattern. Different sizes of marker patterns have different positions of the circles.

The calibration is handled by connecting to file camera 1.praw located in the <path>/Data folder.

The result of calibration is file calibration.txt containing a set of estimates of intrinsic and extrinsic parameters of the external camera and transformation from the native Photoneo 3D Sensor coordinate space to the external camera coordinate space.

To calibrate the application with sample data, use the following command line argument:

./ExternalCameraExample --calibrate ExternalCamera/Data/1.praw Data/frame1.png
Data/frame2.png Data/frame3.png Data/frame4.png Data/frame5.png Data/frame6.png
Data/frame7.png Data/frame8.png Data/frame9.png Data/frame10.png Data/image1.png
Data/image2.png Data/image3.png Data/image4.png Data/image5.png Data/image6.png
Data/image7.png Data/image8.png Data/image9.png Data/image10.png

Note

At least 5 pairs of frames & images need to be processed for a successful calibration.

Aligned Depth Map Computation

During this step, the depth map aligned from the point of view of the external camera is calculated.

The aligned depth map is calculated based on the data loaded from calibration.txt. The application connects to a file camera <folder>/Data/1.praw and then computes the aligned depth map, which is saved as an image in <folder>/fileCamera_1.jpg. The following command line argument is used to calculate the depth map:

./ExternalCameraExample --depthmap

Colored Point Cloud Computation

This step serves to apply the color texture captured by the external camera to the point cloud.

The colored point cloud is calculated based on the data loaded from calibration.txt. The application connects to a file camera <path>/Data/1.praw, loads a color texture from <path>/Data/1.bmp, and applies it to the point cloud. The result is saved as <path>/1.ply. The following command line argument is used to calculate the colored point cloud:

./ExternalCameraExample --colorpc

Integration of External Camera

To use your own external camera with Photoneo 3D Sensor, the following steps are necessary:

  • Preparation of the external camera and configuration of the input files.

  • Implementation of the external camera into the API code.

  • Preparation of marker pattern.

External Camera

For successful integration of the external camera with Photoneo 3D Sensor the following requirements have to be met:

  • The external camera is C++ compatible and the user is able to trigger images via API calls.

  • Its pixel size and focal length (both in millimeters) are known.

  • The position of the external camera with regards to the device is fixed.

  • The field of view of the external camera overlaps the field of view of the device camera as much as possible. It is recommended to mount the external camera very close to the camera unit of the device.

The following files have to be modified with the information about intrinsic parameters of the external camera:

  • <path>/Data/FocalLength.txt - contains the focal length in millimeters.

  • <path>/Data/PixelSize.txt - contains pixel size in millimeters.

Marker Pattern

The calibration of Photoneo 3D Sensor with an external camera requires a marker pattern and text file detailing the positions of circles on the pattern. A standard marker pattern bundled within PhoXi Control is used for this purpose. The folder containing marker patterns (Menu → Tools → Marker Patterns) also contains the subfolder Patterns_with_Metadata with text files where the position of each circle on the marker pattern is written. Each size of the pattern has different positions of the circles.

  • Alternatively, download the Marker Patterns with metadata

  • Please select the appropriate size of the marker pattern based on the device model and scanning distance and print it at 100% scale.

  • Copy the corresponding file with positions of the circles into <path>/Settings/MarkersPositions.txt

  • When choosing the right size of the marker pattern, refer to Table 1 or Table 2.

External Camera Implementation

In order to gather images from the external camera, it is necessary that the user implements the following functions:

ExternalCamera:getCalibrationImage

ExternalCamera:getColorImage

Calibration

Frames and images of the marker pattern from which the transformation to the coordinate space of the external camera is calculated, are acquired during the calibration. The application prompts the user to trigger the frame, after which the method ExternalCamera:getCalibrationImage is called to capture the image from the external camera. At least five pairs of frames and images from different points of view have to be processed for the calibration to be successful. When the point of view is changed, the relative position of the external camera and the native camera of Photoneo 3D Sensor has to remain the same. The output of the calibration is saved into <path>/calibration.txt. The calibration is valid for all consecutive scans.

Using Already Saved Data

This mode allows for quick calibration of the external camera with the device with previously captured files.

--calibrate [full path to praw file] frames... cameraImages...

The full command for calibration with 10 pairs of frames and images would look like this:

./ExternalCameraExample --calibrate D:/ExternalCammera_CPP/Data/1.praw Data/frame1.png Data/frame2.png Data/frame3.png Data/frame4.png Data/frame5.png Data/frame6.png Data/frame7.png Data/frame8.png Data/frame9.png Data/frame10.png Data/image1.png Data/image2.png Data/image3.png Data/image4.png Data/image5.png Data/image6.png Data/image7.png Data/image8.png Data/image9.png Data/image10.png

Aligned Depth Map Computation

The application loads calibration.txt. The aligned depth map is calculated right after a new pair of frames and images is triggered and saved into <folder>/device_1.jpg. The number in the file name is increased every time a new scan is triggered and a new depth map is saved.

Using Already Saved Data

The calculation of the aligned depth map from previously captured data can be achieved with the following command line arguments:

--depthmap [full path to praw file] [output file]

The calibration.txt has to be present for this command to work, otherwise the application will exit with an error. The output file is where the resulting depth map is saved. The full command would look like this:

./ExternalCameraExample --depthmap D:/ExternalCamera_CPP/Data/scan.praw depthmap.jpg

Colored Point Cloud Computation

The application loads calibration.txt. After the new scan is triggered the application calls ExternalCamera:getColorImage method to capture color texture by the external camera. This texture is then applied to the native point cloud and saved into <path>/device_1.ply. The number in the file name is increased each time a new point cloud is saved.

Using Already Saved Data

The alignment of existing color texture over previously captured data can be handled by following command line arguments:

--colorpc [full path to praw file] [color image] [output file]

The calibration.txt has to be present for this command to work, otherwise, the application will exit with an error. The color image is the texture from the external camera that is going to be aligned with the point cloud and the output file is where the resulting point cloud is saved. The full command looks like this:

./ExternalCameraExample --colorpc D:/ExternalCamera_CPP/Data/scan.praw texture.bmp pointcloud.ply