Lightweight Low-Poly Photogrammetry with Object Classification Validation with OpenCV and PointNet

Tim Nguyen

Massachusetts Institute of Technology • Boston Latin School

[Code] [Paper] [Research Posterboard]

Abstract

Images in their purest form can only provide information in a two-dimensional space, which lacks the information that can be found in the three-dimensional space of the real world. Information such as the volumetric properties of the object is lost, and these factors are often a method used to help classify complex objects. To leverage the versatility that images provide and transform them into a medium with more detail, photogrammetry is a popular method to do the task. With programs like COLMAP and Meshroom, the user simply puts in the images of an object, and the program attempts to produce a 3D version of the object using keypoint detection. When paired up with 3D object classification, images can be transformed into 3D objects that can be used for robust classification. While both programs produce 3D objects with color, they require specialized hardware such as a dedicated GPU to run on them. Even without the need for dedicated GPUs, these programs still take a long time to perform reconstruction, These factors alone put a limit on the practicality of photogrammetry in places where image recognition is not adequate. I created a lightweight photogrammetry package that can run on a wide range of form factors, from laptops to Raspberry Pis, and can perform up to 26 times faster than COLMAP while producing up to 21 times more vertices. When leveraging the object classification network to validate the quality of the mesh, we can get an accuracy rate of up to 27% higher than COLMAP. This highlights that a high-quality and lightweight photogrammetry package is possible in the age of edge computing.

Architecture

Image Sequence Input

Using Arduino rigs or manually taking images of an object.

Camera Calibration

Using a checkerboard pattern to calibrate and finding the camera matrix and distortion coefficients.

Key Point and Descriptor Extraction

Using SIFT, ORB, or BRISK to extract key points and descriptors from a pair of images.

Recovering Camera Pose & Triangulation

Using the camera matrix, keypoints, descriptors, and essential matrix to recover the camera pose and triangulate the 3D points.

Bundle Adjustment & Surface Recon.

Refining the camera pose and triangulated points using bundle adjustment and reconstructing the surface using the points.

PointNet Validation

Using PointNet to validate the results of the object being reconstructed by comparing the predicted and actual labels for the point clouds.

Awards

First Place Senior Division & Sanofi Specialty Care

This project was awarded First Place for the Senior Division and the Sanofi Specialty Care at the Boston/Region VI Science & Engineering Fair.

Acknowledgment

The author would like to give thanks to Jingnan Shi, Prof. Luca Carlone, Scott Balicki, Dr. Tina Kapur, and Dr. Alex Golby for their insight and mentoring. The author would also like to acknowledge Isa Gonzalez, Caleb Kohn, Ben Jacobson, Kimberly Nguyen, Sidney Trzepacz, Parker Hastings, Zoe Colimon, and Quyen Vo for their help with obtaining data for this project. This work was partially funded by the NSF CAREER award “Certifiable Perception for Autonomous Cyber-Physical Systems”.

BibTex Citation

If you found this work to be useful, please consider using the following citation.

@misc{low-poly-recon,
        title={Lightweight Low-Poly Photogrammetry with Object Classification Validation with OpenCV and PointNet},
        author={Nguyen, Tim},
        year={2023},
        url={https://thisistimnguyen.com/low-poly-recon.html}
    }