TensorRT C++ API: Code Walkthrough and Explanation

• 0:00:02 Introduction: Cyrus, a Computer Vision Developer at Trueface, introduces the TensorRT C++ API and its purpose.
• 0:00:30 Background: The TensorRT C++ API, an open-source project, simplifies high-performance inference coding with TensorRT.
• 0:00:59 Project Structure: The project uses CMake for compilation and linking, with three main source files: header, main CPP, and entry point main.
• 0:01:34 CMake Configuration: The CMake file sets up the project, links necessary libraries, and ensures proper TensorRT integration using a custom FindTensorRT.cmake file.
• 0:01:54 Main File: The main file defines configuration options, including precision (FP32 or FP16), optimal and maximum batch sizes, maximum workspace size, and device index.
• 0:03:59 Engine Class: The Engine class constructor initializes settings and handles options for static vs. dynamic batch sizes.
• 0:06:31 Model Loading: The example uses the ArcFace ResNet100 model for face recognition, with an input size of 1x3x112x112.
• 0:07:11 Build Function: The build function serializes configuration options into the engine filename to avoid unnecessary rebuilding.
• 0:08:56 TensorRT Engine Building: This part involves creating a builder, network, and parser, parsing the ONNX model, and setting optimization profiles.
• 0:10:26 Optimization Profiles: The code registers optimization profiles for each input, specifying minimum, optimal, and maximum dimensions.
• 0:11:41 Asynchronous Operations: Cuda streams are used for asynchronous memory allocation and network building, improving performance.
• 0:12:33 Engine Serialization: The built engine is serialized and saved to disk, with the filename reflecting the configuration options.
• 0:12:39 Debugging Tips: The trtExec command-line tool is recommended for debugging engine build failures.
• 0:13:40 Load Network Function: The load_network function loads the serialized engine from disk and sets the CUDA device for memory allocation.
• 0:14:36 Memory Allocation: GPU memory is allocated for inputs and outputs using cudaMallocAsync, with sizes based on model dimensions and maximum batch size.
• 0:16:00 Input Preprocessing: Input images are resized, converted to RGB, and uploaded to GPU memory using OpenCV's cuda::GpuMat.
• 0:19:40 Normalization: Input values are normalized to the range of -1 to 1 by default, with options for customization.
• 0:20:53 Run Inference Method: The run_inference method handles input validation, asynchronous CUDA operations, and output retrieval.
• 0:28:44 Dynamic Batch Size Handling: The code supports dynamic batch sizes by updating input dimensions before each inference call.
• 0:29:53 Color Channel Conversion: Input data is converted from NHWC (used by OpenCV) to NCHW (used by TensorRT) format.
• 0:34:44 Output Copying: Inference outputs are copied from GPU to CPU memory using cudaMemcpyAsync.
• 0:37:57 Synchronization: The CUDA stream is synchronized to ensure all asynchronous operations are completed before returning results.
• 0:38:24 Conclusion: The code provides a flexible and efficient way to perform inference with TensorRT, supporting various model configurations.
• 0:38:45 Future Work: Dynamic input height and width handling is planned for future development.
• 0:38:50 Call to Action: Viewers are encouraged to star the GitHub repository if they find the tutorial helpful.