📊 Evaluations
🖼️ I. Qualitative Results (Generated Scene Graphs)
In vS-Graphs, the Building Components Recognition module extracts semantic entities such as walls and ground surfaces from each KeyFrame using a panoptic segmentation backbone that provides pixel-level labels and instance boundaries. A parallel Structural Elements Recognition thread then infers higher-level entities (rooms and floors) by grouping spatially consistent components into enclosed areas and aggregating them across building levels. This hierarchical reasoning enhances spatial understanding and enables geometry-aware optimization in vS-Graphs. Below, you can find some qualitative results of the generated scene graphs on different datasets, where bc and se refer to the building components and structural elements, respectively.
🏷️ AutoSense Dataset
🏷️ ICL Dataset
🏷️ OpenLoris Dataset
🏷️ TUM-RGBD Dataset
📈 II. Absolute Trajectory Error (ATE)
Below table shows the performance of vS-Graphs on different datasets and compared to the state-of-the-art methods. It is measured using ATE reported in meters. For evaluation, each system was evaluated over eight runs on dataset instances.
📍 III. Mapping Performance
Analyzing the accuracy of the reconstructed maps against the ground truch across eight iterations shows that vS-Graphs performs more robost compared to its baseline, ORB-SLAM 3.0. The performance is measured using Root Mean Square Error (RMSE) reported in meters. vS-Graphs achieves superior performance in terms of RMSE, despite generating maps with around 10.15% fewer points (on average).
🖼️ IV. Scene Understanding Performance
Below evaluations show the performance of vS-Graphs in terms of scene understanding. The results are presented in the form of reconstructed maps enriched with building components (i.e., walls and ground surfaces). These building components are later used to infer the structural elements of the environment (i.e., rooms and floors).
| Detected / Real | Precision | Recall | |||||
|---|---|---|---|---|---|---|---|
| Method | Sequence | BC | SE | BC | SE | BC | SE |
| S-Graphs (link) | MR01 | 11 / 14 | 4 / 4 | 0.92 | 1.00 | 0.92 | 1.00 |
| MR02 | 12 / 13 | 4 / 4 | 1.00 | 1.00 | 0.92 | 1.00 | |
| MR03 | 20 / 22 | 6 / 6 | 0.90 | 1.00 | 0.95 | 1.00 | |
| Hydra (link) | MR01 | N/A | 4 / 4 | N/A | 1.00 | N/A | 1.00 |
| MR02 | N/A | 6 / 4 | N/A | 0.75 | N/A | 0.75 | |
| MR03 | N/A | 6 / 6 | N/A | 1.00 | N/A | 0.80 | |
| vGraphs (link) | MR01 | 13 / 14 | 4 / 4 | 0.86 | 1.00 | 1.00 | 1.00 |
| MR02 | 13 / 13 | 4 / 4 | 0.92 | 1.00 | 0.92 | 1.00 | |
| MR03 | 23 / 22 | 6 / 6 | 0.96 | 1.00 | 1.00 | 1.00 | |
⏱️ V. Runtime Analysis
vS-Graphs achieves real-time performance with an average processing rate of 22 ± 3 FPS, exceeding the 20 FPS threshold for real-time operation. Below, you can see the timeline of thread execution while processing a sample dataset instance.
