Skip to content

Latest commit

 

History

History
161 lines (140 loc) · 16.2 KB

benchmark.md

File metadata and controls

161 lines (140 loc) · 16.2 KB

Benchmark

Replica

Here are the detailed comparison results on Replica.

ATE

Note: DPVO run with monocular data and requires scale correction during trajectory alignment.

Algorithm\ATE(RMSE[cm]) Room0 Room1 Room2 Office0 Office1 Office2 Office3 Office4 Average
NICE-SLAM 1.69 2.04 1.55 0.99 0.90 1.39 3.97 3.08 1.95
NICE-SLAM* 2.38 1.78 2.38 1.82 0.71 1.97 3.45 2.29 2.09
Co-SLAM 0.65 1.13 1.43 0.55 0.50 0.46 1.40 0.77 0.86
Co-SLAM* 0.85 1.12 1.42 0.64 0.77 1.90 1.34 0.88 1.11
Vox-Fusion 0.40 0.54 0.54 0.50 0.46 0.75 0.50 0.60 0.54
Vox-Fusion* 0.60 0.58 0.40 0.41 0.36 0.88 0.70 0.61 0.56
Point-SLAM 0.61 0.41 0.37 0.38 0.48 0.54 0.69 0.72 0.52
Point-SLAM* 0.50 0.44 0.37 0.32 0.50 0.55 0.65 0.49 0.47
SplaTAM 0.31 0.40 0.29 0.47 0.27 0.29 0.32 0.55 0.36
SplaTAM* 0.39 0.27 0.27 0.47 0.30 0.39 0.48 0.63 0.40
DPVO* 0.28 0.31 0.19 0.37 0.15 0.32 0.25 0.58 0.31

2D metrics

A single Replica dataset has a total of 2000 frames, rendering is performed every 50 frames, a total of 40 images are rendered. Calculate the 2D metrics of these rendered images and get the average results.

Algorithm Metric Room0 Room1 Room2 Office0 Office1 Office2 Office3 Office4 Average
NICE-SLAM PSNR+ 22.12 22.47 24.52 29.07 30.34 19.66 22.23 24.94 24.42
SSIM+ 0.69 0.76 0.81 0.87 0.89 0.80 0.80 0.86 0.81
LPIPS- 0.33 0.27 0.21 0.23 0.18 0.23 0.21 0.20 0.23
NICE-SLAM* PSNR+ 22.87 25.14 24.58 29.12 30.27 23.68 23.77 26.04 25.68
SSIM+ 0.79 0.84 0.82 0.88 0.91 0.86 0.86 0.88 0.85
LPIPS- 0.44 0.36 0.33 0.32 0.27 0.30 0.26 0.30 0.32
Co-SLAM PSNR+ 27.27 28.45 29.06 34.14 34.87 28.43 28.76 30.91 30.24
SSIM+ 0.91 0.90 0.93 0.96 0.96 0.93 0.94 0.95 0.93
LPIPS- 0.32 0.29 0.26 0.20 0.19 0.25 0.22 0.23 0.25
Co-SLAM* PSNR+ 27.23 28.80 29.18 34.11 34.94 28.48 28.90 31.09 30.34
SSIM+ 0.91 0.91 0.93 0.96 0.96 0.93 0.94 0.95 0.93
LPIPS- 0.32 0.28 0.26 0.21 0.19 0.25 0.22 0.23 0.24
Vox-Fusion PSNR+ 22.39 22.36 23.92 27.79 29.83 20.33 23.47 25.21 24.41
SSIM+ 0.68 0.75 0.80 0.86 0.88 0.79 0.80 0.85 0.80
LPIPS- 0.30 0.27 0.23 0.24 0.18 0.24 0.21 0.20 0.24
Vox-Fusion* PSNR+ 25.44 27.05 27.45 31.83 31.70 25.76 27.12 27.28 27.95
SSIM+ 0.87 0.88 0.90 0.93 0.93 0.90 0.92 0.91 0.90
LPIPS- 0.33 0.30 0.25 0.24 0.24 0.25 0.20 0.25 0.25
Point-SLAM PSNR+ 32.40 34.08 35.50 38.26 39.16 33.99 33.48 33.49 35.17
SSIM+ 0.97 0.98 0.98 0.98 0.99 0.96 0.96 0.98 0.97
LPIPS- 0.11 0.12 0.11 0.10 0.12 0.16 0.13 0.14 0.12
Point-SLAM* PSNR+ 32.40 32.64 34.20 37.52 38.38 32.59 32.52 32.56 34.10
SSIM+ 0.97 0.96 0.97 0.98 0.98 0.97 0.97 0.97 0.97
LPIPS- 0.10 0.11 0.11 0.08 0.10 0.13 0.11 0.13 0.10
SplaTAM PSNR+ 32.86 33.89 35.25 38.26 39.17 31.97 29.70 31.81 34.11
SSIM+ 0.98 0.97 0.98 0.98 0.98 0.97 0.95 0.95 0.97
LPIPS- 0.07 0.10 0.08 0.09 0.09 0.10 0.12 0.15 0.10
SplaTAM* PSNR+ 33.03 33.79 35.47 38.49 39.42 32.37 30.55 32.4 34.44
SSIM+ 0.97 0.96 0.98 0.98 0.98 0.96 0.95 0.94 0.96
LPIPS- 0.07 0.10 0.07 0.08 0.10 0.10 0.11 0.16 0.09

3D metrics

Note: When evaluating 3D metrics, the default value of distance_thresh is 0.01. However, for NeuralRecon, the default value of distance_thresh is 0.05. Therefore, 3D metrics are provided under these two threshold values respectively.

Algorithm Metric Room0 Room1 Room2 Office0 Office1 Office2 Office3 Office4 Average
NICE-SLAM Depth L1[cm] - 1.81 1.44 2.04 1.39 1.76 8.33 4.99 2.01 2.97
Precision [%] + 45.86 43.76 44.38 51.40 50.80 38.37 40.85 37.35 44.10
Recall [%] + 44.10 46.12 42.78 48.66 53.08 39.98 39.04 35.77 43.69
F1[%] + 44.96 44.84 43.56 49.99 51.91 39.16 39.92 36.54 43.86
Depth L1[cm] - 2.11 1.68 2.90 1.83 2.46 8.92 5.93 2.38 3.53
Acc. [cm]- 2.73 2.58 2.65 2.26 2.50 3.82 3.50 2.77 2.85
Comp. [cm]- 2.87 2.47 3.00 2.02 2.36 3.57 3.83 3.84 3.00
Comp. Ratio[<5cm %] + 90.93 92.8 89.07 94.93 92.61 85.2 82.98 86.14 89.33
NICE-SLAM* Depth L1[cm] - 1.93 1.58 2.64 2.31 2.15 4.54 3.45 2.39 2.62
Precision [%] + 50.28 50.62 47.47 42.7 61.1 42.7 41.96 36.2 46.62
Recall [%] + 40.53 41.14 37.96 34.2 46.8 35.0 34.51 30.1 37.53
F1[%] + 44.88 45.39 42.19 38.0 53.0 38.5 37.87 32.8 41.57
Acc. [cm]- 2.10 1.76 2.14 1.87 1.47 2.25 2.33 2.37 2.03
Comp. [cm]- 3.70 3.11 3.31 2.25 2.48 4.19 3.87 4.13 3.38
Comp. Ratio[<5cm %] + 88.47 89.99 88.39 92.7 90.61 83.3 83.9 85.17 87.81
Co-SLAM Depth L1[cm] - 1.05 0.85 2.37 1.24 1.48 1.86 1.66 1.54 1.51
Acc. [cm]- 2.11 1.68 1.99 1.57 1.31 2.84 3.06 2.23 2.10
Comp. [cm]- 2.02 1.81 1.96 1.56 1.59 2.43 2.72 2.52 2.08
Comp. Ratio[<5cm %] + 95.26 95.19 93.58 96.09 94.65 91.63 90.72 90.44 93.44
Co-SLAM* Depth L1[cm] - 1.01 0.63 2.45 1.26 1.36 2.48 2.27 1.65 1.63
Precision [%] + 88.52 88.80 81.88 84.63 91.06 60.25 63.43 86.74 80.66
Recall [%] + 74.43 74.71 68.95 73.68 76.38 55.23 56.87 70.09 68.79
F1[%] + 80.86 81.15 74.86 78.78 83.08 57.63 59.97 77.53 74.23
Acc. [cm]- 1.61 1.30 1.55 1.33 1.03 1.75 1.97 1.76 1.53
Comp. [cm]- 3.32 2.83 2.59 1.65 2.08 3.63 3.46 3.70 2.90
Comp. Ratio[<5cm %] + 89.95 90.82 90.33 94.75 92.05 87.00 87.22 86.39 89.81
Vox-Fusion Depth L1[cm] - 1.09 1.90 2.21 2.32 3.40 4.19 2.96 1.61 2.46
Precision [%] + 75.83 35.88 63.10 48.51 43.50 54.48 69.11 55.40 55.73
Recall [%] + 64.89 33.07 56.62 44.76 38.44 47.85 60.61 46.79 49.13
F1[%] + 69.93 34.38 59.67 46.54 40.81 50.95 64.56 50.72 52.20
Acc. [cm]- 2.41 1.62 3.11 1.74 1.69 2.23 2.84 3.31 2.37
Comp. [cm]- 2.60 2.23 1.93 1.39 1.80 2.71 2.69 2.88 2.28
Comp. Ratio[<5cm %] + 92.87 93.48 94.34 97.21 93.76 90.98 90.73 89.48 92.86
Vox-Fusion* Depth L1[cm] - 0.96 0.46 0.83 0.63 1.18 1.79 1.40 1.05 1.03
Precision [%] + 95.13 93.82 92.30 91.10 89.15 87.23 88.76 78.72 89.52
Recall [%] + 74.74 76.04 74.90 74.69 72.29 67.41 69.59 61.06 71.34
F1[%] + 83.71 84.00 82.70 82.08 79.84 76.05 78.02 68.77 79.39
Acc. [cm]- 1.51 1.22 1.30 1.16 1.01 1.47 1.68 1.81 1.39
Comp. [cm]- 3.24 2.81 2.39 1.57 2.03 3.43 3.29 3.85 2.82
Comp. Ratio[<5cm %] + 89.96 90.77 91.68 95.08 92.32 87.68 87.51 86.07 90.13
Point-SLAM Depth L1[cm] - 0.53 0.22 0.46 0.30 0.57 0.49 0.51 0.46 0.44
Precision [%] + 91.95 99.04 97.89 99.00 99.37 98.05 96.61 93.98 96.99
Recall [%] + 82.48 86.43 84.64 89.06 84.99 81.44 81.17 78.51 83.59
F1[%] + 86.90 92.31 90.78 93.77 91.62 88.98 88.22 85.55 89.77
Point-SLAM* Depth L1[cm] - 0.27 0.22 0.55 0.26 0.47 0.53 0.49 0.27 0.38
Precision [%] + 99.63 99.54 99.24 99.53 99.62 99.14 98.83 98.91 99.30
Recall [%] + 84.84 86.05 84.37 88.65 83.51 81.18 81.05 80.62 83.78
F1[%] + 91.65 92.30 91.20 93.77 90.85 89.26 89.06 88.83 90.86
Acc. [cm]- 1.45 1.14 1.20 1.04 0.85 1.32 1.56 1.48 1.25
Comp. [cm]- 3.62 3.07 3.01 1.69 2.37 3.66 3.53 4.02 3.12
Comp. Ratio[<5cm %] + 87.60 89.13 89.43 93.00 89.42 85.85 85.56 85.23 88.15
NeuralRecon* Precision [%] + 13.06 10.91 13.53 13.64 16.73 17.85 10.98 9.67 13.29
Recall [%] + 6.48 6.63 8.44 8.86 9.09 9.71 5.29 5.01 7.43
F1[%] + 8.66 8.25 10.40 10.74 11.78 12.57 7.13 6.60 9.51
Acc. [cm]- 5.47 6.24 6.31 5.25 3.60 6.06 7.05 7.00 5.87
Comp. [cm]- 24.02 13.70 20.03 10.69 11.48 25.13 28.52 21.38 19.36
Comp. Ratio[<5cm %] + 34.26 39.52 40.36 51.55 45.00 35.44 30.63 28.29 38.13
NeuralRecon* Precision [%] + 66.12 57.03 59.09 69.01 73.34 63.89 60.87 50.10 62.43
distance_thresh=0.05 Recall [%] + 35.06 40.10 40.98 51.96 45.14 35.88 31.72 29.15 38.74
F1[%] + 45.82 47.09 48.40 59.29 55.88 45.95 41.71 36.86 47.62
Acc. [cm]- 5.47 6.23 6.30 5.19 3.60 6.07 7.08 7.03 5.87
Comp. [cm]- 24.11 13.76 19.88 10.71 11.56 25.21 28.43 21.45 19.38
Comp. Ratio[<5cm %] + 34.28 39.42 40.52 51.48 44.86 35.39 30.76 28.29 38.12

Euroc

ATE

Note: DPVO run with monocular data and requires scale correction during trajectory alignment.

Algorithm\ATE(RMSE[cm]) MH01 MH02 MH03 MH04 MH05 V101 V102 V103 V201 V202 V203 Average
DPVO 8.7 5.5 15.8 13.7 11.4 5.0 14.0 8.6 5.7 4.9 21.1 10.5
DPVO* 10.0 7.4 11.8 15.2 8.7 9.4 15.9 10.2 6.6 6.4 12.3 10.4

7-Scenes

3D metrics

Note: The distance_thresh is 0.05 for both NeuralRecon* and NeuralRecon.

The original paper didn't offer individual dataset results. Hence, only XRDSLAM results per dataset are provided here. Also, due to the absence of default parameters for 7-Scenes in the original code, there are minor discrepancies between the results of NeuralRecon* and the original NeuralRecon.

Algorithm Precision [%] + Recall [%] + F1[%] + Acc. [cm]- Comp. [cm]-
NeuralRecon 38.9 22.7 28.2 10.0 22.8
NeuralRecon* 37.51 21.24 26.55 14.23 32.21
Method Metric chess-seq-03 chess-seq-05 fire-seq-03 fire-seq-04 heads-seq-01 office-seq-02 office-seq-06 office-seq-07 office-seq-09 pumpkin-seq-01 pumpkin-seq-07 redkitchen-seq-03 redkitchen-seq-04 redkitchen-seq-06 stairs-seq-01 stairs-seq-04 Average
NeuralRecon* Precision [%] + 48.37 36.21 40.08 47.90 51.00 26.64 24.83 31.73 42.93 39.97 26.39 24.93 44.30 35.07 40.95 38.87 37.51
Recall [%] + 30.98 27.37 31.88 30.45 24.57 18.24 25.93 22.37 26.35 17.91 11.78 9.34 19.80 16.49 15.32 11.15 21.24
F1[%] + 37.77 31.18 35.51 37.24 33.16 21.65 25.37 26.24 32.66 24.74 16.29 13.59 27.37 22.43 22.30 17.31 26.55
Acc. [cm]- 7.68 8.97 8.88 7.56 9.74 25.26 23.20 12.90 8.82 13.71 26.97 25.95 11.24 10.97 16.01 9.96 14.23