Evaluation report, Small Underwater Objects 3D Point Cloud Dataset

We fine-tuned the same object recognition model twice, once on SUOP as released and once on an edited SUOP with the 130 recordings that return under 60% of their own setup's geometry removed, and the edited one scored 5.4 points higher over 800 held out scans.

Fine-tuned on full

33.5%

Object named correctly with all 1,554 recordings.

CI 30.3 to 36.8 · n=800

Fine-tuned on edited

38.9%

Object named correctly with the 130 starved recordings removed.

CI 35.6 to 42.3 · n=800

Lift

+5.4 pp

From training on less data.

95% range +0.7 to +10.1

Removed

8.4%

130 of 1,554 recordings, 1,300 of 15,540 sonar frames.

2.48 GB of raw sonar · 0.31 GB of point cloud

Class from file size

47.3%

Object class recovered from the byte size of point_cloud.xyz alone, against a 20.5% chance rate.

1 nearest neighbour on one scalar · 69.8% inside the 10 m set

Success rate by task

fine-tuned on full Small Underwater Objects 3D Point Cloud Dataset fine-tuned on edited Small Underwater Objects 3D Point Cloud Dataset

The eight tasks are object and range setups taken straight from the release, each held out at 100 scans. A same-size random cut scored 33.2%, so the gain is from which recordings were removed, not from removing recordings.

Gain per task

The setups that gain most are chair at 6 m and drum at 6 m, which is where the cut concentrated. The pooled lift clears zero, most single setups at 100 scans do not, and the chart says so.

What the edited version had removed

Measured on the release. Every recording is one object at one of three ranges, so the fifteen setups each hold between 100 and 110 repeats of the same shot, and a recording can be scored against its own repeats. The number is the size of its point_cloud.xyz over the median of that setup, and the median recording lands at 100%. The 130 recordings under 60% were dropped. They are not scattered, 89 of them form two unbroken opening blocks, drum at 10 m cases 001 to 044 and drum at 6 m cases 006 to 050, each returning about half the points that the later repeats of the identical setup return before the yield snaps back and stays there. Every setup keeps at least 56 recordings and every object keeps at least 218.

Removed	recordings	points vs same setup
drum at 6 m, unbroken block from case_006 to case_050	45	46.7%
drum at 10 m, unbroken block from case_001 to case_044	44	50.2%
chair at 6 m, unbroken block from case_001 to case_017	17	31.5%
net at 10 m, unbroken block from case_001 to case_010	10	49.9%
dummy and tire, scattered	14	50.4%
Removed	130	47.9%
Kept	1,424	100.5%

The removed sonar frames

Measured on the release. Each card is the first of the ten sonar frames published for that recording, and the number is the size of its point cloud against the median of the other repeats of the identical setup.

chair_6m_case_008removed

chair at 6 m, the thinnest point cloud relative to its own setup

1,532,353 byte point cloud · 28% of the 5,418,204 byte median for chair at 6 m

chair_6m_case_003removed

chair at 6 m, inside the same opening block

1,616,185 byte point cloud · 30% of the median for chair at 6 m

drum_6m_case_020removed

drum at 6 m, middle of the 45 recording block

1,544,925 byte point cloud · 46% of the 3,382,645 byte median for drum at 6 m

drum_10m_case_001removed

drum at 10 m, first recording of the 44 recording block

3,217,175 byte point cloud · 49% of the 6,501,182 byte median for drum at 10 m

net_10m_case_004removed

net at 10 m, inside the opening block, sonar stop range set to 10 m

3,484,492 byte point cloud · 50% of the 6,950,543 byte median for net at 10 m

dummy_6m_case_107removed

dummy at 6 m, the smallest point cloud in the release

612,154 byte point cloud · 34% of the 1,814,450 byte median for dummy at 6 m

drum_6m_case_060kept

drum at 6 m, the same setup after the yield snaps back

3,578,109 byte point cloud · 106% of the median for drum at 6 m

chair_6m_case_060kept

chair at 6 m, a repeat of the identical shot that was kept

5,503,328 byte point cloud · 102% of the median for chair at 6 m

SUOP · 1,554 recordings · PointNet++ · 8 setups × 100 scans × 2 arms Wilson 95% · cut = point cloud under 60% of the median for the same object and range