I am fine-tuning DPVO (https://github.com/princeton-vl/DPVO?tab=readme-ov-file, a successor of DROID-SLAM) to estimate 6DoF camera poses from bronchoscopy videos.

• I am using DPVO to extract camera 6DoF trajectories from bronchoscopy videos.

• DPVO is a visual odometry / SLAM model built on top of deep correspondence and bundle adjustment (successor of DROID-SLAM).

• I am fine-tuning DPVO on a custom dataset.

2. Dataset and Ground Truth

• My dataset consists of bronchoscopy simulation videos rendered in Unity Hub.

• For each frame, I have ground-truth 6DoF camera poses (x, y, z, qx, qy, qz, qw) exported directly from Unity.

• These ground-truth poses are therefore expressed in Unity’s coordinate system (left-handed).

3. Problem: Coordinate System Mismatch

• DPVO internally uses a different coordinate system (likely right-handed, but not clearly documented).

• Unity and DPVO clearly do not share the same axis conventions or handedness.

• However, DPVO documentation does not explicitly specify the exact coordinate system conventions (axis directions, handedness, camera-to-world vs world-to-camera).

Therefore, before using Unity poses as supervision, I need to align Unity coordinates to DPVO coordinates.

4. My Attempted Alignment Strategy

Since the exact DPVO coordinate convention is undocumented, I attempted a data-driven alignment approach:

Step 1: Warmup removal

• DPVO has an initial warmup period where poses are unreliable.

• I discard the first N frames and only use stable frames for alignment.

Step 2: Enumerate all axis mappings

• I generate all signed permutation matrices (axis permutation + sign flips):

  • 3! permutations × 2³ sign combinations = 48 candidates

• Each candidate matrix C represents a possible mapping from Unity axes to DPVO axes.

Step 3: Per-sequence alignment

For each video sequence:

• Apply candidate axis mapping:

  Punity′=Punity⋅C⊤P_{unity}' = P_{unity} \cdot C^\topPunity′​=Punity​⋅C⊤

• Estimate a Sim(3) transform (scale s, rotation R, translation t) using Umeyama alignment:

  Pdpvo≈s⋅(Punity′⋅R⊤)+tP_{dpvo} \approx s \cdot (P_{unity}' \cdot R^\top) + tPdpvo​≈s⋅(Punity′​⋅R⊤)+t

• Compute RMSE between aligned Unity positions and DPVO estimated positions.

• Select the axis mapping C_k with minimum RMSE.

Step 4: Axis voting

• Repeat Step 3 for multiple sequences.

• Perform majority voting over the selected axis mappings {C₁, C₂, C₃, …}.

• Select the most frequent axis mapping as the final mapping C*.

Step 5: Global refinement

• Using C*, collect all matched positions from all sequences.

• Re-estimate a single global Sim(3) transform (s*, R*, t*) using Umeyama.

• Final position mapping:

  pdpvo=s∗⋅((punity⋅C∗⊤)⋅R∗⊤)+t∗p_{dpvo} = s^* \cdot ((p_{unity} \cdot C^{*\top}) \cdot R^{*\top}) + t^*pdpvo​=s∗⋅((punity​⋅C∗⊤)⋅R∗⊤)+t∗

This approach yields low positional RMSE and visually aligned trajectories.

Despite good positional alignment, I am observing issues during training and inference (e.g., unstable rotation behavior, spiral-like trajectories).

This raises several questions:

1. Is this alignment strategy conceptually correct for aligning Unity poses to DPVO/DROID-SLAM poses?

   • Especially when DPVO pose conventions are undocumented.

2. Does anyone know the exact coordinate system conventions used by DPVO or DROID-SLAM?

   • Handedness (left/right)

   • Axis directions

   • Camera-to-world vs world-to-camera pose definition

3. Is position-only alignment sufficient, or should rotation (quaternion) ground truth also be explicitly transformed using the same basis change?

   • Currently, only translation is transformed; Unity quaternions are left unchanged.

4. Are there any best practices when using Unity-generated camera poses as supervision for SLAM/VO models like DPVO?

Any insights from people familiar with DPVO, DROID-SLAM, or Unity coordinate systems would be greatly appreciated.