Is there any way to use the native iOS mapbox navigation sdk in an expo module WITHOUT using a third party expo or react-native package?

I'm following this expo guide on how to Wrap third-party native libraries , trying to use the mapbox ios navigation sdk, which seems to be only available via SPM, not Cocoapods (here mapbox says "CocoaPods support is currently in development and will be added in future versions.").

I have set up the ~/.netrc file with my private mapbox key.
I have also created a config plugin that successfully adds all necessary values to the Info.plist files as instructed in the Project Configuration part.

This is my config plugin withMapboxToken.js:

const { withInfoPlist } = require('@expo/config-plugins');

const withMapboxToken = (config) => {
  return withInfoPlist(config, (config) => {
    // Add Mapbox access token
    config.modResults.MBXAccessToken = process.env.MAPBOX_PUBLIC_TOKEN;
    
    // Add location permissions
    config.modResults.NSLocationWhenInUseUsageDescription = 
      "Shows your location on the map and helps improve the map.";
    
    config.modResults.NSLocationAlwaysAndWhenInUseUsageDescription = 
      "Shows your location on the map and helps improve the map.";

    // Add background modes for audio and location updates
    if (!config.modResults.UIBackgroundModes) {
      config.modResults.UIBackgroundModes = [];
    }
    
    if (!config.modResults.UIBackgroundModes.includes('audio')) {
      config.modResults.UIBackgroundModes.push('audio');
    }
    
    if (!config.modResults.UIBackgroundModes.includes('location')) {
      config.modResults.UIBackgroundModes.push('location');
    }

    console.log('✅ Mapbox token and permissions configured');
    return config;
  });
};

module.exports = withMapboxToken; 

There seems to be no problem concerning the linking/bridging between native iOS and XCode, as I successfully managed to write a Simple "Hello World" view in Swift, which shows up in the expo development build.
Now, when trying to install the SDK, this seems to be the part where I fail.
I first tried all ways I could think (manual install in XCode, or using a Package.swift file) of to first MANUALLY install the SDK and have a development build running on my iPhone without any problems.

Keep in mind that I'm using expo managed workflow, so the ios/ and android/ folders are automatically generated and shouldn't be manually changed, I also have them in my .gitignore. I just wanted to try manually installing the SDK first, before trying to automate this for example via a config plugin or a script that I run after prebuild and before creating a new development build.

When trying to run npx expo run:ios --device , I get this error in the output:

  1 | import ExpoModulesCore
> 2 | import MapboxDirections
    |        ^ no such module 'MapboxDirections'
  3 | import MapboxNavigationCore
  4 | import MapboxNavigationUIKit
  5 | import UIKit

› Compiling expo-linking Pods/ExpoLinking » ExpoLinking-dummy.m

› 1 error(s), and 1 warning(s)

CommandError: Failed to build iOS project. "xcodebuild" exited with error code 65.

So it seems to me the Swift package(s) have not been correctly installed.
And this is what the "Package Dependencies" part of my XCode Project Navigator looks like: