I have several ETL jobs from DataStage in .dsx format. I use a PowerShell script to automatically run the migration for a larger number of files. And one job migrates successfully, while another similar one no longer migrates. My code:
$jobsPath = "C:\lakebridge_test\jobs"
Get-ChildItem $jobsPath -Filter *.dsx | ForEach-Object {
$inputFile = $_.FullName
$outputPath = "/Workspace/Users/xxx/lakebridge_out/$($_.BaseName)"
$psi = New-Object System.Diagnostics.ProcessStartInfo
$psi.FileName = "C:\Users\xxx\Desktop\xxx\lakebridge\databricks_cli_0.258.0_windows_amd64\databricks.exe"
$psi.Arguments = @(
"labs lakebridge llm-transpile",
"--input-source `"$inputFile`"",
"--output-ws-folder `"$outputPath`"",
"--volume lakebridge_vol",
"--catalog-name uc-test",
"--schema-name lakebridge_test",
"--source-dialect unknown_etl",
"--accept-terms=true",
"--profile lakebridge"
) -join " "
$psi.RedirectStandardInput = $true
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.UseShellExecute = $false
$psi.CreateNoWindow = $true
$process = [System.Diagnostics.Process]::Start($psi)
$process.StandardInput.WriteLine("0")
$process.StandardInput.Close()
$stdout = $process.StandardOutput.ReadToEnd()
$stderr = $process.StandardError.ReadToEnd()
$process.WaitForExit()
Write-Host "==== $($_.Name) ===="
Write-Host $stdout
if ($process.ExitCode -ne 0) {
Write-Error "FAIL: $($_.Name)"
Write-Error $stderr
}
}
It automatically selects: Select a Foundation Model serving endpoint:
[0] [Recommended] databricks-claude-sonnet-4-5 and starts the migration process to Databricks.
A job consisting of a dataset and a Db2 connector migrates correctly, but a job with dataset → Transformer Stage → Db2 connector fails and returns:
Exception: No records found for conversion. Please check if there are any records with is_conversion_target = true in the result table.
Has anyone migrated ETL this way to Databricks and can share how to figure it out? I have the impression that for very simple ETL jobs this tool somehow manages, but for more “complex” ones not necessarily.