ETL Migration to Databricks via LLM Transpilation
03:12 05 Feb 2026

I have several ETL jobs from DataStage in .dsx format. I use a PowerShell script to automatically run the migration for a larger number of files. And one job migrates successfully, while another similar one no longer migrates. My code:

$jobsPath = "C:\lakebridge_test\jobs"

Get-ChildItem $jobsPath -Filter *.dsx | ForEach-Object {

$inputFile  = $_.FullName
$outputPath = "/Workspace/Users/xxx/lakebridge_out/$($_.BaseName)"

$psi = New-Object System.Diagnostics.ProcessStartInfo
$psi.FileName = "C:\Users\xxx\Desktop\xxx\lakebridge\databricks_cli_0.258.0_windows_amd64\databricks.exe"
$psi.Arguments = @(
    "labs lakebridge llm-transpile",
    "--input-source `"$inputFile`"",
    "--output-ws-folder `"$outputPath`"",
    "--volume lakebridge_vol",
    "--catalog-name uc-test",
    "--schema-name lakebridge_test",
    "--source-dialect unknown_etl",
    "--accept-terms=true",
    "--profile lakebridge"
) -join " "

$psi.RedirectStandardInput  = $true
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError  = $true
$psi.UseShellExecute        = $false
$psi.CreateNoWindow         = $true

$process = [System.Diagnostics.Process]::Start($psi)

$process.StandardInput.WriteLine("0")
$process.StandardInput.Close()

$stdout = $process.StandardOutput.ReadToEnd()
$stderr = $process.StandardError.ReadToEnd()

$process.WaitForExit()

Write-Host "==== $($_.Name) ===="
Write-Host $stdout

if ($process.ExitCode -ne 0) {
    Write-Error "FAIL: $($_.Name)"
    Write-Error $stderr
}
}

It automatically selects: Select a Foundation Model serving endpoint: [0] [Recommended] databricks-claude-sonnet-4-5 and starts the migration process to Databricks.

A job consisting of a dataset and a Db2 connector migrates correctly, but a job with dataset → Transformer Stage → Db2 connector fails and returns: Exception: No records found for conversion. Please check if there are any records with is_conversion_target = true in the result table.

Has anyone migrated ETL this way to Databricks and can share how to figure it out? I have the impression that for very simple ETL jobs this tool somehow manages, but for more “complex” ones not necessarily.

databricks large-language-model claude