How to Insert Data into table with select query in Databricks using spark temp table
19:19 31 Oct 2020

I would like to insert the results of a Spark table into a new SQL Synapse table using SQL within Azure Data Bricks.

I have tried the following explanation [https://learn.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/sql-ref-syntax-ddl-create-table-datasource] but I'm having no luck.

The Synapse table must be created as a result of a SELECT statement. The source should be a Spark / Data Bricks temporary view or Parquet source.

e.g. Temp Table

    # Load Taxi Location Data from Azure Synapse Analytics
        
        jdbcUrl = "jdbc:sqlserver://synapsesqldbexample.database.windows.net:number;
database=SynapseDW" #Replace "suffix" with your own  
        connectionProperties = {
          "user" : "usernmae1",
          "password" : "password2",
          "driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver"
        }
        
        pushdown_query = '(select * from NYC.TaxiLocationLookup) as t'
        dfLookupLocation = spark.read.jdbc(url=jdbcUrl, table=pushdown_query, properties=connectionProperties)
        
        dfLookupLocation.createOrReplaceTempView('NYCTaxiLocation')
        
        display(dfLookupLocation)

e.g. Source Synapse DW

Server: synapsesqldbexample.database.windows.net

Database:[SynapseDW]

Schema: [NYC]

Table: [TaxiLocationLookup]

Sink / Destination Table (not yet in existence):

Server: synapsesqldbexample.database.windows.net

Database:[SynapseDW]

Schema: [NYC]

New Table: [TEST_NYCTaxiData]

SQL Statement I tried:

%sql
CREATE TABLE if not exists TEST_NYCTaxiLocation 
select *
from NYCTaxiLocation
limit 100
sql-server apache-spark create-table azure-databricks azure-synapse