SELECT TIED_ASSET_NO FROM IBA.TTIED_ASSET WHERE UPDDATE > TO_DATE('1900-01-01 00:00:00','YYYY-MM-DD HH24:MI:SS')
The above gives me 232 records from the oracle source.
below is the control table entry we use it to stage table load
parameter_id src_schema src_table tgt_schema tgt_table stg_table watermark_column watermark_column_value table_type created_date active_ind source_system connection_name
237 IBA TTIED_ASSET ODS t_ipa_tide_assets stg_t_ipa_tide_assets UPDDATE 2000-10-27 14:43:32.3814571 L 2026-04-09 14:24:13.720 Y IBA adf-oracle-source-connection-string
Our team uses incremental logic like below
SELECT *
,@{pipeline().parameters.Source_System} AS source_system
FROM @{item().src_schema}.@{item().src_table}
WHERE @{item().watermark_column} > '@{item().watermark_column_value}'
AND @{item().watermark_column} <= '@{pipeline().parameters.PipelineStartTime}'
for your reference
below are the column name from source
TIED_ASSET_NO CONTRACT_NO PROGRAM_NO Q1_PROFESSIONAL_POLICYHOLDER Q2_MEETS_FINANCE Q3_STATUS_OBVIOUS Q4_BROKER_CONFIRMED Q5_BROKER_SUPPORT Q6_CLIENT_RISK_MGMT ALL_DOC_UPLOADED OFFER_EMAIL_INCLUDED "ACTION" INSUSER INSDATE UPDUSER UPDDATE IS_TIED_ASSET_EDITED COMMENTS PROFESSIONAL_STATUS
I have tried the below
@concat(
'SELECT ',
'TIED_ASSET_NO, ',
'CONTRACT_NO, ',
'PROGRAM_NO, ',
'Q1_PROFESSIONAL_POLICYHOLDER, ',
'Q2_MEETS_FINANCE, ',
'Q3_STATUS_OBVIOUS, ',
'Q4_BROKER_CONFIRMED, ',
'Q5_BROKER_SUPPORT, ',
'Q6_CLIENT_RISK_MGMT, ',
'ALL_DOC_UPLOADED, ',
'OFFER_EMAIL_INCLUDED, ',
'"ACTION", ',
'INSUSER, ',
'INSDATE, ',
'UPDUSER, ',
'UPDDATE, ',
'IS_TIED_ASSET_EDITED, ',
'COMMENTS, ',
'PROFESSIONAL_STATUS, ',
'CHR(39) || ', string(pipeline().parameters.Source_System), ' || CHR(39) AS source_system ',
'FROM ',
trim(item().src_schema), '.', trim(item().src_table), ' ',
'WHERE UPDDATE > TO_DATE(''1900-01-01 00:00:00'',''YYYY-MM-DD HH24:MI:SS'')'
It loads the data but only 24 records
why? can someone explain?