I am trying to create an incremental load from Source table into mrr/src tables, and in my Source table I've got created date and 'modified' date.
Adding New Records:
There are two approaches to do the incremental, and I am wondering which way is better:
Merge join task (left join) between Source table and mrr/src table and taking all records which are not exist in mrr/src table.
saving the last upload date and taking all new records, which have higher 'created' date than the last upload date.
I personally choose the second option, but is the first option is more preferable in terms of performance ?
Updating existing Records:
Again there are two options:
Using Conditional Split task and having all comparisons between fields in Source Table and mrr/src table.
Using 'Modified' date in Source table and compare it to last upload date, in order to know which records were updated since the last time i checked the Source table.
Is there a preferred option, and am I missing an option?