11/13/2023 0 Comments Etl processes using ssisto identify which change happened on the record you need to compare the change set with existing records and separate inserted, updated, and deleted records. The change set might contains inserted, updated, or deleted records. This will create the change set for the data table. The idea behind this method is to store the latest ETL run time in a config or log table, and then in the next ETL run just load records from the source table that have modified (with their modified date greater than or equal to) after the latest ETL run datetime. In this post we will go through process of implementing an Incremental Load solution with SSIS using modified datetime column. These datetime columns can be used to implement Incremental Load. In many cases the source system have columns such as Created datetime or Modified datetime that stores the datetime of creating and modifying the record in the table. If you are using some data sources that doesn’t support CDC then your first stop should be thinking about datetime columns or timestamp column in the source data sets. In fact the source database should support CDC. CDC method is performing great, however it is not a possible option in all scenarios. Using Datetime columns to implement incremental loadĪs I mentioned in the top, Previously I’ve explained how to do incremental load with CDC (Change Data Capture). With the methods I explain in Incremental Load posts I intend to make the process of writing and developing Incremental Load easier and faster for you. For example would this small data set stay small 10 years later? if not, do the incremental load! This might be the only acceptable reason to do a full load rather than incremental, However this should be carefully considered with factors such as growth rate of the data set. I’ve seen this approach in many consultancy companies which care about limitation on the chargeable hours for customer more than the quality of work and customer satisfaction!
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |