Hello, everyone!

Today I'm going to introduce you DAYU

background:

        Configuring and running collection tasks is a prerequisite for building data assets. The following examples illustrate how to configure collection tasks to achieve the purpose of flexibly collecting metadata.

Scenario 1: The newly added data table in the user's database, the collection task only collects the newly added table

Configuration example: According to the following configuration, the collection task will only collect table4. (Premise: table1-table3 are already in the data asset)

Scenario 2: A new data table is added to the user's database, and the collection task collects all tables specified in the data source

Configuration example: According to the following configuration, the collection task will collect all tables under default

Scenario 3: There are new data tables in the user's database, and the collection task only collects tables that already exist in the data assets

Configuration example: According to the following configuration, the collection task only collects table1/table2/table3

Scenario 4: The data table in the user's database is deleted, and the collection task can delete the corresponding data table in the data asset

Configuration example: According to the following configuration, the collection task will delete table2 in the data asset

to sum up

• The description of the configuration items of the database and the data table in the collection task:

1. The setting items of the database and data table in the collection task limit the scope of the metadata scanned by the collection task.
   

2. The database setting item limits the scope of scanning the database. When no setting is made, all databases are selected by default;
   

3. The data table setting item limits the scope of the scanned data table, and can be set for each selected database. When no data table is set under a certain database, all data tables under this database are scanned by default.
   

4. When the database and data table setting items are empty, all data tables under the data connection are scanned by default.

Note: The configuration items in the metadata collection specify the update and delete strategy when the collection task performs metadata collection. It should be noted that the configured update and delete strategy is applied within the scope of the database and data table configured by the user.

• The data source metadata updated option refers to how the metadata in the data directory is updated when the metadata in the data source changes.

1. Update only the metadata in the data directory: The collection task only updates the metadata that has been collected in the data directory

2. Only add new metadata: The collection task only collects metadata that exists in the data source, but does not exist in the data directory

3. Update the metadata in the data directory and add new metadata: the collection task fully synchronizes the metadata in the data source

4. Ignore update and add operations: do not collect metadata from the data source

• Data source metadata deleted refers to the way the metadata in the data directory is deleted when the metadata in the data source changes.

1. Delete metadata from the data directory: When some metadata in the data source has been deleted, the corresponding metadata in the data directory will also be deleted synchronously

2. Ignore deletion: When some metadata in the data source has been deleted, the corresponding metadata will not be deleted synchronously in the data directory