Hello, everyone!
Today I'm going to introduce you DAYU
1. DAYU data blood relationship realization plan
In the DAYU platform, the blood relationship is generated by configuring the data processing migration type operator in the data development module. At present, the blood relationship generated by static configuration of the operator and the blood relationship of some operator instances are collected.
2. Types of operators supported by data bloodlines
2.1 SQL type operator
Currently , it supports blood analysis of DLISQL , DWSSQL , HIVESQL operators, and supports multiple SQL analysis and column-level blood analysis. When there are temporary tables in the statement, related temporary table entities will be automatically created in the data assets.
2.1.1 DLISQL operator
1) Support the analysis of the blood relationship generated by the data insertion between the table and the table in the DLI .
2) Support the blood relationship between the OBS file generated by the table creation statement and the DLI table
2.1.2 DWSSQL operator
1) Support the blood relationship between DWS tables generated by DDL operations such as Create table like/as
2) Support the blood relationship between DWS tables generated by DML operations such as Insert into
2.1.3 HIVESQL operator
3) Support Create table like / as like ddl generated operation HIVE blood between the table
4) Support the blood relationship between DWS tables generated by DML operations such as Insert into/overwrite
2.2 Data integration type operator
2.2.1 CDM operator
Support MRSHIVE , DLI , DWS , RDS , OBS , CSS between the table file migration caused by blood.
2.2.2 ETLJOB operator
Support the blood relationship between the etl task between DLI , OBS , MYSQL and DWS
2.2.3 OBSManager operator
Support the blood relationship between directory and file copy migration between obs
2.3 Other types of operators
3. Guidance for blood collection
3.1 Automatic analysis of blood relationship configuration
1. In dayu , job information in data development can be synchronized to data assets for presentation.
a. In data development, when the job is " executed " , the related information of the job will be synchronized to the data asset
b. On the data catalog page of the data asset, you can query the data development jobs and operators. The job in data development corresponds to the Job type, and the operator corresponds to the Node type. Note: The job information in dlf does not belong to any data connection, so if you check the data connection in the search criteria, the result will not be queried.
c. In the details page of the job (including the operator of the job), you can jump to the data development page
d. On the operator (need to be an operator type that supports blood ties) details page, you can view the script content of the operator and blood ties information. (By clicking the " + " and " - " icons at the left and right ends of the node in the blood relationship graph, you can further expand and view the upper and lower links of the blood relationship.)
3.2 Manual configuration of blood relationship
Currently manual configuration of bloodlines supports DLI, DWS, HIVE, CSS, OBS and custom types
Supported operator types include CDMJOB , DWSSQL , HIVESQL , DLISQL , ETLJOB , MRSSPARK
