Got it

Oozie Basic Information

Latest reply: Dec 23, 2021 08:41:33 374 1 1 0 0

Hello, everyone!

Today I'm going to introduce you FusionInsight(FI) Oozie. It helps beginners learn about FusionInsight)FI faster.


Component Introduction

Oozie is a workflow scheduling management system for Hadoop jobs.

Oozie workflow is a group of actions placed in the directed acyclic graph (DAG) of control dependence, which ensures that subsequent operations are started after the preceding operation finishes.

Oozie coordinator triggers the Oozie workflow by time (frequency) and valid data.

Oozie supports multiple Hadoop jobs, including HDFS, MapReduce, Hive, Streaming MapReduce, Loader, Spark, and Distcp) and system-related operations, for example, Java and Shell.

The Oozie engine is a web application integrated into Tomcat by default. Oozie uses PostgreSQL databases.

Oozie provides an Ext-based web console, through which users can view and monitor Oozie workflows. Oozie provides an external REST web service API for the Oozie client to control workflows (such as starting and stopping operations), and orchestrate and run Hadoop MapReduce jobs.

Oozie framework


Log Description

Log path:Oozie logs are stored in the following paths by default:

  • Run logs: /var/log/Bigdata/oozie

  • Audit logs: /var/log/Bigdata/audit/oozie

Log archive role: Oozie logs are categorized into run logs, script logs, and audit logs. The maximum size of a run log file is 20 MB, and a maximum of 20 run log files can be reserved. The maximum size of an audit log file is 20 MB, and a maximum of 20 audit log files can be reserved.

A compressed log file is generated for ozzie.log every hour. 720 files (total log files generated in one month) are reserved by default.

Oozie log list

Log Type

Log File Name

Description

Run log

access.log

Access log file

catalina.out

Tomcat startup log file

check-serviceDetail.log

Service check log file

cleanupDetail.log

Cleanup operation log file

oozie-import-crt.log

Certificate import log file

oozie-instrumentation.log

Oozie parameter log file

oozie-jpa.log

openJPa run log file

oozie.log

Oozie run log file

oozie-omm-gc.log.0.current

GC log file

Script log

postinstallDetail.log

Preinstallation log file

prestartDetail.log

Pre-startup log file

startDetail.log

Service startup log file

stopDetail.log

Service stop log file

Audit log

oozie-audit.log

Audit log file

  • You can check the following logs to obtain the information about errors occurring during installation, start, and stop processes.

    • Run log of the installation process: /var/log/Bigdata/oozie/postinstallDetail.log

    • Run log of the pre-startup process: /var/log/Bigdata/oozie/prestartDetail.log

    • Run log of the startup process: /var/log/Bigdata/oozie/startDetail.log

    • Database initialization log of the startup process: /var/log/Bigdata/oozie/startDetail.log

  • You can check the following logs to obtain the information about errors (such as concerning and automatic restart) occurring during the operation process.

    • Tomcat run logs: /var/log/Bigdata/oozie/catalina.out

    • NodeAgent run log: /var/log/Bigdata/nodeagent/agentlog/agent.log

    • NodeAgent monitoring log: /var/log/Bigdata/nodeagent/monitorlog/monitor.log

    • Controller run log: /var/log/Bigdata/controller/controller.log

  • You can check the following logs to obtain the information about job execution failures.

    Oozie run logs: /var/log/Bigdata/oozie/oozie.log

  • stdout log of Yarn, which is the most important log that determines whether the task is successfully executed. Generally, all job execution errors are output to this log.

Hope this helps!

  • x
  • convention:

olive.zhao
Admin Created Dec 23, 2021 08:41:33

Thanks for your sharing!
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.