Got it

Hive:Basic Information

Latest reply: Dec 12, 2021 13:20:28 1296 3 6 0 0

1.1 Basic Information

1.1.1 Architecture Overview

Hive is a data warehouse framework built on Hadoop. It provides storage of structured data using the Hive query language (HQL), a language like the structured query language (SQL). Hive converts HQL statements to MapReduce jobs for querying and analyzing massive data stored in Hadoop clusters.

The following figure shows the architecture.

094101m03m3fv1kfvom1o3.png?2.png

Hive provides the following functions:

l   Analyzes massive structured data and summarizes analysis results.

l   Allows complex MapReduce jobs to be compiled in SQL languages.

l   Supports flexible data storage formats, including JavaScript object notation (JSON), comma separated values (CSV), TextFile, RCFile, and SequenceFile.

Hive component description:

Name

Description

HiveServer

Multiple HiveServers can be deployed in a cluster to share loads. The HiveServer provides Hive database services externally, translates HQL statements submitted by users into related Yarn tasks or HDFS operations to complete data extraction, conversion, and analysis.

MetaStore

l  Multiple MetaStores can be deployed in a cluster to share loads. The MetaStore provides Hive metadata services, and reads, writes, maintains and modifies the Hive table structure and property information.

l  It also provides Thrift interfaces for MetaStore clients such as HiveServer, Impala, and WebHCat to access and operate metadata.

WebHCat

Multiple WebHCats can be deployed in a cluster to share loads. The WebHCat provides REST interfaces and runs the Hive command through REST to submit MapReduce jobs.

Hive client

The Hive client includes the human-machine interaction command line Beeline, JDBC drive for JDBC applications, Python drive for Python applications, and JAR files of HCatalog for MapReduce.

 

1.1.2 Log Overview

Hive logs consist of running and audit logs. If the failure happens during MapReduce execution, MapReduce logs need to be checked. For details, see related sections of MapReduce logs.

Log path:

l   Hive run logs are saved in /var/log/Bigdata/hive.

l   Hive audit logs are saved in /var/log/Bigdata/audit/hive.

Table 1-1 describes log functions.

Table 1-1 Hive log list

Log Type

Log File Name

Description

HiveServer run log

hive.log

Hive run log

hive.xxxx-xx-xx.log.zip

Archive of historical run logs, identified by the archive date. For example, hive.2016-02-29_13-08-24.[7].log.zip indicates that logs are archived at 13:08:24 on 2016-02-29.

hive-omm-gc.log

HiveServer GC log

hadoop-omm-jar-192-168-1-18.out

Console log, containing System.out output information of applications

cleanupDetail.log

Logs related to Hive startup and stop processes. If Hive startup or stop fails, related information is recorded in the logs.

prestartDetail.log

startDetail.log

stopDetail.log

MetaStore run log

metastore.log

MetaStore run log

metastore.xxxx-xx-xx.log.zip

Archive of historical run logs, identified by the archive date. For example, metastore.2016-02-29_13-08-24.[7].log.zip indicates that logs are archived at 13:08:24 on 2016-02-29.

hive-omm-gc.log.0.current

MetaStore GC log

postinstallDetail.log

Log related to MetaStore startup and stop processes. If MetaStore startup or stop fails, related information is recorded in the logs.

prestartDetail.log

startDetail.log

stopDetail.log

HiveServer audit log

hive-audit.log

HiveServer audit log

hive-audit.2016-03-09_17-57-12.[1].log.zip

Archive file of HiveServer audit logs

MetaStore audit log

metastore-audit.log

MetaStore audit log

metastore-audit.2016-03-03_22-02-31.[3].log.zip

Archive file of MetaStore audit logs

 

1.1.3 Log Viewing

1.1.3.1 How Do I Locate a Log Related to an Error?

Problem Locating

If a problem occurs, locate the step at which the problem occurs. Hive-related problems can be classified into the following categories:

094243lrr6u0uro3r0z15e.png?3.png

Installation and Startup Troubleshooting Guidelines

Hive installation and startup are in a continuous process. If a problem occurs during installation and startup, an error message is prompted on the FusionInsight Manager page. Perform the following steps to troubleshoot installation and startup problems:

094322js884651b1qykyx5.png?4.png

Reference 1:

Check the following logs comprehensively and locate the problem based on clues provided in the logs.

In the /var/log/Bigdata/hive/metastore/ directory:

metastore.log, prestartDetail.log, startDetail.log

In the /var/log/Bigdata/hive/hiveserver directory:

hive.log, prestartDetail.log, startDetail.log

Common errors in the logs and solutions:

Error message 1:

org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "160.164.0.146", database "HIVEMETA"

Solution 1:

On the node with DBService installed,

1.         Modify /srv/BigData/dbdata_service/data/pg_hba.conf by adding host hivemeta hive 0.0.0.0/0 sha256 to the file.

2.         Run the source /opt/huawei/Bigdata/dbservice/.dbservice_profile command and then the gs_ctl -D $GAUSSDATA reload command to make new configurations take effect.

3.         Restart the MetaStore service after modification.

Error message 2:

dbservice in a read-only transactiongsql

Solution 2:

The main cause may be that DBService connected to MetaStore is not on the active node due to an active/standby DBService switchover. Restart MetaStore after DBService is ready.

Other information:

a) source /opt/huawei/Bigdata/dbservice/.dbservice_profile 
b) gsql -h 192.168.xx.xx (Floating IP address of DB) -p 20051 -d hivemeta -U hive -W HiveUser@

Perform the preceding steps to ensure that DBService can be connected and the hivemeta database can be accessed. If DBService cannot be connected after the preceding steps are performed, follow instructions in section Common Faults.

Health Check Troubleshooting Guidelines

If there is a health check problem, generally, services are unavailable and the following service status is displayed on the FusionInsight Manager page:

l   Bad

l   Active-Active (As C30 version is in active/standby mode, active-active mode is abnormal.)

l   Concerning

l   UnKnown

The following figure provides the troubleshooting guidelines:

Figure 1-1 Flow chart

094403kl66zchafj6kjojy.png?5.png

 

Reference 2:

Hive highly depends on the following components:

l   Ldap

l   KrbServer

l   ZooKeeper

l   HDFS

l   YARN

l   DBService

Reference 3:

Generally, you can find useful information in the HiveServer and MetaStore run logs, for example, the common security authentication and DBService connection problems.

Security authentication problems:

1.         Go to the /opt/huawei/Bigdata/hive/hive-1.1.0/bin path.

2.         Check whether hive.keytab exists.

3.         Check whether hive.keytab has read and write permission.

4.         Check whether hive.keytab is available as follows:

[root@192.168-1-18 bin]# kinit -kt hive.keytab hive/hadoop.hadoop.com 
[root@192-168-1-18 bin]# klist  
Ticket cache: FILE:/tmp/krb5cc_0 
Default principal: hive/hadoop.hadoop.com@HADOOP.COM 
Valid starting     Expires            Service principal 
03/18/16 15:19:10  03/19/16 15:19:10  krbtgt/HADOOP.COM@HADOOP.COM

DBService connection problems:

1.         Access the node with MetaStore installed.

2.         Check whether hive.keytab exists.

Run the following command to check whether DBService is connected.

a) source /opt/huawei/Bigdata/dbservice/.dbservice_profile 
b) gsql -h 192.168.xx.xx (Floating IP address of DB) -p 20051 -d hivemeta -U hive -W HiveUser@

Reference 4:

If no exception is found after the preceding steps are performed, the process and mechanism may have some problems.

1.         View the monitor.log of NodeAgent of each HiveServer to check whether a log with the non-zero status code exists.

Path: /var/log/Bigdata/nodeagent/monitorlog/monitor.log

2015-11-12 04:00,153 INFO  [ServiceCheckThread1_10_HiveServer] errordes: 15001 com.huawei.hadoop.om.agent.monitoring.serviceChecker.JmxServiceStatusChecker.jmxCheckServiceStatus(JmxServiceStatusChecker.java:90)

2.         If it exists, check the corresponding hive.log. Solve the problem based on the log content similar to the following.

2015-11-12 04:00,756 | ERROR | HiveServiceHealthChecker | failed to connection to hiveserver. | org.apache.hadoop.hive.om.monitor.HiveServiceHealthChecker.checkHiveServer(HiveServiceHealthChecker.java:145)

3.         Check whether the JMX service and the health check thread of Hive are normal. The log of the thread is in the hive.log.

2015-11-12 04:00,756 | ERROR | HiveServiceHealthChecker | failed to connection to hiveserver. | org.apache.hadoop.hive.om.monitor.HiveServiceHealthChecker.checkHiveServer(HiveServiceHealthChecker.java:145)

4.         Check whether the dual health check threads of the NodeAgent are working properly. Enable the NodeAgent DEBUG log to check whether the following running records exist. If it does not exist, NodeAgent running or configuration is incorrect.

2015-11-12 04:00,121 DEBUG [ServiceCheckThread1_10_HiveServer] objectName:org.apache.hadoop.hive.om.hiveservice:type=HiveServiceStatusMBean,serviceKey:ServiceStatusCode com.huawei.hadoop.om.agent.monitoring.serviceChecker.JmxServiceStatusChecker.jmxCheckServiceStatus(JmxServiceStatusChecker.java:66)

5.         Check whether dual health check threads are normally started. View agent.log of the NodeAgent to check whether there are normally initialized logs.

2015-11-12 04:00,297 INFO  [HealthMonitorThread1_10_HiveServer] 1_10_HiveServer state changed from UNKNOWN to GOOD com.huawei.hadoop.om.agent.services.HealthMonitorService$ProcessHealthMonitorThread.changeProcessState(HealthMonitorService.java:631)

6.         If the preceding check results are normal, collect logs of the Controller and the NodeAgent and turn to FusionInsight Manager O&M personnel.

[Troubleshooting guidelines for the running phase]

Over 90% Hive problems occur in the running phase. The following provides general troubleshooting guidelines:

Reference 5:

To tackle tricky suspension problems without any returned error codes, collect and analyze the following logs:

Collect and analyze jstack information about Beeline to check whether the client is suspended.

1.         Check the HiveServer audit log. If the audit log shows that no request is received, the fault may occur on the client. In this case, go to step 2. If the audit log shows that the request is received, the Hive service has received and is dealing with the SQL execution request. In this case, go to step 3.

2.         Run the following command to locate the Beeline PID that is 28487:

[root@192-168-1-18 monitorlog]# ps -ef|grep beeline 
root     28487 28427  0 15:54 pts/0    00:00:05 java -cp /opt/duan/client/Hive/Beeline/conf:/opt/duan/client/Hive/Beeline

Run the following command every 20 seconds and redirect the command output to different files. Collect information for over three times.

[root@192-168-1-18 monitorlog]# jstack –F 28487 > /tmp/1.txt

3.         Check the HiveServer run log and jstack information and analyze the current execution status to determine whether to analyze the MetaStore process status. If the analysis is necessary, go to step 4. To obtain HiveServer, find the HiveServer process pid(ps –ef | grep org.apache.hadoop.hive.com.service.HiveService) and run the jstack command in step 2 to redirect HiveServer to a local file based on the PID.

4.         Check and analyze the MetaStore run log and jstack information. To obtain MetaStore, find the MetaStore process pid(ps –ef | grep org.apache.hadoop.metastore.HiveMetaStore) and run the jstack command in step 2 to redirect MetaStore to a local file based on the PID.

5.         Additionally, check the HiveServer and MetaStore GC logs to analyze whether slow execution occurs due to frequent Full GCs.

Reference 6:

1.         Log in to the node housing HiveServer. Two nodes involve changes.

2.         Change INFO to DEBUG for log4j.rootLogger = INFO,DRFA,EventCounter in the /opt/huawei/Bigdata/etc/*_*_HiveServer/hive-log4j.properties file.

1.1.3.2 What Is Log Content?

Audit Log

An audit log records at what time a user sends a request through which statement to HiveServer and MetaStore from which IP address.

The following HiveServer audit log shows that at 14:51:22 on February 1st, 2016, user_chen sent a show tables request to HiveServer from the 192.168.1.18 IP address.

094451bioidr1o5cdji1oa.png?6.png

The following MetaStore audit log shows that at 11:31:15 on January 29, 2016, user hive sent a shutdown request to MetaStore from the 192.168.1.18 IP address.

094459ntdt9m1dhdsnpgk0.png?7.png

Generally, the audit log does not play a role in actual error location. However, the audit log must be checked to solve the following problems:

1.         There is no response after a client sends a request. The audit log can be used to check whether the task suspends on the client or server. If the audit log has no related information, the task suspends on the client. If the audit log has related information, view the run log to locate where the program suspends.

2.         The audit log can be used to check the number of requests in a specified period of time.

HiveServer Run Log

HiveServer receives an SQL statement from a client, compiles the SQL statement to a MapReduce job, executes the MapReduce job at the local end or by submitting to Yarn, and interact with MetaStore to obtain metadata information. The HiveServer run log records a complete SQL execution process.

Generally, if SQL statement running fails, check the HiveServer run log first.

MetaStore Run Log

Typically, if the HiveServer run log contains MetaException or MetaStore connection failure, check the MetaStore run log.

GC Log

Both HiveServer and MetaStore have GC logs. If GC-related problems occur, view the GC logs to quickly locate the cause. For example, if HiveServer or MetaStore frequently restarts, check its GC log.

1.1.3.3 How to View HiveServer Logs

Logs in the Compilation Phase

As shown in the following, Parsing command and Parsing Completed indicates the start and completion of the compilation phase, respectively.

If an error is reported in the compilation phase, the log locates between Parsing command and Parsing Completed.

Note that an SQL statement corresponds to two thread numbers in the log. Information similar to HiveServer2-Handler-Pool: Thread-64 is in the compilation phase and information similar to HiveServer2-Background-Pool: Thread-9502 is in the execution phase. HiveServer stores run logs of all tasks. Therefore, the thread number is required to search the run log.

2015-11-12 04:00,536 | INFO  | HiveServer2-Handler-Pool: Thread-64 | <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121)
2015-11-12 04:00,536 | INFO  | HiveServer2-Handler-Pool:Thread-64 | 
Parsing command: select count(*) from src | org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:185)
2015-11-12 04:00,538 | INFO  | HiveServiceHealthChecker | Will try to open client transport with JDBC Uri: jdbc:hive2://160.138.1.18:21066/default;principal=hive/hadoop.hadoop.com@HADOOP.COM;sasl.qop=auth-conf;auth=kerberos;inner.client.marker=1454571037511 | org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:253)
……
 2015-11-12 04:00,549 | INFO  | HiveServer2-Handler-Pool: Thread-64 | Parse Completed | org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:206)
2015-11-12 04:00,549 | INFO  | HiveServer2-Handler-Pool: Thread-64 | </PERFLOG method=parse start=1454571037536 end=1454571037549 duration=13 from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogEnd(PerfLogger.java:148)
2015-11-12 04:00,549 | INFO  | HiveServer2-Handler-Pool: Thread-64 | <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121)

Summary: If problems occur in the compilation phase, ParseException or SemanticException is reported on the client. Causes of these problems are easy to locate. In most cases, the causes are that SQL syntax is incorrect or keywords that are not supported are entered.

Logs in the Running Phase

A log in the running phase contains three parts: task submission, task execution, and task completion tags, as shown in the following.

org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121)
2015-11-12 04:00,424 | INFO  | HiveServer2-Background-Pool: Thread-9502 | Starting command: select count(*) from src | org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1401)
2015-11-12 04:00,424 | INFO  | HiveServer2-Background-Pool: Thread-9502 | <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.security.authorization.plugin.DisallowTransformHook from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121)
2015-11-12 04:00,424 | INFO  | HiveServer2-Background-Pool: Thread-9502 | </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.security.authorization.plugin.DisallowTransformHook start=1454571039424 end=1454571039424 duration=0 from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogEnd(PerfLogger.java:148)
……
2015-11-12 04:00,425 | INFO  | HiveServer2-Background-Pool: Thread-9502 | <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121)
2015-11-12 04:00,425 | INFO  | HiveServer2-Background-Pool: Thread-9502 | <PERFLOG method=task.MAPRED.Stage-1 from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121)
2015-11-12 04:00,447 | INFO  | HiveServer2-Background-Pool: Thread-9502 | Launching Job 1 out of 1 | org.apache.hadoop.hive.ql.session.SessionState$LogHelper.printInfo(SessionState.java:855)
Tarting
……
2015-11-12 04:00,525 | INFO  | HiveServer2-Background-Pool: Thread-9502 | Total MapReduce CPU Time Spent: 8 seconds 860 msec | org.apache.hadoop.hive.ql.session.SessionState$LogHelper.printInfo(SessionState.java:855)
2015-11-12 04:00,525 | INFO  | HiveServer2-Background-Pool: Thread-9502 | OK | org.apache.hadoop.hive.ql.session.SessionState$LogHelper.printInfo(SessionState.java:855)
2015-11-12 04:00,525 | INFO  | HiveServer2-Background-Pool: Thread-9502 | <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121)
2015-11-12 04:00,525 | INFO  | HiveServer2-Background-Pool: Thread-9502 | </PERFLOG method=releaseLocks start=1454571146525 end=1454571146525 duration=0 from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogEnd(PerfLogger.java:148)
2015-11-12 04:00,526 | INFO  | HiveServer2-Background-Pool: Thread-9502 | </PERFLOG method=Driver.run start=1454571039423 end=1454571146526 duration=107103 from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogEnd(PerfLogger.java:148)

1.1.3.4 Keywords for Quick Problem Locating

1.         If the result fails to be returned after the select count(*) from src statement is executed, search for select count(*) from src in the HiveServer logs.

2.         The HiveServer logs contain all statement information. Therefore, one statement may correspond to multiple search results. Filter information by time.

3.         After the incorrect statement is located in a specified period of time, two thread numbers correspond to the same SQL statement. View the log according to the thread number. As shown in the following, Thread-64 is the thread number of the compilation phase and Thread-9502 is the task number of the execution phase.

2015-11-12 04:00,536 | INFO  | HiveServer2-Handler-Pool: Thread-64 | Parsing command: select count(*) from src | org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:185) 
2015-11-12 04:00,538 | INFO  | HiveServiceHealthChecker | Will try to open client transport with JDBC Uri: jdbc:hive2://160.138.1.18:21066/default;principal=hive/hadoop.hadoop.com@HADOOP.COM;sasl.qop=auth-conf;auth=kerberos;inner.client.marker=1454571037511 | org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:253) 
…… 
2015-11-12 04:00,424 | INFO  | HiveServer2-Background-Pool: Thread-9502 | Starting command: select count(*) from src | org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1401) 
2015-11-12 04:00,424 | INFO  | HiveServer2-Background-Pool: Thread-9502 | <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.security.authorization.plugin.DisallowTransformHook from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121) 
2015-11-12 04:00,424 | INFO  | HiveServer2-Background-Pool: Thread-9502 | </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.security.authorization.plugin.DisallowTransformHook start=1454571039424 end=1454571039424 duration=0 from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogEnd(PerfLogger.java:148)

4.         Run the simple command to locate the desired log information, for example, grep "Thread-64" hive.log > bak.log.

5.         Other common search keywords:

           Exception: Indicates abnormal stack information. You can search by exception type, for example, MetaException and ConnectionLossException.

           Error: Prints error log information.

           Parsing command: Indicates the start tag for SQL statement parsing.

           Starting command: Indicates the start tag for SQL statement execution.

           Launching Job: Starts to submit a MapReduce job.

           Trying to connect to metastore: Indicates the start tag for HiveServer to try to connect to MetaStore.

           Full GC: Checks whether Full GC exists in the GC log.

1.1.3.5 How to Quickly Locate Problem Causes Through Log Analysis

Symptom

The client reports the following error when a user is executing a job.

Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)

Problem Cause Location Procedure

                               Step 1      Locate where a problem occurs, for example, client, HiveServer, MetaStore, and MapReduce.

If the log shows that MapRedTask has an error, the problem occurs on MapReduce. MapReduce logs need to be checked.

Typically, the returned codes of MapReduce jobs have the following meaning:

l  code=1

It is an error code that is returned when a job fails to be submitted. There is detailed abnormal stack information about the error on HiveServer.

l  code=2

It is an error code that is returned when no exception is thrown during job submission, but RuntimeExeception is thrown during Map or Reduce phases of job running. Generally, a Hive exception occurs when each operator processes each line of data. To locate the problem, check the job history log using the job ID.

l  code=3

1. It is an error code that is returned when dynamic partitioning jobs fail at the close phase when data is transferred to the target table or partition.

2. It is an error code that is returned when joining jobs fail at the close phase when data is transferred to the final path.

l  code=5

It is an error code that is returned when a temporary folder fails to be created during job loading.

                               Step 2      Locate job IDs of MapReduce in a Hive log.

Check the Hive log to locate the job ID of MapReduce.

In the HiveServer log, use the statement as a keyword for searching.

Two records corresponding to two thread numbers are searched, as shown in the following:

l   Starting with Parsing command: Exclude it. The thread is for the compilation phase.

l   Starting with Starting command: Locate the thread number, that is, Thread-919.

2015-11-12 04:00,916 | INFO  | HiveServer2-Handler-Pool: Thread-114 | Parsing command: select count(*) from d1 a join d1 b join d1 c | org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:185) 
2015-11-12 04:00,917 | INFO  | HiveServer2-Handler-Pool: Thread-114 | Parse Completed |org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:206) 
…… 
2015-11-12 04:00,375 | INFO  | HiveServer2-Background-Pool: Thread-919 | Concurrency mode is disabled, not creating a lock manager | org.apache.hadoop.hive.ql.Driver.checkConcurrency(Driver.java:159)2015-11-12 04:00,375 | INFO  | HiveServer2-Background-Pool: Thread-919 | <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> | org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:121)2015-11-12 04:00,375 | INFO  | HiveServer2-Background-Pool: Thread-919 | Starting command: select count(*) from d1 a join d1 b join d1 c | org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1401)

Continue to view the log of Thread-919 to locate the keyword starting with Starting job and the job ID, that is, job_1458004189946_0005.

016-03-15 15:27:00,683 | INFO  | HiveServer2-Background-Pool: Thread-919 | Starting Job = job_1458004189946_0005, Tracking URL = https://10-10-1-182:26001/proxy/application_1458004189946_0005/ | org.apache.hadoop.hive.ql.session.SessionState$LogHelper.printInfo(SessionState.java:855)

                               Step 3      View the log corresponding to the job ID on Yarn.

Go to the FusionInsight Manager page and choose Services > Yarn > ResourceManagers, as shown in the following figure.

094551rjbgldjdrdgn1gbg.png?8.png

On the Yarn page, locate the job ID, as shown in the following figure.

094601gdcljssrdbnnnozy.png?9.png

Click the link to open its history logs, as shown in the following:

2015-11-12 04:00,844 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start 
2015-11-12 04:00,844 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length 
2015-11-12 04:00,844 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 
2015-11-12 04:00,325 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child :java.lang.OutOfMemoryError: Java heap space 
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:986) 
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:442) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

Therefore, the problem cause is OOM at the Map phase.

----End

Solution

Increase Map memory.


This article contains more resources

You need to log in to download or view. No account? Register

x
  • x
  • convention:

Lucas_Zhao
Created Dec 17, 2019 09:24:01

good case!
View more
  • x
  • convention:

Unicef
MVE Created Dec 12, 2021 12:24:35

NOTE
View more
  • x
  • convention:

maithi
Created Dec 12, 2021 13:20:28

Good share
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.