Got it

An Error Occurs When Spark Thrift Accesses Spark (2)

Latest reply: Dec 17, 2021 03:10:21 241 1 1 0 0

Hello, everyone!

Today I'm going to introduce you FI Spark. It helps beginners learn about FI faster.

Applicable Versions

6.5.x

Symptom

The user develops a SparkSQL application and accesses the application in JDBC mode.

However, an error occurs when the application is being commissioned on IDE. The error information is displayed as follows:

4355 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.memory.max=1819MB 4355 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.memory.total=123MB java.lang.IllegalArgumentException: A HostProvider may not be empty! at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:78) at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1224) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:933) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:681) at org.apache.hive.jdbc.utils.ZkSample.initiateFIZK(ZkSample.java:41) at org.apache.hive.jdbc.utils.ZkSample.<init>(ZkSample.java:36) at org.apache.hive.jdbc.HiveConnection.setUpZkWatcher(HiveConnection.java:258) at org.apache.hive.jdbc.HiveConnection.createClient(HiveConnection.java:239) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:197) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.huawei.bigdata.spark.examples.ThriftServerQueriesTest.executeSql(ThriftServerQueriesTest.java:68) at com.huawei.bigdata.spark.examples.ThriftServerQueriesTest.main(ThriftServerQueriesTest.java:49)

Fault Locating

1. The configuration information for accessing ZooKeeper fails to be obtained. The hive-site.xml file that contains the required configuration information is not put in classpath but other relevant configuration items can be read properly.

2. After further analysis, it is found that the name (spark.deploy.zookeeper.url.client) of the configuration item in the configuration file is inconsistent with that (spark.deploy.zookeeper.url) in the implementation code. When the code parses the configuration file, the spark.deploy.zookeeper.url configuration item is blank, resulting in the error.

3. On the Linux client, the name of the {Client installation path}/Spark/spark/conf/hive-site.xml file is spark.deploy.zookeeper.url, which is consistent with that in the implementation code. The error is displayed in the {Decompression path of the client installation package}\Spark\config\hive-site.xml file on the Windows client.

Solution

On the installed Linux Spark client, obtain the hive-site.xml configuration file and put the file in classpath.

For example, put the file in the following path:

hostname:/opt/client/Spark/spark/conf #  ...... hive-site.xml ......

Previous: An Error Occurs When Spark  Thrift Accesses Spark (1)    

  • x
  • convention:

olive.zhao
Admin Created Dec 17, 2021 03:10:21

Thanks for your sharing!
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.