Hello, everyone!

Today I'm going to introduce you FI Spark. It helps beginners learn about FI faster.

Applicable Versions

6.5.x

Symptom

The user develops a SparkSQL application and accesses the application in JDBC mode.

However, an error occurs when the application is being commissioned on IDE. The error information is displayed as follows:

4355 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.memory.max=1819MB 4355 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:os.memory.total=123MB java.lang.IllegalArgumentException: A HostProvider may not be empty! at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:78) at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1224) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:933) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:681) at org.apache.hive.jdbc.utils.ZkSample.initiateFIZK(ZkSample.java:41) at org.apache.hive.jdbc.utils.ZkSample.<init>(ZkSample.java:36) at org.apache.hive.jdbc.HiveConnection.setUpZkWatcher(HiveConnection.java:258) at org.apache.hive.jdbc.HiveConnection.createClient(HiveConnection.java:239) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:197) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.huawei.bigdata.spark.examples.ThriftServerQueriesTest.executeSql(ThriftServerQueriesTest.java:68) at com.huawei.bigdata.spark.examples.ThriftServerQueriesTest.main(ThriftServerQueriesTest.java:49)

Fault Locating

1. The configuration information for accessing ZooKeeper fails to be obtained. The hive-site.xml file that contains the required configuration information is not put in classpath but other relevant configuration items can be read properly.

2. After further analysis, it is found that the name (spark.deploy.zookeeper.url.client) of the configuration item in the configuration file is inconsistent with that (spark.deploy.zookeeper.url) in the implementation code. When the code parses the configuration file, the spark.deploy.zookeeper.url configuration item is blank, resulting in the error.

3. On the Linux client, the name of the {Client installation path}/Spark/spark/conf/hive-site.xml file is spark.deploy.zookeeper.url, which is consistent with that in the implementation code. The error is displayed in the {Decompression path of the client installation package}\Spark\config\hive-site.xml file on the Windows client.

Solution

On the installed Linux Spark client, obtain the hive-site.xml configuration file and put the file in classpath.

For example, put the file in the following path:

hostname:/opt/client/Spark/spark/conf #  ...... hive-site.xml ......

Previous: An Error Occurs When Spark  Thrift Accesses Spark (1)