spark-sql 执行语句异常,什么问题?
spark版本 2.3.0
hdfs版本 3.1.1.3.1
hive版本 3.0.0.3.1
Hive Session ID = 84330dfe-a042-4517-a169-7538cc362a37
23/03/16 17:36:31 WARN session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
23/03/16 17:36:31 WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 5s. listPartitionsWithAuthInfo
org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178)
at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_partitions_ps_with_auth(ThriftHiveMetastore.java:3225)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_ps_with_auth(ThriftHiveMetastore.java:3212)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsWithAuthInfo(HiveMetaStoreClient.java:1526)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsWithAuthInfo(HiveMetaStoreClient.java:1517)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitionsWithAuthInfo(SessionHiveMetaStoreClient.java:1000)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
at com.sun.proxy.$Proxy29.listPartitionsWithAuthInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
Cannot write to null outputStream
输出流为空啊
参考GPT和自己的思路:
根据日志信息,推测此次 spark-sql 执行异常的原因可能是连接到 HiveMetaStoreClient 失败。具体来说,可能是因为在连接过程中出现了网络中断或者连接超时等问题,导致无法向 outputStream 写入数据。
建议您检查网络连接是否正常,并尝试重新连接 HiveMetaStoreClient。您还可以查看相关的日志信息,以进一步确定问题的具体原因。另外,您也可以尝试升级或降级 Hadoop、Hive 或 Spark 的版本,以解决此类问题。