DataGrip with Hive cannot introspect Schema (Required field 'functionName' is unset!)
Hi, I'm trying to use DataGrip with Hive server, but I keep getting an error when trying to introspect the schema. Stack trace and environment details are below. It appears that the Hive protocol expects the `functionNamePattern` parameter in the call to `DatabaseMetaData.getFunctions()` to NOT be null, but looks like DataGrip might be passing `null`.
Is there a way to disable fetching functions? I don't see a `function` type in the object filter examples.
Hive Version (JDBC and HiveServer2): 2.1.1
DataGrip 2018.1
Build #DB-181.4203.585, built on April 2, 2018
Licensed to Aaron Hiniker
You have a perpetual fallback license for this version
Subscription is active until June 6, 2019
JRE: 1.8.0_152-release-1136-b20 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
macOS 10.13.5
2018-07-26 18:24:39,229 [1311293888] WARN - lij.database.util.ErrorHandler - Required field 'functionName' is unset! Struct:TGetFunctionsReq(sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:72 EB 62 58 4A E0 4B 73 B2 9E 73 18 36 2D 58 E1, secret:D3 CA 36 0B 05 16 43 9D 8C 46 65 1F 84 0C 84 8A)), catalogName:, schemaName:default, functionName:null)
java.sql.SQLException: Required field 'functionName' is unset! Struct:TGetFunctionsReq(sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:72 EB 62 58 4A E0 4B 73 B2 9E 73 18 36 2D 58 E1, secret:D3 CA 36 0B 05 16 43 9D 8C 46 65 1F 84 0C 84 8A)), catalogName:, schemaName:default, functionName:null)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getFunctions(HiveDatabaseMetaData.java:330)
at com.intellij.database.remote.jdbc.impl.RemoteDatabaseMetaDataImpl.getFunctions(RemoteDatabaseMetaDataImpl.java:565)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
at com.sun.proxy.$Proxy117.getFunctions(Unknown Source)
at sun.reflect.GeneratedMethodAccessor1559.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.execution.rmi.RemoteUtil.invokeRemote(RemoteUtil.java:179)
at com.intellij.execution.rmi.RemoteUtil.access$300(RemoteUtil.java:39)
at com.intellij.execution.rmi.RemoteUtil$RemoteInvocationHandler.invoke(RemoteUtil.java:275)
at com.sun.proxy.$Proxy118.getFunctions(Unknown Source)
at com.intellij.database.introspection.jdbcMetadataWrappers.DatabaseMetaDataWrapper.getFunctionsInner(DatabaseMetaDataWrapper.java:375)
at com.intellij.database.introspection.jdbcMetadataWrappers.DatabaseMetaDataWrapper.lambda$routines$1(DatabaseMetaDataWrapper.java:356)
at com.intellij.database.introspection.jdbcMetadataWrappers.DatabaseMetaDataWrapper.lambda$handleExceptions$2(DatabaseMetaDataWrapper.java:399)
at com.intellij.database.introspection.jdbcMetadataWrappers.ClosableIt$3.nextIt(ClosableIt.java:88)
at com.intellij.database.introspection.jdbcMetadataWrappers.ClosableIt$LazyAppendableClosableIt.nextImpl(ClosableIt.java:72)
at com.intellij.util.containers.JBIterator.peekNext(JBIterator.java:150)
at com.intellij.util.containers.JBIterator.hasNext(JBIterator.java:110)
at com.intellij.database.introspection.GenericIntrospector.introspectRoutinesInSchema(GenericIntrospector.java:417)
at com.intellij.database.introspection.GenericIntrospector.lambda$introspectSchemasByCatalogs$7(GenericIntrospector.java:213)
at java.lang.Iterable.forEach(Iterable.java:75)
at com.intellij.database.introspection.GenericIntrospector.lambda$forEachSchemaInCatalog$15(GenericIntrospector.java:249)
at com.intellij.database.introspection.GenericIntrospector.forEachCatalog(GenericIntrospector.java:244)
at com.intellij.database.introspection.GenericIntrospector.forEachSchemaInCatalog(GenericIntrospector.java:249)
at com.intellij.database.introspection.GenericIntrospector.introspectSchemasByCatalogs(GenericIntrospector.java:211)
at com.intellij.database.introspection.GenericIntrospector.introspectSchemas(GenericIntrospector.java:200)
at com.intellij.database.introspection.GenericIntrospector.lambda$null$0(GenericIntrospector.java:157)
at org.jetbrains.dekaf.core.BaseFacade.inSession(BaseFacade.java:125)
at com.intellij.database.introspection.GenericIntrospector.lambda$introspectAuto$1(GenericIntrospector.java:151)
at com.intellij.database.model.impl.BaseModel.modify(BaseModel.java:115)
at com.intellij.database.model.impl.BaseModel.modify(BaseModel.java:100)
at com.intellij.database.model.impl.BaseModel.modify(BaseModel.java:85)
at com.intellij.database.introspection.GenericIntrospector.introspectAuto(GenericIntrospector.java:151)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.introspectDatabases(DatabaseModelLoader.java:419)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$null$2(DatabaseModelLoader.java:312)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.withFacade(DatabaseModelLoader.java:533)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$introspect$3(DatabaseModelLoader.java:294)
at com.intellij.database.dataSource.DataSourceSyncManager.lambda$null$0(DataSourceSyncManager.java:41)
at com.intellij.database.dataSource.DatabaseConnectionManager$Executor.perform(DatabaseConnectionManager.java:363)
at com.intellij.database.dataSource.DatabaseConnectionManager$Executor.lambda$sync$2(DatabaseConnectionManager.java:302)
at com.intellij.database.dataSource.AsyncUtil.withAsyncFriendly(AsyncUtil.java:161)
at com.intellij.database.dataSource.DatabaseConnectionManager$Executor.sync(DatabaseConnectionManager.java:298)
at com.intellij.database.dataSource.DatabaseConnectionManager$Builder.sync(DatabaseConnectionManager.java:112)
at com.intellij.database.dataSource.DataSourceSyncManager.lambda$static$1(DataSourceSyncManager.java:40)
at com.intellij.database.dataSource.DataSourceSyncManager$SyncProcessor$1.perform(DataSourceSyncManager.java:262)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.introspect(DatabaseModelLoader.java:291)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$run$0(DatabaseModelLoader.java:271)
at com.intellij.database.dataSource.LocalDataSource.performBatch(LocalDataSource.java:1192)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.run(DatabaseModelLoader.java:269)
at com.intellij.database.dataSource.DataSourceSyncManager$SyncProcessor.performSync(DataSourceSyncManager.java:264)
at com.intellij.database.dataSource.AsyncUtil.lambda$null$6(AsyncUtil.java:58)
at com.intellij.database.dataSource.AsyncUtil.lambda$underProgress$14(AsyncUtil.java:130)
at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:580)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:525)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:85)
at com.intellij.database.dataSource.AsyncUtil.underProgress(AsyncUtil.java:136)
at com.intellij.database.dataSource.AsyncUtil.underProgress(AsyncUtil.java:130)
at com.intellij.database.dataSource.AsyncUtil.lambda$captureIndicator$7(AsyncUtil.java:58)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:305)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.thrift.protocol.TProtocolException: Required field 'functionName' is unset! Struct:TGetFunctionsReq(sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:72 EB 62 58 4A E0 4B 73 B2 9E 73 18 36 2D 58 E1, secret:D3 CA 36 0B 05 16 43 9D 8C 46 65 1F 84 0C 84 8A)), catalogName:, schemaName:default, functionName:null)
at org.apache.hive.service.cli.thrift.TGetFunctionsReq.validate(TGetFunctionsReq.java:542)
at org.apache.hive.service.cli.thrift.TCLIService$GetFunctions_args.validate(TCLIService.java:10277)
at org.apache.hive.service.cli.thrift.TCLIService$GetFunctions_args$GetFunctions_argsStandardScheme.write(TCLIService.java:10334)
at org.apache.hive.service.cli.thrift.TCLIService$GetFunctions_args$GetFunctions_argsStandardScheme.write(TCLIService.java:10303)
at org.apache.hive.service.cli.thrift.TCLIService$GetFunctions_args.write(TCLIService.java:10254)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:71)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at org.apache.hive.service.cli.thrift.TCLIService$Client.send_GetFunctions(TCLIService.java:388)
at org.apache.hive.service.cli.thrift.TCLIService$Client.GetFunctions(TCLIService.java:380)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1443)
at com.sun.proxy.$Proxy2.GetFunctions(Unknown Source)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getFunctions(HiveDatabaseMetaData.java:328)
at com.intellij.database.remote.jdbc.impl.RemoteDatabaseMetaDataImpl.getFunctions(RemoteDatabaseMetaDataImpl.java:565)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
... 3 more
Please sign in to leave a comment.
Hello, I am encountering the same problem, I have also tried to add object filter "routine:.*||operator:.*||table:.*||function:.*", but it did not help. It would be very nice to find a fix a workaround. Now Datagrip does not proceed with the DB introspection and also other object types (e.g. tables) are not scanned.
Thanks in advance!
Best regards
Evgeny
Hi, I have solved the problem. It seems that Intellij/Datagrip sends wrong parameters to JDBC Driver (details also here: https://youtrack.jetbrains.com/issue/DBE-3115). So I recompiled the HiveJDBC driver with some corrections (tested on linux):
1) git clone https://git-wip-us.apache.org/repos/asf/hive.git
2) cd hive-release-1.2.1/
3) change source code: vi jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java #see here: https://youtrack.jetbrains.com/issue/DBE-3115#focus=streamItem-27-1778089-0-0
4) mvn clean install -DskipTests -Phadoop-2
5) scp jdbc/target/hive-jdbc-1.2.1-standalone.jar my_pc:my_dir
Good luck!
Best regards
Evgeny
I submitted a bug report for this because the JDBC Spec at https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getFunctions-java.lang.String-java.lang.String-java.lang.String- requires the function name pattern to be set.
That is fixed. But you won't get any functions from driver, as this function does not work as expected. Providing schema yields empty result set, while without schema it yields functions with empty schema name and `schema.function` as function name.
Understand, I'm working on the Apache side to get that issue resolved. What is the DataGrip behavior during introspection? Does it send a schema name or not? I would expect not sending a schema to return all functions and sending a schema would only show functions in that schema, is that what you'd expect?
According to my understanding DataGrip sends requests only about the schemas that are defined by a user. But according to Alexander Hive does not respond in the correct manner. Hive should send back schema name as FUNCTION_SCHEM and and function name as FUNCTION_NAME separately.
@Sweeks We introspect schemas separately, so @Esgorbachev is right
I've made a hack on our side to receive function list (for hive I supply no schema and then filter by schema in function name if no separate schema supplied).
I'll be glad to remove that, when that will be fixed in driver :)