DataGrip with Hive cannot introspect Schema (Required field 'functionName' is unset!)

Hi, I'm trying to use DataGrip with Hive server, but I keep getting an error when trying to introspect the schema.  Stack trace and environment details are below.  It appears that the Hive protocol expects the `functionNamePattern` parameter in the call to `DatabaseMetaData.getFunctions()` to NOT be null, but looks like DataGrip might be passing `null`.  

Is there a way to disable fetching functions?  I don't see a `function` type in the object filter examples.

Hive Version (JDBC and HiveServer2): 2.1.1

DataGrip 2018.1
Build #DB-181.4203.585, built on April 2, 2018
Licensed to Aaron Hiniker
You have a perpetual fallback license for this version
Subscription is active until June 6, 2019
JRE: 1.8.0_152-release-1136-b20 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
macOS 10.13.5

 

2018-07-26 18:24:39,229 [1311293888] WARN - lij.database.util.ErrorHandler - Required field 'functionName' is unset! Struct:TGetFunctionsReq(sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:72 EB 62 58 4A E0 4B 73 B2 9E 73 18 36 2D 58 E1, secret:D3 CA 36 0B 05 16 43 9D 8C 46 65 1F 84 0C 84 8A)), catalogName:, schemaName:default, functionName:null) 
java.sql.SQLException: Required field 'functionName' is unset! Struct:TGetFunctionsReq(sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:72 EB 62 58 4A E0 4B 73 B2 9E 73 18 36 2D 58 E1, secret:D3 CA 36 0B 05 16 43 9D 8C 46 65 1F 84 0C 84 8A)), catalogName:, schemaName:default, functionName:null)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getFunctions(HiveDatabaseMetaData.java:330)
at com.intellij.database.remote.jdbc.impl.RemoteDatabaseMetaDataImpl.getFunctions(RemoteDatabaseMetaDataImpl.java:565)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
at com.sun.proxy.$Proxy117.getFunctions(Unknown Source)
at sun.reflect.GeneratedMethodAccessor1559.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.execution.rmi.RemoteUtil.invokeRemote(RemoteUtil.java:179)
at com.intellij.execution.rmi.RemoteUtil.access$300(RemoteUtil.java:39)
at com.intellij.execution.rmi.RemoteUtil$RemoteInvocationHandler.invoke(RemoteUtil.java:275)
at com.sun.proxy.$Proxy118.getFunctions(Unknown Source)
at com.intellij.database.introspection.jdbcMetadataWrappers.DatabaseMetaDataWrapper.getFunctionsInner(DatabaseMetaDataWrapper.java:375)
at com.intellij.database.introspection.jdbcMetadataWrappers.DatabaseMetaDataWrapper.lambda$routines$1(DatabaseMetaDataWrapper.java:356)
at com.intellij.database.introspection.jdbcMetadataWrappers.DatabaseMetaDataWrapper.lambda$handleExceptions$2(DatabaseMetaDataWrapper.java:399)
at com.intellij.database.introspection.jdbcMetadataWrappers.ClosableIt$3.nextIt(ClosableIt.java:88)
at com.intellij.database.introspection.jdbcMetadataWrappers.ClosableIt$LazyAppendableClosableIt.nextImpl(ClosableIt.java:72)
at com.intellij.util.containers.JBIterator.peekNext(JBIterator.java:150)
at com.intellij.util.containers.JBIterator.hasNext(JBIterator.java:110)
at com.intellij.database.introspection.GenericIntrospector.introspectRoutinesInSchema(GenericIntrospector.java:417)
at com.intellij.database.introspection.GenericIntrospector.lambda$introspectSchemasByCatalogs$7(GenericIntrospector.java:213)
at java.lang.Iterable.forEach(Iterable.java:75)
at com.intellij.database.introspection.GenericIntrospector.lambda$forEachSchemaInCatalog$15(GenericIntrospector.java:249)
at com.intellij.database.introspection.GenericIntrospector.forEachCatalog(GenericIntrospector.java:244)
at com.intellij.database.introspection.GenericIntrospector.forEachSchemaInCatalog(GenericIntrospector.java:249)
at com.intellij.database.introspection.GenericIntrospector.introspectSchemasByCatalogs(GenericIntrospector.java:211)
at com.intellij.database.introspection.GenericIntrospector.introspectSchemas(GenericIntrospector.java:200)
at com.intellij.database.introspection.GenericIntrospector.lambda$null$0(GenericIntrospector.java:157)
at org.jetbrains.dekaf.core.BaseFacade.inSession(BaseFacade.java:125)
at com.intellij.database.introspection.GenericIntrospector.lambda$introspectAuto$1(GenericIntrospector.java:151)
at com.intellij.database.model.impl.BaseModel.modify(BaseModel.java:115)
at com.intellij.database.model.impl.BaseModel.modify(BaseModel.java:100)
at com.intellij.database.model.impl.BaseModel.modify(BaseModel.java:85)
at com.intellij.database.introspection.GenericIntrospector.introspectAuto(GenericIntrospector.java:151)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.introspectDatabases(DatabaseModelLoader.java:419)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$null$2(DatabaseModelLoader.java:312)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.withFacade(DatabaseModelLoader.java:533)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$introspect$3(DatabaseModelLoader.java:294)
at com.intellij.database.dataSource.DataSourceSyncManager.lambda$null$0(DataSourceSyncManager.java:41)
at com.intellij.database.dataSource.DatabaseConnectionManager$Executor.perform(DatabaseConnectionManager.java:363)
at com.intellij.database.dataSource.DatabaseConnectionManager$Executor.lambda$sync$2(DatabaseConnectionManager.java:302)
at com.intellij.database.dataSource.AsyncUtil.withAsyncFriendly(AsyncUtil.java:161)
at com.intellij.database.dataSource.DatabaseConnectionManager$Executor.sync(DatabaseConnectionManager.java:298)
at com.intellij.database.dataSource.DatabaseConnectionManager$Builder.sync(DatabaseConnectionManager.java:112)
at com.intellij.database.dataSource.DataSourceSyncManager.lambda$static$1(DataSourceSyncManager.java:40)
at com.intellij.database.dataSource.DataSourceSyncManager$SyncProcessor$1.perform(DataSourceSyncManager.java:262)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.introspect(DatabaseModelLoader.java:291)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.lambda$run$0(DatabaseModelLoader.java:271)
at com.intellij.database.dataSource.LocalDataSource.performBatch(LocalDataSource.java:1192)
at com.intellij.database.dataSource.DatabaseModelLoader$IntrospectionSession.run(DatabaseModelLoader.java:269)
at com.intellij.database.dataSource.DataSourceSyncManager$SyncProcessor.performSync(DataSourceSyncManager.java:264)
at com.intellij.database.dataSource.AsyncUtil.lambda$null$6(AsyncUtil.java:58)
at com.intellij.database.dataSource.AsyncUtil.lambda$underProgress$14(AsyncUtil.java:130)
at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:580)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:525)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:85)
at com.intellij.database.dataSource.AsyncUtil.underProgress(AsyncUtil.java:136)
at com.intellij.database.dataSource.AsyncUtil.underProgress(AsyncUtil.java:130)
at com.intellij.database.dataSource.AsyncUtil.lambda$captureIndicator$7(AsyncUtil.java:58)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:305)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.thrift.protocol.TProtocolException: Required field 'functionName' is unset! Struct:TGetFunctionsReq(sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:72 EB 62 58 4A E0 4B 73 B2 9E 73 18 36 2D 58 E1, secret:D3 CA 36 0B 05 16 43 9D 8C 46 65 1F 84 0C 84 8A)), catalogName:, schemaName:default, functionName:null)
at org.apache.hive.service.cli.thrift.TGetFunctionsReq.validate(TGetFunctionsReq.java:542)
at org.apache.hive.service.cli.thrift.TCLIService$GetFunctions_args.validate(TCLIService.java:10277)
at org.apache.hive.service.cli.thrift.TCLIService$GetFunctions_args$GetFunctions_argsStandardScheme.write(TCLIService.java:10334)
at org.apache.hive.service.cli.thrift.TCLIService$GetFunctions_args$GetFunctions_argsStandardScheme.write(TCLIService.java:10303)
at org.apache.hive.service.cli.thrift.TCLIService$GetFunctions_args.write(TCLIService.java:10254)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:71)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at org.apache.hive.service.cli.thrift.TCLIService$Client.send_GetFunctions(TCLIService.java:388)
at org.apache.hive.service.cli.thrift.TCLIService$Client.GetFunctions(TCLIService.java:380)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1443)
at com.sun.proxy.$Proxy2.GetFunctions(Unknown Source)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getFunctions(HiveDatabaseMetaData.java:328)
at com.intellij.database.remote.jdbc.impl.RemoteDatabaseMetaDataImpl.getFunctions(RemoteDatabaseMetaDataImpl.java:565)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
... 3 more
7 comments
Comment actions Permalink

Hello, I am encountering the same problem, I have also tried to add object filter "routine:.*||operator:.*||table:.*||function:.*", but it did not help. It would be very nice to find a fix a workaround. Now Datagrip does not proceed with the DB introspection and also other object types (e.g. tables) are not scanned.

Thanks in advance!

Best regards

Evgeny

1
Comment actions Permalink

Hi, I have solved the problem. It seems that Intellij/Datagrip sends wrong parameters to JDBC Driver (details also here: https://youtrack.jetbrains.com/issue/DBE-3115). So I recompiled the HiveJDBC driver with some corrections (tested on linux):

1) git clone https://git-wip-us.apache.org/repos/asf/hive.git

2) cd hive-release-1.2.1/

3) change source code: vi jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java  #see here: https://youtrack.jetbrains.com/issue/DBE-3115#focus=streamItem-27-1778089-0-0

4) mvn clean install -DskipTests -Phadoop-2

5) scp jdbc/target/hive-jdbc-1.2.1-standalone.jar my_pc:my_dir

Good luck!

Best regards

Evgeny

 

 

0
Comment actions Permalink

I submitted a bug report for this because the JDBC Spec at https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getFunctions-java.lang.String-java.lang.String-java.lang.String- requires the function name pattern to be set.

0
Comment actions Permalink

That is fixed. But you won't get any functions from driver, as this function does not work as expected. Providing schema yields empty result set, while without schema it yields functions with empty schema name and `schema.function` as function name.

0
Comment actions Permalink

Understand, I'm working on the Apache side to get that issue resolved. What is the DataGrip behavior during introspection? Does it send a schema name or not? I would expect not sending a schema to return all functions and sending a schema would only show functions in that schema, is that what you'd expect?

0
Comment actions Permalink

According to my understanding DataGrip sends requests only about the schemas that are defined by a user. But according to Alexander Hive does not respond in the correct manner. Hive should send back schema name as FUNCTION_SCHEM and and function name as FUNCTION_NAME separately.

0
Comment actions Permalink

@Sweeks We introspect schemas separately, so @Esgorbachev  is right
I've made a hack on our side to receive function list (for hive I supply no schema and then filter by schema in function name if no separate schema supplied).
I'll be glad to remove that, when that will be fixed in driver :)

1

Please sign in to leave a comment.