Refresh On BigQuery tables is not working

Answered

Using DataGrip 2022.2.1

With the Java BQ Driver: DBMS: Google BigQuery (ver. 2.0) Case sensitivity: plain=mixed, delimited=exact Driver: SimbaJDBCDriverforGoogleBigQuery (ver. 01.02.25.1029, JDBC4.2) Effective version: BigQuery (ver. 1.2.25.1029)  Connection successful (45 sec, 697 ms)

When a new table is added to a BQ data set it doesn't show up in DataGrip.  We have tried:

  1. right click refresh on the data source in Data Grip
  2. close / open DataGrip
  3. File/Invalidate Caches
  4. Running a select agains the new BQ table (results return, but still no table listed in the database explorer

One item does cause a refresh and that is when we add a new table via DataGrip, we get a refresh and can see all of the tables in the BQ Dataset.

 

I have also cycled through all of the JDBC driver versions that are available and they all have the same issue.  This makes me think it is not a driver issue, but maybe a cache issue from the datagrip app.

 

Any suggestions are appreciated.

 

26 comments
Comment actions Permalink
Official comment

Hi folks! During this release cycle I've fix a number of problems. Now it's very difficult to understand which of your problems are old and already fixed and which are new. Especially because there are so many variables that can affect connection e.g. whether you connect to the same project that is set in url, where your datasets are located, what permissions you have.

Right now I can only suggest you to set port in url as described in this comment and if it doesn't help please wait for 2022.3 release (2-3 weeks from now).

If your problem will not be fixed after 2022.3 it'll be better if you create an issue via this link (you may also comment here and post a link to your issue so other people see it)

https://youtrack.jetbrains.com/newIssue?project=DBE&c=Assignee%20Liudmila.Kornilova&c=DBMS%20BigQuery&c=Subsystem%20Connectivity


Comment actions Permalink

Hey Andy,

Please right-click on data source name in Database Explorer, select Diagnostics | Diagnostic Refresh and upload a file to https://uploads.jetbrains.com/, then provide its ID.

0
Comment actions Permalink

Hello Daniil:  Thank you for the help on this. 

Upload id: 2022_08_12_p9cnET27nRYrjp7u3zNZYd (file: introspector.0.0.log)

 

-Andy

0
Comment actions Permalink

Andy Powell,

Make sure, you create table in dataset/project in the same location/region via Web and via DG.  You can try to create any table via DataGrip and see where it was created.

Also, verify user, you're connecting with, has rights to see tables. Since there different way you connect to Web UI and JDBC.

Have you specified any data set in Default dataset?

0
Comment actions Permalink

Hey vasily chernov we have performed these steps in the same dataset project, we have a default dataset, we are connecting with Service Accounts via DataGrip and the SA has the same roles as the user accounts in the BQ Project.  After speaking with the team more, this is a recent issue after upgrading the application version.  Do you see new tables appear in the DG Database explorer without refreshing or after refreshing?  There is not a consistent way for us to update the tables visible via the database explorer.  Also happy to hop on a shared desktop session if that resolves this quicker.

 

0
Comment actions Permalink

Hello Daniil Maslov do you have any feedback on this?  Is there a way to install the previous version or 2 versions back to see if the issue is present in those versions?

0
Comment actions Permalink

Andy Powell, we're in the process of reproducing that issue.

You can download previous versions from https://www.jetbrains.com/datagrip/download/other.html and install it separately from the current one.

0
Comment actions Permalink

Andy Powell, can you reproduce it on 2022.1.5 version?

0
Comment actions Permalink

I think we need to go back to 2021.3 to be where we were before the upgrades.  I am going to work on testing it later today. I will let you know what I observe.

1
Comment actions Permalink

Andy Powell, sorry for the ping. Any updates?

0
Comment actions Permalink

Hey Daniil Maslov sorry for the delay.  I am getting the expected refresh results from DataGrip 2021.3.4

After initially setting up a new data connection for BigQuery it did not list all of the tables or even show there were tables in all of the datasets.  When I right-click and refresh on the connection, it does display the tables as expected.

 

Let me know if you would like any further details.

 

Thank you,

 

-Andy

 

1
Comment actions Permalink

Any progress here? I have the same issue on 2022.2.2.

0
Comment actions Permalink

FWIW, I just ran a CREATE TABLE statement from Datagrip and it triggered some kind of refresh where now it's showing all the new tables. So, a manually triggered refresh seems to be different from whatever Datagrip does after running DDL statements. Hopefully this helps nail down how to fix the bug.

0
Comment actions Permalink

Is the Port value set for your data source? I had the same issue with no tables appearing for any schema and then realized I had forgot to set the port to 443 for my BigQuery connection. Setting that fixed the issue and I can now see the tables.

2
Comment actions Permalink

Andy Powell, Erik

Could you please open BigQuery connection settings, set port 443 there and also make sure that Project ID is mentioned in URL like this: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<my-project-id>;
Does it help?

0
Comment actions Permalink

I can confirm this same problem.  Refreshing the schema does not show all tables (only those added before the broken update) and creating a new datasource does not populate the schema.  However, running a DDL script to create a temporary table does refresh the schema.

0
Comment actions Permalink

Also regarding vasily's question, setting port and project-id explicitly did not yield any results.

1
Comment actions Permalink

For all who stumble across this.  I have downgraded our versions to 2021.3.4 and the refresh is working.

You can download previous versions from https://www.jetbrains.com/datagrip/download/other.html 

Once there is an update, I will try this again, but for now the lack of visibility into the tables from the DB browser is not a working option.  Thank you.

 

0
Comment actions Permalink

Currently I have a problem with Datagrip that it does not see the latest added table in BQ (I did not add any table using DataGrip, it s just that new tables have been added outside of DG). The schema does not refresh and the new tables are not detected in Command lines, but if I run a query calling those unseen tables the query runs fine.

With my pycharm/DataBase I don t have the same issue. So I don t think it s a service account connection issue. I have no clue of what to do. 

0
Comment actions Permalink

new problem: I click refresh on Bigquery and the refresh last forever. It never ends. Also the table don t show up. Should I install a previsou vesion of DataGrip, like andy Powell advises it? I work on ubuntu. How do I install previous version of Datagrip on ubuntu? I installed the my current Datagrip using the Jetbrains toolbox

0
Comment actions Permalink

Ok I installed the version  2021.3.4, I hve the same problem: BQ refresh  never ends up and the tables don t show up. Only the schema show up but not the tables. Very disappointed with that product. I was using the Databse of pycharm and went to Datagrip cause I liked the Database tool and my firt experience with it is a bug that I can t fix and that make me waste hours.  

0
Comment actions Permalink

Ok I ve let this time the neverending wheel of refreshing runs this time (I had stopped the DG in my previous trial) and the tables are starting to appear. Ok the new tables are appearing. So with the old version the bug is not there.

0
Comment actions Permalink

I just need to cross my fingers this bug does not appear again, since I have no clue of why it happened and why it disappeared

0
Comment actions Permalink

Our prior workaround now no longer works.  Even manually adding schemas is fruitless.

We are building new processes using other tools and have switched away from PyCharm and DataGrip to avoid the lost days this bug continues to cause.  If it ever gets resolved, we are subscribed to this thread.

Good luck.

0
Comment actions Permalink

Hey JB Team.  I recommended DataGrip to my customer and they ending buying licenses, now we are closing in on 90 days with out a fix for this and I feel like I have some egg on my face recommending this since we are just playing work arounds with the IDE.  I know you have a really good team of developers and that they should be able to understand the changes that caused this error, can we get it prioritized please?

 

 

1

Please sign in to leave a comment.