Refresh On BigQuery tables is not working
Using DataGrip 2022.2.1
With the Java BQ Driver: DBMS: Google BigQuery (ver. 2.0) Case sensitivity: plain=mixed, delimited=exact Driver: SimbaJDBCDriverforGoogleBigQuery (ver. 01.02.25.1029, JDBC4.2) Effective version: BigQuery (ver. 1.2.25.1029) Connection successful (45 sec, 697 ms)
When a new table is added to a BQ data set it doesn't show up in DataGrip. We have tried:
- right click refresh on the data source in Data Grip
- close / open DataGrip
- File/Invalidate Caches
- Running a select agains the new BQ table (results return, but still no table listed in the database explorer
One item does cause a refresh and that is when we add a new table via DataGrip, we get a refresh and can see all of the tables in the BQ Dataset.
I have also cycled through all of the JDBC driver versions that are available and they all have the same issue. This makes me think it is not a driver issue, but maybe a cache issue from the datagrip app.
Any suggestions are appreciated.
Please sign in to leave a comment.
Hi folks! During this release cycle I've fix a number of problems. Now it's very difficult to understand which of your problems are old and already fixed and which are new. Especially because there are so many variables that can affect connection e.g. whether you connect to the same project that is set in url, where your datasets are located, what permissions you have.
Right now I can only suggest you to set port in url as described in this comment and if it doesn't help please wait for 2022.3 release (2-3 weeks from now).
If your problem will not be fixed after 2022.3 it'll be better if you create an issue via this link (you may also comment here and post a link to your issue so other people see it)
https://youtrack.jetbrains.com/newIssue?project=DBE&c=Assignee%20Liudmila.Kornilova&c=DBMS%20BigQuery&c=Subsystem%20Connectivity
Hey Andy,
Please right-click on data source name in Database Explorer, select Diagnostics | Diagnostic Refresh and upload a file to https://uploads.jetbrains.com/, then provide its ID.
Hello Daniil: Thank you for the help on this.
Upload id: 2022_08_12_p9cnET27nRYrjp7u3zNZYd (file: introspector.0.0.log)
-Andy
Andy Powell,
Make sure, you create table in dataset/project in the same location/region via Web and via DG. You can try to create any table via DataGrip and see where it was created.
Also, verify user, you're connecting with, has rights to see tables. Since there different way you connect to Web UI and JDBC.
Have you specified any data set in Default dataset?
Hey vasily chernov we have performed these steps in the same dataset project, we have a default dataset, we are connecting with Service Accounts via DataGrip and the SA has the same roles as the user accounts in the BQ Project. After speaking with the team more, this is a recent issue after upgrading the application version. Do you see new tables appear in the DG Database explorer without refreshing or after refreshing? There is not a consistent way for us to update the tables visible via the database explorer. Also happy to hop on a shared desktop session if that resolves this quicker.
Hello Daniil do you have any feedback on this? Is there a way to install the previous version or 2 versions back to see if the issue is present in those versions?
Andy Powell, we're in the process of reproducing that issue.
You can download previous versions from https://www.jetbrains.com/datagrip/download/other.html and install it separately from the current one.
Andy Powell, can you reproduce it on 2022.1.5 version?
I think we need to go back to 2021.3 to be where we were before the upgrades. I am going to work on testing it later today. I will let you know what I observe.
Andy Powell, sorry for the ping. Any updates?
Hey Daniil sorry for the delay. I am getting the expected refresh results from DataGrip 2021.3.4
After initially setting up a new data connection for BigQuery it did not list all of the tables or even show there were tables in all of the datasets. When I right-click and refresh on the connection, it does display the tables as expected.
Let me know if you would like any further details.
Thank you,
-Andy
Any progress here? I have the same issue on 2022.2.2.
FWIW, I just ran a CREATE TABLE statement from Datagrip and it triggered some kind of refresh where now it's showing all the new tables. So, a manually triggered refresh seems to be different from whatever Datagrip does after running DDL statements. Hopefully this helps nail down how to fix the bug.
We've got some issue with it at the moment:
Is the Port value set for your data source? I had the same issue with no tables appearing for any schema and then realized I had forgot to set the port to 443 for my BigQuery connection. Setting that fixed the issue and I can now see the tables.
Andy Powell, Erik
Could you please open BigQuery connection settings, set port 443 there and also make sure that Project ID is mentioned in URL like this:
jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<my-project-id>;Does it help?
I can confirm this same problem. Refreshing the schema does not show all tables (only those added before the broken update) and creating a new datasource does not populate the schema. However, running a DDL script to create a temporary table does refresh the schema.
Also regarding vasily's question, setting port and project-id explicitly did not yield any results.
For all who stumble across this. I have downgraded our versions to 2021.3.4 and the refresh is working.
You can download previous versions from https://www.jetbrains.com/datagrip/download/other.html
Once there is an update, I will try this again, but for now the lack of visibility into the tables from the DB browser is not a working option. Thank you.
Currently I have a problem with Datagrip that it does not see the latest added table in BQ (I did not add any table using DataGrip, it s just that new tables have been added outside of DG). The schema does not refresh and the new tables are not detected in Command lines, but if I run a query calling those unseen tables the query runs fine.
With my pycharm/DataBase I don t have the same issue. So I don t think it s a service account connection issue. I have no clue of what to do.
new problem: I click refresh on Bigquery and the refresh last forever. It never ends. Also the table don t show up. Should I install a previsou vesion of DataGrip, like andy Powell advises it? I work on ubuntu. How do I install previous version of Datagrip on ubuntu? I installed the my current Datagrip using the Jetbrains toolbox
Ok I installed the version 2021.3.4, I hve the same problem: BQ refresh never ends up and the tables don t show up. Only the schema show up but not the tables. Very disappointed with that product. I was using the Databse of pycharm and went to Datagrip cause I liked the Database tool and my firt experience with it is a bug that I can t fix and that make me waste hours.
Ok I ve let this time the neverending wheel of refreshing runs this time (I had stopped the DG in my previous trial) and the tables are starting to appear. Ok the new tables are appearing. So with the old version the bug is not there.
I just need to cross my fingers this bug does not appear again, since I have no clue of why it happened and why it disappeared
Our prior workaround now no longer works. Even manually adding schemas is fruitless.
We are building new processes using other tools and have switched away from PyCharm and DataGrip to avoid the lost days this bug continues to cause. If it ever gets resolved, we are subscribed to this thread.
Good luck.
Hey JB Team. I recommended DataGrip to my customer and they ending buying licenses, now we are closing in on 90 days with out a fix for this and I feel like I have some egg on my face recommending this since we are just playing work arounds with the IDE. I know you have a really good team of developers and that they should be able to understand the changes that caused this error, can we get it prioritized please?
Have BigQuery issues been resolved yet?
I'm using 2022.3.2. I can see my projects, I can even see the datasets within them, but I can't select them or do anything else with them.
Yes, it is working for me. This deserves a big thank you to the JB team for fixing. We've been able to transition back our data science team back to PyCharm, which is great.
Still doesn't appear to be working in DataGrip. Only thing I can think it might be is the default BigQuery location, but I can't see where that is set
Hi Scott,
We may assume this has a different cause if upgrading to DataGrip 2022.3.3 didn't solve the abovementioned behavior. If so, please open a ticket so we can review your incident individually:
https://youtrack.jetbrains.com/newIssue?project=DBE&c=Assignee%20Liudmila.Kornilova&c=DBMS%20BigQuery&c=Subsystem%20Connectivity