topic Re: GC Driver Error in Data Engineering

GC Driver Error

aschiff — Fri, 23 Sep 2022 19:43:14 GMT

I am using a cluster in databricks to connect to a Tableau workbook through the JDBC connector. My Tableau workbook has been unable to load due to resources not being available through the data connection. I went to look at the driver log for my cluster and I see Full GC (Ergonomics) errors and Full GC Allocation errors. How do I resolve this? I've tried increasing the storage of my driver and worker by changing them in my cluster but that didn't fix it.

Re: GC Driver Error

Dooley — Mon, 26 Sep 2022 21:14:58 GMT

Well you should be using the Databricks ODBC connector and not a JDBC connector for Tableau. Let me know when you make that switch if that error keeps happening.

Re: GC Driver Error

aschiff — Tue, 27 Sep 2022 16:41:56 GMT

I downloaded the ODBC connector from here: https://www.databricks.com/spark/odbc-drivers-download?_ga=2.75890453.1274928885.1664211300-1388717875.1663615888 . Now how do I use this with my cluster? I was connecting to the cluster from Tableau through the server host name and HTTP path already to the JDBC so I am just wondering how I connect to the ODBC instead of the JDBC.

Re: GC Driver Error

aschiff — Tue, 27 Sep 2022 20:05:23 GMT

I now also downloaded an ODBC manager and tried creating a User DSN in it but am not too sure where these helpers are going or how to use them. I am following the steps given here: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html#odbc-driver-guide .

Re: GC Driver Error

aschiff — Tue, 27 Sep 2022 20:28:56 GMT

Sorry for the constant updates, but I have successfully created an ODBC connector using the Simba Spark Driver. It has my databricks cluster host name, http path, port number and, for authentication, my databricks login info. Now that this is established, how do I get Tableau Online (not Tableau Desktop) to connect using this ODBC stuff and not the JDBC stuff causing the garbage collection error.

Re: GC Driver Error

Dooley — Tue, 27 Sep 2022 21:26:58 GMT

You are going to publish workbooks from Tableau Desktop to Tableau Online. These are the instructions.

Re: GC Driver Error

aschiff — Tue, 27 Sep 2022 21:34:40 GMT

I don't have Tableau Desktop, I only have access to Tableau Online.

Re: GC Driver Error

Dooley — Wed, 28 Sep 2022 19:32:24 GMT

Okay, what is your cluster configuration? Did you try a bigger cluster and had the same problem? When you ran the connector, did you look at your metrics for your cluster to see the memory usage? What do you see? Also are you using the spot or on demand cluster? What DRT version are you using? Are you using a Unity Catalog enabled cluster?

Re: GC Driver Error

aschiff — Wed, 28 Sep 2022 19:49:08 GMT

I am using a i3.xlarge worker and i3.2xlarge driver cluster. I have played around with the larger memory sizes of the cluster and still get the same issue. I don't know how to use to ODBC connector but when I was connecting before (I guess to the JDBC) using the server host name and http path in tableau, I get errors like in the picture in my cluster's driver log. And it just keeps repeating the GC Allocation Failure errors and returns a Full GC Ergonomics error at some point too. When looking at the cluster's metrics every 15 minutes it's only showing about 230 KB each time. I'm not sure which type of cluster I am using in terms of spot or on demand, but my "Instance" field in configuration is blank and through my research it looks like it would appear there. My DRT is 11.2 And I don't think it is a Unity Catalog cluster; I have not used any AWS cloud capabilities with this cluster.

Re: GC Driver Error

aschiff — Wed, 28 Sep 2022 20:36:27 GMT

Re: GC Driver Error

Dooley — Fri, 30 Sep 2022 00:49:18 GMT

One thing I would try is to have more workers with memory-optimized instances with larger memories to see if that will fixed it. I know you played with it a bit, but maybe with more workers & larger memory that might fix this. But let's take a look at the Spark UI a bit to try to troubleshoot.

(1) Under Stages of the job that ran that created the error - under the summary metrics, do you see data spill? If you do not see it there then you can see data spill in the SQL tab where you find the SQL query associated to the job number with the error and then you can click the query to see the DAG. You can click the + in the boxes to see the write out and see if you see a data spill in there.

(2) You go to Spark UI and then go to JDBC/ODBC connector, do you see data leaving?

(3) Also under stages, can you take a screenshot of what you see for this job? Can you sort by shuffle read?

(4) Do you see anything cached in storage under the "Storage" tab?

So there is a reference to PSYoungGen, I believe it didn't have enough to allocate memory for possible a large object and thus a GC was triggered by allocation failure. Did this happen multiple times in a 10s interval?

Re: GC Driver Error

aschiff — Fri, 30 Sep 2022 16:44:53 GMT

I increased the minimum amount of worker out of 8 from 2 to 7.

(1) I see this under Stages in Spark UI, doesn't look to show a data spill. Then in the SQL tab, its just blank:

This is as far as I got for now before the Spark UI tab itself is just blank and not loading anything.

Re: GC Driver Error

aschiff — Fri, 30 Sep 2022 17:13:04 GMT

Once restarting the cluster I started to see the information:

(1) Attached are what I see in the Stages and SQL tab. Doesn't look like any data spill.

(2) How would I see data leaving? I just see session statistics and SQL statistics:

(3) I also attached an image (in a word doc because it was too big to paste here) of what I see under Stages tab sorted by Shuffle Read

(4) Here is what is cached under Storage tab

And here is an image of the PSYoungGen errors where there are multiple "Times" for each error. sys (system) says 2 seconds, then real says about 50 seconds. This is a lot to understand and look at so forgive me if I am not providing you with the correct information.

Re: GC Driver Error

aschiff — Fri, 30 Sep 2022 17:13:22 GMT

Re: GC Driver Error

aschiff — Fri, 30 Sep 2022 17:13:31 GMT

Re: GC Driver Error

Dooley — Mon, 03 Oct 2022 17:33:13 GMT

Thanks for the files.

(1) So in this Databricks Shell - Details for Query file, in the graph you need to expand the blocks and see if you see any data spill in there.

(2) Can you please submit the query that you were doing that ran into this problem? How much data are you running this on? Is that data partitioned by you in a custom way?

Side note: I did see in one of the stages you are doing a count distinct and I wanted to draw your attention to a Approx Count Distinct SQL function.

Re: GC Driver Error

aschiff — Mon, 03 Oct 2022 19:41:35 GMT

I tried to get these answers for you but all the spark UI tabs are coming up blank/default again. I've tried restarting my cluster to solve the issue like before but no dice. Is it possible I can give you permission to look at my cluster?

Re: GC Driver Error

Dooley — Mon, 03 Oct 2022 22:09:42 GMT

Usually it is blank because the page is still loading. You should try to wait to see it comes up. Plus the cluster should be on and active & you just run the job to see it in Spark UI.

Re: GC Driver Error

Dooley — Mon, 03 Oct 2022 22:12:52 GMT

Also can you run the command in tableau that caused that issue and then look at the cluster to identify the job number that is associated to that effort? I can see the SQL queries in your stages tab but they are cut off, so I would like to see the full queries that are made that give you this type of error.

Re: GC Driver Error

aschiff — Tue, 04 Oct 2022 00:18:12 GMT

I thought the query was select * from default.salesforce_export_1_explorium_15sept2022 but it is a lot more. Attached is the 878 pages of word document it took to copy and paste the query. The table I am asking select * from is only 700KB. The job ids associated with this ridiculous query are 293, 294, and 295