- 1576 Views
- 2 replies
- 2 kudos
Autoloader (GCP) Custom PubSub Queue
I want to know if what I describe below is possible with AutoLoader in the Google Cloud Platform.Problem Description:We have GCS buckets for every client/account. Inside these buckets is a path/blob for each client's instances of our platform. A clie...
- 1576 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @Ryan Ebanks​ Please let us know if more help is needed on this.
- 2 kudos
- 8210 Views
- 6 replies
- 3 kudos
Resolved! How to load a json file in pyspark with colon character in file name
Hi,I'm trying to load this json file which contains the colon character in its name: file_name.2022-03-05_11:30:00.json but I get the error in screenshot below saying that there is a relative path in an absolute url - Any idea how to read this file...
- 8210 Views
- 6 replies
- 3 kudos
- 3 kudos
Hi @Laura Blancarte​ I hope that @Pearl Ubaru​'s answer would have helped you in resolving your issue.Please let us know if you need more help on this.
- 3 kudos
- 2481 Views
- 2 replies
- 2 kudos
How can we connect to the databricks managed metastore
Hi, I am trying to take advantage of the treasure trove of the information that metastore contains and take some actions to improve performance. In my case, the metastore is managed by databricks, we don't use external metastore.How can I connect to ...
- 2481 Views
- 2 replies
- 2 kudos
- 2 kudos
@AKSHAY PALLERLA​ to get the jdbc/odbc information you can get it from the cluster configuration. In the cluster configuration page, under advanced options, you have JDBC/ODBC tab. Click on that tab and it should give you the details you are looking ...
- 2 kudos
- 5122 Views
- 6 replies
- 5 kudos
Resolved! Databricks runs cell, but stops output and hangs afterwards.
tl;dr: A cell that executes purely on the head node stops printed output during execution, but output still shows up in the cluster logs. After execution of the cell, Databricks does not notice the cell is finished and gets stuck. When trying to canc...
- 5122 Views
- 6 replies
- 5 kudos
- 5 kudos
As that library work on pandas problem can be that it doesn't support pandas on spark. On the local version, you probably use non-distributed pandas. You can check behavior by switching between:import pandas as pd import pyspark.pandas as pd
- 5 kudos
- 2028 Views
- 1 replies
- 1 kudos
Resolved! Mounting of S3 bucket via Terraform is frequently timing out
Summary of the problemWhen mounting an S3 bucket via Terraform the creation process is frequently timing out (running beyond 10 minutes). When I check the Log4j logs in the GP cluster I see the following error message repeated:```22/07/26 05:54:43 ER...
- 2028 Views
- 1 replies
- 1 kudos
- 1 kudos
Solved. See here: https://github.com/databricks/terraform-provider-databricks/issues/1500
- 1 kudos
- 1105 Views
- 1 replies
- 0 kudos
JDBC Connection closes between 'stmt.execute( ... ) and stmt.executeQuery( ... )
I'm running a Java application that registers a CSV table with HIVE and then checks the number of rows imported. Its done in several steps.:Statement stmt = con.createStatement();....stmt.execute( "CREATE TABLE ( <definition> < > );.....ResultSet rs...
- 1105 Views
- 1 replies
- 0 kudos
- 0 kudos
@Reto Matter​ Are you running a jar job or using dbconnect to run java code? Please provide how are you trying to make a connection and full exception stack trace.
- 0 kudos
- 2271 Views
- 3 replies
- 2 kudos
Resolved! Making py connector to raise an error for wrong SQL when asking to plan a query
Hey all,My aim is to validate a given SQL string without actually running it.I thought I could use the `EXPLAIN` statement to do so.So I tried using the `databricks-sql-connector` for python to explain a query, and so determine whether it's valid or ...
- 2271 Views
- 3 replies
- 2 kudos
- 2 kudos
Please try in pyspark:try: spark.sql("SELECT BAD-QUERY AS FOO")._jdf.queryExecution().toString() except: print("incorrect query")or just:try: spark.sql("SELECT BAD-QUERY AS FOO").explain() except: print("incorrect query")
- 2 kudos
- 2288 Views
- 4 replies
- 3 kudos
Resolved! Couple of Delta Lake questions
Hey guys,We're considering Delta Lake as the storage for our project and have a couple questions. The first one is what's the pricing for Delta Lake - can't seem to find a page that says x amount costs y.The second question is more technical - if we...
- 2288 Views
- 4 replies
- 3 kudos
- 3 kudos
delta lake itself is free. It is a file format. But you will have to pay for storage and compute of course.If you want to use Databricks with delta lake, it will not be free unless you use the community edition.Depending on what you are planning to...
- 3 kudos
- 1688 Views
- 2 replies
- 0 kudos
Delta Live Tables
is DLT supported for Scala? Any reference implementations or wikis to get started?
- 1688 Views
- 2 replies
- 0 kudos
- 0 kudos
is DLT supported for Scala? Any reference implementations or wikis to get started?
- 0 kudos
- 2125 Views
- 3 replies
- 1 kudos
- 2125 Views
- 3 replies
- 1 kudos
- 1 kudos
Try looking into the structured streaming API. There you will learn about how to join streams and static data, how to set triggers for the streams, minibatching and other things that are important to the reliability of your application.Structured Str...
- 1 kudos
- 3539 Views
- 5 replies
- 2 kudos
How to do python within sql query in Databricks ?
Can anyone show me one use case how to do python within sql query ?
- 3539 Views
- 5 replies
- 2 kudos
- 2 kudos
To run Python within a SQL query you have to first define a Python function and then register it as a UDF. Once that is done you are able to call that UDF within a SQL query. Please take a look at this documentation here:https://docs.databricks.com/s...
- 2 kudos
- 1010 Views
- 2 replies
- 0 kudos
- 1010 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Komal Gyanani​,AQE was a major improvement added to Spark 3.0. It was added since Databricks runtime 7.3 LT (Spark 3.0) https://docs.databricks.com/release-notes/runtime/releases.html and here is docs on AQE https://docs.databricks.com/spark/late...
- 0 kudos
- 1392 Views
- 2 replies
- 3 kudos
What’s the best way to pass dependency versions dynamically to a cluster
I am using init scripts and would like to be able to control the version of a component that we release internally and frequently. We are now manually updating a dbfs requirement.txt file but I think that this problem may have been encountered befor...
- 1392 Views
- 2 replies
- 3 kudos
- 3 kudos
You can programmatically create cluster templates in JSON files and include config JSON files with libraries needed. Cluster deployment in that scenario needs to be controlled via API https://docs.databricks.com/dev-tools/api/latest/clusters.html
- 3 kudos
- 1179 Views
- 1 replies
- 2 kudos
Have you compared the performance between ADWC and Delta Lake for data warehousing?
Just want to know if you have a benchmark or some tests comparing Oracle ADWC vs Delta lake for data warehousing
- 1179 Views
- 1 replies
- 2 kudos
- 2 kudos
Just want to know if you have a benchmark or some tests comparing Oracle ADWC vs Delta lake for data warehousing
- 2 kudos
- 2014 Views
- 1 replies
- 0 kudos
How do I create an athena table (instead of hive table) in databricks?
My dashboard uses Athena as data source for its availability (I don't need to fire up the cluster and manually refresh the data), but it requires me to create the tables manually. Wondering if there is a similar method like the .saveAsTable() to crea...
- 2014 Views
- 1 replies
- 0 kudos
- 0 kudos
My dashboard uses Athena as data source for its availability (I don't need to fire up the cluster and manually refresh the data), but it requires me to create the tables manually. Wondering if there is a similar method like the .saveAsTable() to crea...
- 0 kudos