- 1005 Views
- 1 replies
- 0 kudos
JDBC Connection closes between 'stmt.execute( ... ) and stmt.executeQuery( ... )
I'm running a Java application that registers a CSV table with HIVE and then checks the number of rows imported. Its done in several steps.:Statement stmt = con.createStatement();....stmt.execute( "CREATE TABLE ( <definition> < > );.....ResultSet rs...
- 1005 Views
- 1 replies
- 0 kudos
- 0 kudos
@Reto Matter Are you running a jar job or using dbconnect to run java code? Please provide how are you trying to make a connection and full exception stack trace.
- 0 kudos
- 2018 Views
- 4 replies
- 2 kudos
Resolved! Making py connector to raise an error for wrong SQL when asking to plan a query
Hey all,My aim is to validate a given SQL string without actually running it.I thought I could use the `EXPLAIN` statement to do so.So I tried using the `databricks-sql-connector` for python to explain a query, and so determine whether it's valid or ...
- 2018 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Nativ Issac, We haven’t heard from you on the last response from @Hubert Dudek , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be helpful to o...
- 2 kudos
- 1944 Views
- 4 replies
- 3 kudos
Resolved! Couple of Delta Lake questions
Hey guys,We're considering Delta Lake as the storage for our project and have a couple questions. The first one is what's the pricing for Delta Lake - can't seem to find a page that says x amount costs y.The second question is more technical - if we...
- 1944 Views
- 4 replies
- 3 kudos
- 3 kudos
delta lake itself is free. It is a file format. But you will have to pay for storage and compute of course.If you want to use Databricks with delta lake, it will not be free unless you use the community edition.Depending on what you are planning to...
- 3 kudos
- 1592 Views
- 3 replies
- 1 kudos
Delta Live Tables
is DLT supported for Scala? Any reference implementations or wikis to get started?
- 1592 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Karthik Munipalle, Delta Live Tables queries can be implemented in Python or SQL.Here are few articles best explaining about DLT. Please have a look.https://docs.databricks.com/data-engineering/delta-live-tables/index.htmlhttps://databricks.com/...
- 1 kudos
- 1960 Views
- 4 replies
- 2 kudos
- 1960 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Dhaval Patel, We haven’t heard from you on the last response from @Ralph David Lagos , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to o...
- 2 kudos
- 3069 Views
- 5 replies
- 2 kudos
How to do python within sql query in Databricks ?
Can anyone show me one use case how to do python within sql query ?
- 3069 Views
- 5 replies
- 2 kudos
- 2 kudos
To run Python within a SQL query you have to first define a Python function and then register it as a UDF. Once that is done you are able to call that UDF within a SQL query. Please take a look at this documentation here:https://docs.databricks.com/s...
- 2 kudos
- 898 Views
- 2 replies
- 0 kudos
- 898 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Komal Gyanani,AQE was a major improvement added to Spark 3.0. It was added since Databricks runtime 7.3 LT (Spark 3.0) https://docs.databricks.com/release-notes/runtime/releases.html and here is docs on AQE https://docs.databricks.com/spark/late...
- 0 kudos
- 1254 Views
- 3 replies
- 4 kudos
What’s the best way to pass dependency versions dynamically to a cluster
I am using init scripts and would like to be able to control the version of a component that we release internally and frequently. We are now manually updating a dbfs requirement.txt file but I think that this problem may have been encountered befor...
- 1254 Views
- 3 replies
- 4 kudos
- 4 kudos
You can programmatically create cluster templates in JSON files and include config JSON files with libraries needed. Cluster deployment in that scenario needs to be controlled via API https://docs.databricks.com/dev-tools/api/latest/clusters.html
- 4 kudos
- 1077 Views
- 2 replies
- 2 kudos
Have you compared the performance between ADWC and Delta Lake for data warehousing?
Just want to know if you have a benchmark or some tests comparing Oracle ADWC vs Delta lake for data warehousing
- 1077 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Edelweiss Kammermann, Check these article out for a comparison.https://www.trustradius.com/compare-products/databricks-lakehouse-platform-vs-oracle-autonomous-data-warehouse
- 2 kudos
- 1839 Views
- 2 replies
- 0 kudos
Resolved! How do I create an athena table (instead of hive table) in databricks?
My dashboard uses Athena as data source for its availability (I don't need to fire up the cluster and manually refresh the data), but it requires me to create the tables manually. Wondering if there is a similar method like the .saveAsTable() to crea...
- 1839 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Howard Zhang, Here's a fantastic article for your use case. Please have a read.
- 0 kudos
- 1114 Views
- 2 replies
- 2 kudos
- 1114 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @xiaosong n, Unity Catalog requires a Databricks account on the Premium plan. In this guide, learn all about it.
- 2 kudos
- 2384 Views
- 3 replies
- 2 kudos
How to send metrics from GCP Databricks to Grafana Cloud through Prometheus?
While connecting the Databricks and Grafana, I have gone through the following approach.Install Grafna Agent in Databrics Clusters from Databricks console --> Not working since the system is not booted with systemd as init systemSince Spark 3 has Pro...
- 2384 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Avinash Goje, We haven’t heard from you on the last response from @Hubert Dudek , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be helpful to o...
- 2 kudos
- 986 Views
- 2 replies
- 0 kudos
missing notebook from workshop
Hi,I was going through this sessionhttps://tinyurl.com/databrickshcarebut on slides there is link to notebook which is broken. can you guys fix and share the link so I could try these notebooks ?this is mentioned in the slides for notebook linkhttps:...
- 986 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Muhammad Moazzam, Thank you for reaching out!Let us look into this for you, and we'll check back with an update.
- 0 kudos
- 4288 Views
- 4 replies
- 2 kudos
Can we use a variable to mention the path in the %run command
To compile the Python scripts in Azure notebooks, we are using the magic command %run.The first parameter for this command is the notebook path, is it possible to mention that path in a variable (we have to construct this path dynamically during the ...
- 4288 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Thushar R , We haven’t heard from you on the last response from @Akash Bhat , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be helpful to oth...
- 2 kudos
- 2500 Views
- 3 replies
- 0 kudos
How to migrate DBFS from one tenant to another tenant
I am working on Databricks workspace migration, where I need to copy the Databricks workspace including DBFS from source to target (both source and target are in different subscription/account). Can someone suggest what could be approach to migrate D...
- 2500 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Aquib Javeed, We haven’t heard from you on the last response from me, and I was checking back to see if my suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others.
- 0 kudos