cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Saikrishna2
by New Contributor III
  • 931 Views
  • 0 replies
  • 2 kudos

Databricks SQL user has limitation with 10 queries ?

•Power BI is a publisher that uses AD group authentication to publish result sets. Since the publisher's credentials are maintained, the same user can access the data bricks database.•Number of the users are retrieving the data from the power bi or i...

  • 931 Views
  • 0 replies
  • 2 kudos
NSRBX
by Contributor
  • 3804 Views
  • 8 replies
  • 19 kudos

Databricks-connect not available on Databricks Runtime > 10.4

Hello Databricks Team,Databricks-connect doesn't work on databricks runtime 11.3.Databricks recommends that we use dbx for Databricks Lab instead of databricks-connect. Databricks plans no new feature development for Databricks Connect at this time.D...

  • 3804 Views
  • 8 replies
  • 19 kudos
Latest Reply
xiangzhu
Contributor III
  • 19 kudos

thx @Landan George​ do you have any ETA about the public preview ?

  • 19 kudos
7 More Replies
AnubhavG
by Contributor
  • 5657 Views
  • 8 replies
  • 29 kudos

Resolved! Are Python External UDFs supported in Databricks SQL warehouse?

I tried running a python UDF in the Databricks SQL warehouse but it did not run and gave the "Python UDF is not supported" error.Can i get a clear picture if the Python External UDFs are supported or not?

  • 5657 Views
  • 8 replies
  • 29 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 29 kudos

It's a private preview and It will be supported only in PRO SQL Warehouse and the Serverless SQL Warehouse

  • 29 kudos
7 More Replies
mr_poola49
by New Contributor III
  • 2079 Views
  • 0 replies
  • 5 kudos

Azure Databricks Jobs Connection Timeout (Read Failed)

Azure Databricks Jobs failed intermittently due to connection timeout (Read Failed) while executing a MS SQL stored procedure which is in Azure SQL database.My requirement is to process delta records(Get delta records using last refresh date) from Da...

  • 2079 Views
  • 0 replies
  • 5 kudos
640913
by New Contributor III
  • 3014 Views
  • 3 replies
  • 2 kudos

Resolved! Databricks hiding error stack trace when plotting?

Hi,I have been working with data visualisations in Databricks the last week and came across a very peculiar behaviour where it seems like Databricks is actively suppressing errors to the user. An example:```import matplotlib.pyplot as pltx = [1, 2, 3...

  • 3014 Views
  • 3 replies
  • 2 kudos
Latest Reply
LandanG
Databricks Employee
  • 2 kudos

It might be the DBR version. I ran that code on DBR 11.3 and I got the plot + error. Let me confirm if it's the DBR and get back to you here

  • 2 kudos
2 More Replies
Pat
by Honored Contributor III
  • 1275 Views
  • 0 replies
  • 14 kudos

Automating DDLs and Privilege management

How do you automate table creation and management of privilege / grants on securable objects (tables, views, etc.).I had idea to use terraform to manage above, but terraform doesn't work with managed tables - it allows to create table but then there ...

  • 1275 Views
  • 0 replies
  • 14 kudos
elgeo
by Valued Contributor II
  • 1750 Views
  • 1 replies
  • 2 kudos

Databricks - DBeaver error

Hello experts. While trying to setup the connection between DBeaver and Databricks, we receive the following error:[Databricks][DatabricksJDBCDriver](700120) Host adb-xxxxxxxxxx.azuredatabricks.net cannot be resolved through DnsResolver com.databrick...

  • 1750 Views
  • 1 replies
  • 2 kudos
Latest Reply
elgeo
Valued Contributor II
  • 2 kudos

We identified the problem. It was due to the proxy that the hostname couldn't be resolved.

  • 2 kudos
Anonymous
by Not applicable
  • 12810 Views
  • 4 replies
  • 17 kudos

Resolved! SQL query takes too long to run

Hi all,I have a set up SQL query run with 5 hours but the SQL endpoint take too long to start up with each run. Currently I don't know how to fix this Could you please help me how to improve this?

  • 12810 Views
  • 4 replies
  • 17 kudos
Latest Reply
Unforgiven
Valued Contributor III
  • 17 kudos

or refer on this : Faster SQL Queries on Delta Lake with Dynamic File Pruninghttps://www.databricks.com/blog/2020/04/30/faster-sql-queries-on-delta-lake-with-dynamic-file-pruning.html

  • 17 kudos
3 More Replies
lawrence009
by Contributor
  • 3394 Views
  • 3 replies
  • 7 kudos

Photon does not fully support the query because of dynamic pruning

Does it still make sense to run this job on a cluster with Photon enable when I am receiving the following?This is the code I ran:CREATE OR REPLACE TABLE ${tbl_name}_dups SELECT src.*, ROW_NUMBER() OVER ( PARTITION BY src.id ...

  • 3394 Views
  • 3 replies
  • 7 kudos
Latest Reply
PriyaAnanthram
Contributor III
  • 7 kudos

hmm could you show us what your query is

  • 7 kudos
2 More Replies
638555
by New Contributor III
  • 6314 Views
  • 3 replies
  • 8 kudos

Resolved! Auto termination for clusters, jobs, and delta live tables does not terminate clusters on GCP.

Hello,I am new to Databricks, and I have been trying to understand how auto termination works, but I am unsure if it is a problem based on my configuration or something else. The problem is the same in all cases, the cluster on GCP created by Databri...

  • 6314 Views
  • 3 replies
  • 8 kudos
Latest Reply
LandanG
Databricks Employee
  • 8 kudos

Hi @Tilemachos Charalampous​ ,The compute resources in your GCP account might not be the Spark clusters, rather the GKE cluster that Databricks spins up for the Databricks architecture in your account. The note in the blue highlight in docs here http...

  • 8 kudos
2 More Replies
Mado
by Valued Contributor II
  • 1324 Views
  • 0 replies
  • 0 kudos

How to change branch without using UI?

Hi, I want to change the branch in Databricks Repo. When I click on the current branch name, there is a message that: As you can see in the above image, I cannot see the list of branches to select. In fact, I selected a branch by mistake and I am not...

Change_Branch
  • 1324 Views
  • 0 replies
  • 0 kudos
xiangzhu
by Contributor III
  • 5169 Views
  • 3 replies
  • 2 kudos

Could jobs do everything delta live tables do ?

Hello,I've read the posts:Jobs - Delta Live tables difference (databricks.com)andDifference between Delta Live Tables and Multitask Jobs (databricks.com)My understanding is that delta live tables are more like a DSL that simplfies the workflow defini...

  • 5169 Views
  • 3 replies
  • 2 kudos
Latest Reply
xiangzhu
Contributor III
  • 2 kudos

@Landan George​ "Jobs won't be able to do what DLT does", I read some blogs, and watched some videos too, but I still cannot figure out the difference between jobs vs DLT. Does it mean without Databricks DLT, Databricks jobs cannot handle delta table...

  • 2 kudos
2 More Replies
apayne
by New Contributor III
  • 959 Views
  • 0 replies
  • 0 kudos

Python Databricks SQL Connector vs Databricks Connect?

Connecting several Databricks tables to a Plotly Dash application. Can't seem to find much documentation on the differences between SQL Connector and Connect. Currently using the Connect approach to read tables into pyspark dataframes, is one better ...

  • 959 Views
  • 0 replies
  • 0 kudos
Deiry
by New Contributor III
  • 1180 Views
  • 0 replies
  • 0 kudos

Why is the whole list not displayed in dbutil.widgets.multiselect?

I have been studying the Apache Spark in Databricks Academy and I don't understand why the whole list is nos displayed? Creation of widgets:dbutils.widgets.text("name", "Brickster", "Name") dbutils.widgets.multiselect("colors","orange", ["orange", "r...

image
  • 1180 Views
  • 0 replies
  • 0 kudos
Kash
by Contributor III
  • 3288 Views
  • 5 replies
  • 0 kudos

Cannot install TA-LIB via cluster libraries

Hi there,I can't seem to find a way to install Ta-lib on a databricks server. I can manually install it in the notebook using the code below but if we detach the notebook then I have to install it again. Please let me know if you've found a fix for t...

  • 3288 Views
  • 5 replies
  • 0 kudos
Latest Reply
Kash
Contributor III
  • 0 kudos

Hi Vivian,Thanks for the note. I tried that and it shows the package was installed but when I try to import talib in the notebook it says no module names 'talib'Any thoughts on what might be happening?Thanks,K

  • 0 kudos
4 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels