Databricks Community

pass a tuple as parameter to sql query

at_lst = ['131','132','133'] at_tup = (*at_lst,) print(at_tup) ('131','132','133')<div> <div><span>In my sql query, i am trying to pass this on a parameter, however, it doesn't work. <div> <div><div><div><span>%sql<br /><div><span>select * from ma...

Ingesting PowerBI Tables to databricks

Hi Community,I am looking for a way to access the Power BI tables from databricks and import them as a spark dataframe into my databricks notebook.As far as I have seen, there is a Power BI connector to load data from databricks into Power BI but not...

Changing StreamingWrite API in DBR 13.1 may lead to incompatibility with Spark 3.4

I'm using StarRocks Connector[2] to ingest data to StarRocks on DataBricks 13.1 (powered by Spark 3.4.0). The connector could run on community Spark 3.4, but fail on the DBR. The reason is (the full stack trace is attached)java.lang.IncompatibleClass...

Facing UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE

[UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during parsing: [<field_name>], which can be fixed by an automatic retry: trueI am using Azure Databricks, and write with python code. Want to catch the error and raise. Tried wi...

How do you create relationships between tables in a database?

What is normalization and why is it important in database design?

What is a relational database management system (RDBMS)?

Data categories for databases

Is there a way to automate data categorisation with OpenAI API?

Permissions on Delta Live Table (DLT) pipelines

I have a large collection - and growing daily - of DLT pipelines and I need to grant access to non-admin users. Do I need to assign permissions on each individual DLT pipeline or is there a better approach?

How do I regulate notebook cache

I am experiencing error of over caching in databricks notebook. If i display different dfs one of the dfs get cache which after the result of others afterwards. Please how can I avoid the cache memory while using the notebook?

Problem starting cluster

I try to start cluster that i used to start it 7 times before and it gave me this error Cloud provider is undergoing a transient resource throttling. This is retryable. 1 out of 2 pods scheduled. Failed to launch cluster in kubernetes in 1800 seconds...

Fail start cluster

I try to start cluster that i used to start it 7 times before and it gave me this error Cloud provider is undergoing a transient resource throttling. This is retryable. 1 out of 2 pods scheduled. Failed to launch cluster in kubernetes in 1800 seconds...

inegstion time clustering

Hello, in rerence to https://www.databricks.com/blog/2022/11/18/introducing-ingestion-time-clustering-dbr-112.htmlI have a silly question how to use it. So let's assume that I have a few TB of not partitioned data. So, if I would like to query on dat...

Databricks Terraform Cluster Issue.

Error: default auth: cannot configure default credentials. Config: token=***. Env: DATABRICKS_TOKENon cluster.tf line 27, in data “databricks_spark_version” “latest_lts”:27: data “databricks_spark_version” “latest_lts” {