Databricks Community

GCP hosted Databricks - DBFS temp files - Not Found

I've been working on obtaining DDL at the schema level in Hive Metastore within GCP-hosted Databricks. I've implemented a Python code that generates SQL files in the dbfs/temp directory. However, when running the code, I'm encountering a "file path n...

Unable to find permission button in Sql Warehouse for providing Access

Hi Everyone, am unable to see the permission button in sql warehouse to provide access to other users.I have admin rights and databricks is premium subscription.  

SQL Serverless - cost view

Hi,Anyone knows how I'm able to monitor cost of the SQL Serverless? I'm using Databricks in Azure and I'm not sure where to find cost generated by compute resources hosted on Databricks.

Error while creating external table in unity catalog

I am trying to create an external table using csv file which is stored in ADLS gen2 My account owner has created a storage credential and an external location I am a databricks user who all privileges on external location when trying to create a tabl...

Issue with Databricks notebooks and 'requests' python library. Incosistent output.

I have a strange issue with Databricks notebooks and Google Colab notebooks, where I cannot get the results from requests library that would be consistent to what I get on my local computer. Dir Surveys (wyo.gov) ("http://pipeline.wyo.gov/r_Direction...

REST API

Creating an application to capture cluster metrics and sending HTTP REST request to the Spark History Server's API endpoint to retrieve a list of applications. This request doesn't generate logs in the Spark History Server's log files. The Spark Hist...

Monitoring job metrics

Hi,We need to monitor Databricks jobs and we have made a setup where are able to get the prometheus metrics, however, we are lagging an overview of which metrics refer to what.Namely, we need to monitor the following:failed jobs : is a job failedtabl...

ListBucket

{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "s3:ListBucket"      ],     "Resource": [        "arn:aws:s3:::<s3-bucket-name>"      ]    },    {      "Effect": "Allow",      "Action": [        "s3:Pu...

pass a tuple as parameter to sql query

at_lst = ['131','132','133'] at_tup = (*at_lst,) print(at_tup) ('131','132','133')<div> <div><span>In my sql query, i am trying to pass this on a parameter, however, it doesn't work. <div> <div><div><div><span>%sql<br /><div><span>select * from ma...

Ingesting PowerBI Tables to databricks

Hi Community,I am looking for a way to access the Power BI tables from databricks and import them as a spark dataframe into my databricks notebook.As far as I have seen, there is a Power BI connector to load data from databricks into Power BI but not...

Changing StreamingWrite API in DBR 13.1 may lead to incompatibility with Spark 3.4

I'm using StarRocks Connector[2] to ingest data to StarRocks on DataBricks 13.1 (powered by Spark 3.4.0). The connector could run on community Spark 3.4, but fail on the DBR. The reason is (the full stack trace is attached)java.lang.IncompatibleClass...

Facing UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE

[UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during parsing: [<field_name>], which can be fixed by an automatic retry: trueI am using Azure Databricks, and write with python code. Want to catch the error and raise. Tried wi...

How do you create relationships between tables in a database?

What is normalization and why is it important in database design?

What is a relational database management system (RDBMS)?