- 1209 Views
- 10 replies
- 0 kudos
Translating SQL Value Function For XML To Databricks SQL
Trying to translate this line of a SQL query that evaluates XML to Databricks SQL.SELECT MyColumn.value('(/XMLData/Values/ValueDefinition[@colID="10"]/@Value)[1]', 'VARCHAR(max)') as Color The XML looks like this:<XMLData><Values><ValueDefinition c...
- 1209 Views
- 10 replies
- 0 kudos
- 0 kudos
Yes, now they support XML parse directly in databricks 14.3 or higher, else earlier you could have leveraged spark xml library jars to parse it.You can still leverage xpath in case where one of data column hold XML value in a dataset. As @BNV is look...
- 0 kudos
- 548 Views
- 1 replies
- 1 kudos
Resolved! How can I dedupe from a table created from a Kinesis change data capture feed.
Here I have a table named organizations_silver that was build from a bronze table created from a Kinesis change data capture feed.@dlt.table(name="kinesis_raw_stream", table_properties={"pipelines.reset.allowed": "false"})def kinesis_raw_stream(): ...
- 548 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @aliacovella, Looks like there are duplicate records in your source table that match the same target record. This is indeed the case since your source table, organizations_silver, contains duplicates due to the append-only nature of the Kinesis...
- 1 kudos
- 5714 Views
- 3 replies
- 4 kudos
Resolved! Spot instance in Azure Databricks
When I run a job enabling using spot instances , I would like to know how many number of workers are using spot and how many number of workers are using on demand instances for a given job run In order to identify the spot instances we got for any...
- 5714 Views
- 3 replies
- 4 kudos
- 7298 Views
- 2 replies
- 0 kudos
Create external table using multiple paths/locations
I want to create an external table from more than a single path. I have configured my storage creds and added an external location, and I can successfully create a table using the following code;create table test.base.Example using csv options ( h...
- 7298 Views
- 2 replies
- 0 kudos
- 0 kudos
You do not have to create all the partition folders yourslef. You just need to specify the parent folder like CREATE OR REPLACE TABLE <catalog>.<schema>.<table-name> USING <format> PARTITIONED BY (<partition-column-list>) LOCATION 's3://<bucket-path...
- 0 kudos
- 3864 Views
- 5 replies
- 0 kudos
permission denied listing external volume when using vscode databricks extension
hey, i'm using the Db extension for vscode (Databricks connect v2). When using dbutils to list an external volume defined in UC like so: dbutils.fs.ls("/Volumes/dev/bronze/rawdatafiles/") i get this error: "databricks.sdk.errors.mapping.PermissionD...
- 3864 Views
- 5 replies
- 0 kudos
- 0 kudos
great, thanks for confirming. This feature was under development in the early quarter last year. Now it is available.
- 0 kudos
- 4638 Views
- 5 replies
- 5 kudos
Databricks Academy webpage showing insecure connection (in Chrome)
When I was trying to visit the Databricks Academy website https://customer-academy.databricks.com, it showed insecure connection as below.This happened at 8 January 2023 (AEDT) around 12:30pm.
- 4638 Views
- 5 replies
- 5 kudos
- 17683 Views
- 13 replies
- 5 kudos
Is it possible to connect QGIS to Databricks using ODBC?
I can connect ArcGIS to Databricks using ODBC, but using the same ODBC DSN for QGIS I get an error - Unable to initialize ODBC connection to DSNHas anyone got this working?
- 17683 Views
- 13 replies
- 5 kudos
- 5 kudos
That should probably help, but I tried and my table has 0 rows when the same table loaded with all the schema analyzed has 835...Still have testing to do, but with that, you can now choose a single file to add using the connection stringODBC:token/yo...
- 5 kudos
- 1198 Views
- 2 replies
- 1 kudos
Install notebook dependency via terraform for serverless notebook tasks
I am trying to install a wheel file as a dependency for a serverless notebook task via terraform. According to https://docs.databricks.com/en/compute/serverless/dependencies.html , dependencies in serverless notebooks can be configured via the base e...
- 1198 Views
- 2 replies
- 1 kudos
- 1 kudos
Currently, the databricks_job resource in Terraform does not support configuring the environment for notebook tasks directly. You can upload the YAML file and configure the environment as mentioned in https://docs.databricks.com/en/compute/serverless...
- 1 kudos
- 1965 Views
- 10 replies
- 0 kudos
Which process is eating up my driver memory?
Hi,We're running DBR 14.3 on a shared multi-node cluster.When checking the metrics of the driver, I see that the Memory utilization and Memory swap utilization are increasing a lot and are almost never decreasing. Even if no processes are running any...
- 1965 Views
- 10 replies
- 0 kudos
- 0 kudos
On OS level you will not see notebooks, you will see the mem consumption of the spark application (so this is all notebooks).For that there is the spark ui.I'd look for collect(), broadcast() statements. Python code outside of spark, tons of graphics...
- 0 kudos
- 2030 Views
- 3 replies
- 0 kudos
Resolved! Problem with installing Python WHEEL in an existed cluster
Hi community,I was running a workflow based on different tasks but also taking into account the existed cluster to execute those tasks, but I was getting error in configurations: run failed with error message Library installation failed for library d...
- 2030 Views
- 3 replies
- 0 kudos
- 0 kudos
You can do it by following steps in https://docs.databricks.com/en/compute/serverless/dependencies.html
- 0 kudos
- 7326 Views
- 5 replies
- 5 kudos
Support of running multiple cells at a time in databricks notebook Hi all,Now databricks notebook supports parallel run of commands in a single notebo...
Support of running multiple cells at a time in databricks notebookHi all,Now databricks notebook supports parallel run of commands in a single notebook that will help run ad hoc queries simultaneously without creating a separate notebook.Once you run...
- 7326 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi Team,I am observing that the functionality is not working as expected in the Trial workspace of Databricks. Is there a setting that needs to be enabled to allow independent SQL cells in a Databricks notebook to run in parallel, while dependent cel...
- 5 kudos
- 1446 Views
- 4 replies
- 0 kudos
Resolved! Integrating PySpark DataFrame into SQL Dashboard for Enhanced Visualization
I have created a DataFrame in a notebook using PySpark and am considering creating a fully-featured dashboard in SQL. My question is whether I need to first store the DataFrame as a table in order to use it in the dashboard, or if it's possible to di...
- 1446 Views
- 4 replies
- 0 kudos
- 0 kudos
Sorry, I vaugely remember we used to create persistent views on dataframe earlier.Currently, spark dataframe doesn't allow you to create pesistent view on dataframe, rather you have to create table to use it in SQL warehouse.# Assuming there is an ex...
- 0 kudos
- 874 Views
- 3 replies
- 1 kudos
Requirement to remove/skip column(s) in the downstream tables/views while PII data masking
Hi there,As a compliance measure, I'm tasked with masking the PII data starting from bronze to silver and all the tables and views downstream. I suggested my clients to use row filters and column masks as mentioned in the doc.However, when a user who...
- 874 Views
- 3 replies
- 1 kudos
- 1 kudos
You are right, on this case we might need to open a feature request through https://docs.databricks.com/en/resources/ideas.html#ideas
- 1 kudos
- 4992 Views
- 1 replies
- 0 kudos
[SQL_CONF_NOT_FOUND] The SQL config "/Volumes/xxx...." canot be found. Please verify that the confi
I am getting the below error when trying to stream data from Azure Storage path to a Delta Live Table ([PATH] is the path to my files which I have redacted here):[SQL_CONF_NOT_FOUND] The SQL config "/Volumes/[PATH]" cannot be found. Please verify tha...
- 4992 Views
- 1 replies
- 0 kudos
- 0 kudos
I believe you are not setting spark.conf.set("/Volumes/[PATH]", "your_actual_path_here") hence when you try to get the conf, it fails. data_source_path = spark.conf.get("/Volumes/[PATH]") "/Volumes/[PATH]" becomes the conf name, you would not want ...
- 0 kudos
- 5284 Views
- 1 replies
- 0 kudos
Azure Databricks: Cannot create volumes or tables
If I try to create a Volume, I get this error:Failed to access cloud storage: AbfsRestOperationException exceptionTraceId=fa207c57-db1a-406e-926f-4a7ff0e4afddWhen i try to create a table, I get this error:Error creating table[RequestId=4b8fedcf-24b3-...
- 5284 Views
- 1 replies
- 0 kudos
- 0 kudos
It seems like you are encountering issues with accessing cloud storage while trying to create a volume and a table in Databricks on Azure. The errors you are seeing, AbfsRestOperationException and INVALID_STATE.UC_CLOUD_STORAGE_ACCESS_FAILURE, indica...
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
