I created a table including location such as: wasb://<container>@<storageaccount>.blob.core.windows.net/foldername We have updated access to storage accounts to use abfssI am trying to execute the following command: alter table mydatabase.mytable set...
Spark supports dynamic partition overwrite for parquet tables by setting the config: spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic") before writing to a partitioned table. With delta tables is appears you need to manually specif...
@SamCallister wrote: Spark supports dynamic partition overwrite for parquet tables by setting the config:spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")before writing to a partitioned table. With delta tables is appears you need ...
Hello,What is the easiest way to do web scraping in Databricks? Let's imagine that from this link: http://automated.pythonanywhere.com , I need to grab this element "/html/body/div[1]/div/h1[1]" and return a text, how can I do it? Can somebody write ...
I realize that the best practice would be to integrate our service with Kafka as a streaming source for Databricks, but given that the service already stores data into TimescaleDB, how can I stream data from TimescaleDB into DBX? Debezium doesn't wor...
https://github.com/noctarius/timescaledb-event-streamer/Might help.
Hi team,I have a requirement to get the metadata of tables available in databricks hive metastore.Is there any way to get the metadata of all the tables inspite of looping through tables using Describe table_name.As hive metastore doesnot support inf...
My job fails with Driver is temporarily unavailable. Apparently, it's permanently unavailable, because the job is not pausing but failing.
I am facing the same issues . I am writing in batches using a simple for loop. I don't have any collect statements inside the loop. I am rewriting the partitions with partition overwrite dynamic mode in a huge wide delta table - several tb. The incr...
I have code files (.hql) in s3 which was running on Hive. But now, I need to run them on the Databricks cluster. I can rename the files to .sql and add comment "-- Databricks notebook source" on top of the files to treat them as single cell notebook....
try_to_timestamp should return NULL for following expression but it is throwing error: try_to_timestamp( '2019-02-28 23:59:59.000', 'yyyy-MM-dd HH:mm:ss' ) I understand above expression is wrong as the date format does no...
Hi @PriyanshuKumar,Thank you for your question in the Databricks community.The try_to_timestamp() function is designed to attempt to convert a string to timestamp format based on a specified pattern and return null if it fails to do so. However, in t...
Hello,I am doing some testing with this feature Change Data Feed using Databricks and Pyspark, of course the Delta format and I don't understand something:I created a tableSaved some data insideEnabled Change Data Feed featureApply a merge with a dat...
@Mihai_Cog You have to split your merge statement into 2 parts. (Update and Insert/Delete).MERGE INTO test t USING src s ON s.Id = t.Id and s.date_field = t.date_field and s.fields <> t.fields WHEN MATCHED THEN UPDATE SET * MERGE INTO test t USING sr...
Hi,I gave Databricks Certified Associate Developer for Apache Spark 3.0 exam today but missed by one percent. I got 68.33% and pass is 70%.I am planning to reattempt the exam, could you kindly give me another opportunity and provide reattempt voucher...
Hi,I gave Databricks Certified Associate Developer for Apache Spark 3.0 Python exam yesterday but missed by three percent. I got 66.66% and pass is 70%.I am planning to reattempt the exam, could you kindly give me another opportunity and provide reat...
Has anybody succeeded in querying Databricks SQL with an on-prem SSRS (so an on-prem Report Server and Report Builder)?I manage to create a connection that works (according to the connection test anyway), but the moment I try to create a dataset on t...
Hi @-werners- , I am able to build connection between Microsoft visual studio and data bricks using Simba Spark ODBC driver. I can query delta tables sitting in Databricks from Microsoft Visual studio (SSRS). However, when I am deploying the report t...
Hi Databricks Team/Community,I'm planning for Databricks Data Engineering Professional Certification but what I have seen is that Databricks don't give any practice test for this professional/advanced certification. I don't know why, If anyone is rea...
Hello everyone! Basically, I have a dropdown widget, and at some point I want to change the options available, but still maintain the same selected option. The problems that I find are: - If I want to delete the widget and create a new one, it seems...
I know this is 5+ years late but I would like to add an answer (or work around) to this in case anyone else was looking for the same solution.What I did was to essentially 'reset' the widget whenever I had to assign (or force) a value to it.#Cell #1 ...
I was trying to read some delta data from databricks[Hive metastore] sql endpoint using pyspark, but while doing so I encountered that all the values of the table after fetching are same as the column name.Even when I try to just show the data it giv...
Encountered the same issue and downgrading to 2.6.22 helped me resolve this issue.
How to expand display of workspace name in top right window? My workspace name is little bit longer and it cuts out, could not find a way to expand name window.
@pg1 Is your workspace Unity Catalog enabled?We have identified this behaviour in Unity Catalog enabled workspaces and already working to show up the entire workspace name when we hover over it.
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
