- 37537 Views
- 14 replies
- 15 kudos
Resolved! How to convert records in Azure Databricks delta table to a nested JSON structure?
Let's say I have a delta table in Azure databricks that stores the staff details (denormalized). I wanted to export the data in the JSON format and save it as a single file on a storage location. I need help with the databricks sql query to group/co...
- 37537 Views
- 14 replies
- 15 kudos
- 3231 Views
- 3 replies
- 3 kudos
Where is the learning material to get Fundamentals of the Databricks Lakehouse Platform Accreditation?
Please provide me some information about how to get the martial to pass Fundamentals of the Databricks Lakehouse Platform Accreditation?
- 3231 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Shanthala Baleer​,Just a friendly follow-up. Are you still looking for help? adding @Vidula Khanna​ for visibility
- 3 kudos
- 2976 Views
- 1 replies
- 0 kudos
How to restart snowflake connector?
After using spark.read.format("snowflake").options(**options).option("dbtable", "table_name").load() to read a table from Snowflake, when I then change the table from Snowflake and read it again, it gives me the first version of the table. I have wor...
- 2976 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, that would work. However, it is a longish Snowflake query producing a number of tables that are all called by the Databricks notebook, so it requires quite a few changes. I'll use this alternative if I automate the process. However, I think this...
- 0 kudos
- 11300 Views
- 5 replies
- 7 kudos
Resolved! Why was SQL formatting removed inside spark.sql functions? Now it looks like a plain string.
Previously we were able to see SQL queries inside spark.sql() like this:But now it just looks like a plain string: I know it's not a big issue, but it's still annoying to have to code in SQL while having it all be blue, it makes debugging more cumber...
- 11300 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @Emilio Garza​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 7 kudos
- 8751 Views
- 4 replies
- 0 kudos
Creating a spot only single-node job compute cluster policy
Hi there,I need some help creating a new cluster policy that utilizes a single spot-instnace server to complete a job. I want to set this up as a job-compute to reduce costs and also utilize 1 spot instance.The jobs I need to ETL are very short and c...
- 8751 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Avkash Kana​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 0 kudos
- 3263 Views
- 4 replies
- 0 kudos
how to optimize the runtime in 10.4 cluster
i am loading the 1billion data from spark dataframe into target table, but in the 7.3 cluster it takes 3 hours to complete but after migrated to 10.4 cluster its taking 8 hours to complete , how can i reduce the time duration​
- 3263 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Mohammed sadamusean​,Could you provide more details on what are you doing? What type of transformations/actions are you doing? whats your source and sink? batch or streaming? all that information will help.
- 0 kudos
- 6698 Views
- 3 replies
- 0 kudos
Query Databricks from Power BI with Row Level Security
I am trying to apply RLS to the solution but Power BI only connects to Databricks(DB) using a token which cant be used in DB groups. Is there no other way to apply Row Level security using Power BI?
- 6698 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Daniel Gomes​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 0 kudos
- 4100 Views
- 3 replies
- 2 kudos
Unable to connect/read files from ADLS Gen2 using account key
It gives error[RequestId=5e57b66f-b69f-4e8b-8706-3fe5baeb77a0 ErrorClass=METASTORE_DOES_NOT_EXIST] No metastore assigned for the current workspace.using the following codespark.conf.set( "fs.azure.account.key.mystorageaccount.dfs.core.windows.net", ...
- 4100 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Farooq ur rehman​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 2 kudos
- 8842 Views
- 5 replies
- 0 kudos
Delta Live Tables data quality rules application.
I have a requirement, where I need to apply inverse DQ rule on a table to track the invalid data. For which I can use the following approach:import dltrules = {}quarantine_rules = {}rules["valid_website"] = "(Website IS NOT NULL)"rules["valid_locatio...
- 8842 Views
- 5 replies
- 0 kudos
- 0 kudos
You can get additional info from DLT event log which is in delta so you can load it as table https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-event-log.html#data-quality
- 0 kudos
- 2674 Views
- 1 replies
- 10 kudos
Since databricks runtime 12.1 "WHEN NOT MATCHED BY SOURCE" was added to MERGE syntax. For example, using that option, we can quickly delete ...
Since databricks runtime 12.1 "WHEN NOT MATCHED BY SOURCE" was added to MERGE syntax. For example, using that option, we can quickly delete all target rows which doesn't match any source.
- 2674 Views
- 1 replies
- 10 kudos
- 10 kudos
Thank you for sharing @Hubert Dudek​
- 10 kudos
- 9600 Views
- 3 replies
- 4 kudos
DLT streaming table and LEFT JOIN
I'm trying to build gold level streaming live table based on two streaming silver live tables with left join.This attempt fails with the next error:"Append mode error: Stream-stream LeftOuter join between two streaming DataFrame/Datasets is not suppo...
- 9600 Views
- 3 replies
- 4 kudos
- 4 kudos
Thanks Fatma,I do understand the need for watermarks, but I'm just wondering if this supported by SQL syntax?
- 4 kudos
- 5460 Views
- 1 replies
- 1 kudos
add custom metadata to avro file with pyspark
Hello, i need to add a custom metadata into a avro file. The avro file containt data. we have tried to use "option" within the write function but it's not taken without generated any error.df.write.format("avro").option("avro.codec", "snappy").option...
- 5460 Views
- 1 replies
- 1 kudos
- 3332 Views
- 1 replies
- 1 kudos
What is the maximum number of workspaces per account using Databricks on AWS?
I've been looking through the documentation and I swear this used to be listed somewhere, but for the life of me I can't find it anymore.
- 3332 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks @Kaniz Fatma​
- 1 kudos
- 1709 Views
- 0 replies
- 0 kudos
- 1709 Views
- 0 replies
- 0 kudos
- 17597 Views
- 7 replies
- 10 kudos
Resolved! Removing files saved in the root FileStore
We have just started working with databricks in one of my university modules, and the lecturers gave us a set of commands to practice saving data in the FileStore. One of the commands was the following:dbutils .fs.cp("/ databricks - datasets / weathh...
- 17597 Views
- 7 replies
- 10 kudos
- 10 kudos
you can delete files using the data explorer in Databricks web UI.another option is to use %fs or %sh in a notebook.
- 10 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
