- 2810 Views
- 1 replies
- 0 kudos
How to restart snowflake connector?
After using spark.read.format("snowflake").options(**options).option("dbtable", "table_name").load() to read a table from Snowflake, when I then change the table from Snowflake and read it again, it gives me the first version of the table. I have wor...
- 2810 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, that would work. However, it is a longish Snowflake query producing a number of tables that are all called by the Databricks notebook, so it requires quite a few changes. I'll use this alternative if I automate the process. However, I think this...
- 0 kudos
- 10351 Views
- 5 replies
- 7 kudos
Resolved! Why was SQL formatting removed inside spark.sql functions? Now it looks like a plain string.
Previously we were able to see SQL queries inside spark.sql() like this:But now it just looks like a plain string: I know it's not a big issue, but it's still annoying to have to code in SQL while having it all be blue, it makes debugging more cumber...
- 10351 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @Emilio Garza​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 7 kudos
- 8222 Views
- 4 replies
- 0 kudos
Creating a spot only single-node job compute cluster policy
Hi there,I need some help creating a new cluster policy that utilizes a single spot-instnace server to complete a job. I want to set this up as a job-compute to reduce costs and also utilize 1 spot instance.The jobs I need to ETL are very short and c...
- 8222 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Avkash Kana​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 0 kudos
- 3077 Views
- 4 replies
- 0 kudos
how to optimize the runtime in 10.4 cluster
i am loading the 1billion data from spark dataframe into target table, but in the 7.3 cluster it takes 3 hours to complete but after migrated to 10.4 cluster its taking 8 hours to complete , how can i reduce the time duration​
- 3077 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Mohammed sadamusean​,Could you provide more details on what are you doing? What type of transformations/actions are you doing? whats your source and sink? batch or streaming? all that information will help.
- 0 kudos
- 6429 Views
- 3 replies
- 0 kudos
Query Databricks from Power BI with Row Level Security
I am trying to apply RLS to the solution but Power BI only connects to Databricks(DB) using a token which cant be used in DB groups. Is there no other way to apply Row Level security using Power BI?
- 6429 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Daniel Gomes​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 0 kudos
- 3870 Views
- 3 replies
- 2 kudos
Unable to connect/read files from ADLS Gen2 using account key
It gives error[RequestId=5e57b66f-b69f-4e8b-8706-3fe5baeb77a0 ErrorClass=METASTORE_DOES_NOT_EXIST] No metastore assigned for the current workspace.using the following codespark.conf.set( "fs.azure.account.key.mystorageaccount.dfs.core.windows.net", ...
- 3870 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Farooq ur rehman​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 2 kudos
- 8315 Views
- 5 replies
- 0 kudos
Delta Live Tables data quality rules application.
I have a requirement, where I need to apply inverse DQ rule on a table to track the invalid data. For which I can use the following approach:import dltrules = {}quarantine_rules = {}rules["valid_website"] = "(Website IS NOT NULL)"rules["valid_locatio...
- 8315 Views
- 5 replies
- 0 kudos
- 0 kudos
You can get additional info from DLT event log which is in delta so you can load it as table https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-event-log.html#data-quality
- 0 kudos
- 2510 Views
- 1 replies
- 10 kudos
Since databricks runtime 12.1 "WHEN NOT MATCHED BY SOURCE" was added to MERGE syntax. For example, using that option, we can quickly delete ...
Since databricks runtime 12.1 "WHEN NOT MATCHED BY SOURCE" was added to MERGE syntax. For example, using that option, we can quickly delete all target rows which doesn't match any source.
- 2510 Views
- 1 replies
- 10 kudos
- 10 kudos
Thank you for sharing @Hubert Dudek​
- 10 kudos
- 9210 Views
- 3 replies
- 4 kudos
DLT streaming table and LEFT JOIN
I'm trying to build gold level streaming live table based on two streaming silver live tables with left join.This attempt fails with the next error:"Append mode error: Stream-stream LeftOuter join between two streaming DataFrame/Datasets is not suppo...
- 9210 Views
- 3 replies
- 4 kudos
- 4 kudos
Thanks Fatma,I do understand the need for watermarks, but I'm just wondering if this supported by SQL syntax?
- 4 kudos
- 5292 Views
- 1 replies
- 1 kudos
add custom metadata to avro file with pyspark
Hello, i need to add a custom metadata into a avro file. The avro file containt data. we have tried to use "option" within the write function but it's not taken without generated any error.df.write.format("avro").option("avro.codec", "snappy").option...
- 5292 Views
- 1 replies
- 1 kudos
- 3122 Views
- 1 replies
- 1 kudos
What is the maximum number of workspaces per account using Databricks on AWS?
I've been looking through the documentation and I swear this used to be listed somewhere, but for the life of me I can't find it anymore.
- 3122 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks @Kaniz Fatma​
- 1 kudos
- 1597 Views
- 0 replies
- 0 kudos
- 1597 Views
- 0 replies
- 0 kudos
- 16504 Views
- 7 replies
- 10 kudos
Resolved! Removing files saved in the root FileStore
We have just started working with databricks in one of my university modules, and the lecturers gave us a set of commands to practice saving data in the FileStore. One of the commands was the following:dbutils .fs.cp("/ databricks - datasets / weathh...
- 16504 Views
- 7 replies
- 10 kudos
- 10 kudos
you can delete files using the data explorer in Databricks web UI.another option is to use %fs or %sh in a notebook.
- 10 kudos
- 2272 Views
- 1 replies
- 3 kudos
Workspace usage for the partners
We have 11 people working on the Data Engineering Associate certification using Data Engineering with Databricks V3. We just got done with the Foundation one and start the Engineering journey. We are Registered partners and Data Engineering with Dat...
- 2272 Views
- 1 replies
- 3 kudos
- 3 kudos
Hell Shanthala, you can send an email to partnerops@databricks.com who then provide information how to set this up
- 3 kudos
- 11740 Views
- 7 replies
- 0 kudos
Resolved! Pipeline failed while calling Databricks Notebook - Cluster Terminated
Hello,We have an Azure Data Factory pipeline running during the night, and one of the activities calls a Databricks Notebook with dynamic DatabricksInstancePoolId, ClusterVersion and Workers. Yesterday, it failed with with the following error:Cluster...
- 11740 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @Rita Fernandes​,What are you trying to install in your init script? only the ODBC driver or some other libraries/dependencies?
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
