- 7220 Views
- 1 replies
- 3 kudos
I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\
.location(target_path)\
.addColumns(df.sche...
- 7220 Views
- 1 replies
- 3 kudos
Latest Reply
Hi @Chris Chung, Can you re-check by trying the below code?df.write.format("delta").option("delta.columnMapping.mode", "name").save("s3://anotherpath")Now you can load it into a Spark dataframe:SELECT * FROM new_table;delta_df = spark.read.format("d...
- 1738 Views
- 1 replies
- 9 kudos
I have a command that is running notebooks in parallel using threading. I want the command to fail whenever one of the notebooks that is running fails. Right now it is just continuing to run the command.Below is the command line that I'm currently ru...
- 1738 Views
- 1 replies
- 9 kudos
Latest Reply
Hi @Coleman Milligan,You can run multiple Azure Databricks notebooks in parallel by using the dbutils library.Here is a python code based on the sample code from the Azure Databricks documentation on running notebooks concurrently and on Notebook wo...
- 2058 Views
- 5 replies
- 4 kudos
Hello.I want to mount and share for the one group the container from Azure Blob Storage (It could be simple blob storage or Azure Data Lake Storage gen 2). But I am not able to do it because I am using Cluster with Table Access Control.This is my cod...
- 2058 Views
- 5 replies
- 4 kudos
Latest Reply
I have a good solution to the problem:I am using Python library.There are some documentation.Topic to be closed.Best regards,Łukasz
4 More Replies
- 195 Views
- 0 replies
- 0 kudos
RS-485 IoT Gateway is used for transmitting data from one device to another usually far away by using a wired LAN or a Wi-Fi. HashStudioz Technologies Inc. provides Smart IoT Gateway Solutions for Businesses like Pharma industries Etc. Our IoT Gatewa...
- 195 Views
- 0 replies
- 0 kudos
- 5659 Views
- 6 replies
- 4 kudos
Hello, I'm facing some problems while writing data to Google BigQuery. I'm able to read data from the same table, but when I try to append data I get the following error.Error getting access token from metadata server at: http://169.254.169.254/compu...
- 5659 Views
- 6 replies
- 4 kudos
Latest Reply
Sometime this error occur when your Private key or your service account key is not going in request header, So if you are using Spark or Databricks then you have to configure the JSON Key in Spark config so it will be added in request header.
5 More Replies
- 870 Views
- 3 replies
- 3 kudos
I have read that delta live tables will keep a history of 7 days. However after creating a streaming live table and using the dlt.apply_changes function. With this codedef run_pipeline(table_name,keys,sequence_by):
lower_table_name = table_name.l...
- 870 Views
- 3 replies
- 3 kudos
Latest Reply
Hi @Logan Nicol Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
2 More Replies
by
Muni
• New Contributor II
- 3546 Views
- 5 replies
- 0 kudos
We are trying to expose a REST API endpoint using python notebook. Does databricks allow this?
- 3546 Views
- 5 replies
- 0 kudos
Latest Reply
Hi @Muniyappan Mani Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
4 More Replies
- 1025 Views
- 5 replies
- 2 kudos
I attended "The Best Data Warehouse Is a Lakehouse" on September 20 but have not recieved my voucher although I recieved a mail which mentioned below details.@Kaniz Fatma Please help me on same.Mail Id: bose.innovator8@gmail.comPlease complete the ...
- 1025 Views
- 5 replies
- 2 kudos
Latest Reply
Hi @Arindam Bose That's great to hear! Please mark the answer as best.Thanks and Regards
4 More Replies
- 2066 Views
- 5 replies
- 6 kudos
I tried to enable the photon acceleration in ML runtime 9.1 LTS ML (Scala 2.12,Spark 3.1.2) but getting error "selected runtime version does not support photon".I tried for other versions of ML runtime with single and multinode , access mode being s...
- 2066 Views
- 5 replies
- 6 kudos
Latest Reply
Hi @Sajid Thavalengal Rahiman Does @Kaniz Fatma response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
4 More Replies
by
140015
• New Contributor III
- 1321 Views
- 6 replies
- 1 kudos
Hello,I'm wondering if there is an option to make an expectation on DLT that will compare the number of records between two stages and e.g. fail if there is a difference between those counts?I mean something like this:@dlt.table()def bronze(): Some...
- 1321 Views
- 6 replies
- 1 kudos
Latest Reply
Hi @Jacek Dembowiak Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
5 More Replies
- 729 Views
- 4 replies
- 1 kudos
I have cleared Databricks Foundational Accreditation today and received the completion certificate. But I didnt receive the badge for the same. By when I will receive the badge ?
- 729 Views
- 4 replies
- 1 kudos
Latest Reply
Hi @Sakshi Puchnanda Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
3 More Replies
- 681 Views
- 3 replies
- 1 kudos
I have tried pulling a single row from a .csv using df.query()However, the data being returned doesn't coincide with the data I'm expecting - it is pulling a different row. Here is my code:df = spark.read.option("header",True).csv(data_fldr + "config...
- 681 Views
- 3 replies
- 1 kudos
Latest Reply
Hi @Peter Ott Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
2 More Replies
- 1023 Views
- 2 replies
- 1 kudos
When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.I understand DLT pipelines utilising their own special clusters...
- 1023 Views
- 2 replies
- 1 kudos
Latest Reply
Hi @Bo Zhu Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
1 More Replies
- 954 Views
- 2 replies
- 0 kudos
The latest version of Databricks-jdbc available through Maven (2.6.29) now has these two vulnerabilities:https://nvd.nist.gov/vuln/detail/CVE-2022-42004https://nvd.nist.gov/vuln/detail/CVE-2022-42003All due to depending on and including in the jar th...
- 954 Views
- 2 replies
- 0 kudos
Latest Reply
Hi @Lars Joreteg Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
1 More Replies
by
AJDJ
• New Contributor III
- 1100 Views
- 5 replies
- 1 kudos
Hi there,I'm learning Databricks using the community edition. I noticed I don't have way to practice the pipeline in community edition. (The icon below compute). Says I need to upgrade. Is there any way to practice pipeline and follow learning lesson...
- 1100 Views
- 5 replies
- 1 kudos
Latest Reply
Hi @AJ DJ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
4 More Replies
