- 1333 Views
- 0 replies
- 1 kudos
- 1333 Views
- 0 replies
- 1 kudos
Forum Posts
- 2514 Views
- 1 replies
- 2 kudos
Moving data to a delta table keeping the old surrogate ids intact
Hello experts! We have a table in our current system that we need to move it (one-off) to a delta in Databricks keeping its Ids (surrogate keys) intact. We think to of the following steps: 1. create a new delta table with a "BIGINT GENERATED BY DEFAU...
- 2514 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
same here, I submitted an idea in the azure databricks portalhttps://feedback.azure.com/d365community/idea/d403303c-6761-ed11-a81b-000d3ae5ae95SET IDENTITY_INSERT ONwhen a column is defined as GENERATED ALWAYS, we often need to reload data with exact...
- 2 kudos
- 2121 Views
- 0 replies
- 2 kudos
Schema supported by Autoloader
We do not want to use schema inference with schema evolution in Autoloader. Instead, we want to apply our schema and use the merge option. Our schema is very complex, with multiple nested following levels. When I apply this schema to Autoloader, it r...
- 2121 Views
- 0 replies
- 2 kudos
- 3297 Views
- 2 replies
- 6 kudos
Resolved! new learner
i am absolutely new for data bricks .. can someone suggest best medium to learn at the quick pace
- 3297 Views
- 2 replies
- 6 kudos
Latest Reply
- 6 kudos
In addition to paid content, there are bunch of really good free courses on Databricks Academy now : https://customer-academy.databricks.com/learn/catalog?ctldoc-catalog-0=t-_%22learning_plan%22~p-0Just pick the Data engineer or the Data analyst lear...
- 6 kudos
- 4665 Views
- 3 replies
- 19 kudos
Resolved! UC - Service Principal/Terraform
Hi,do you know if there is a way to create Unity Catalog metastore using Service Principal?Here I can see that for creating account-level resources we need to provide a user and password (https://registry.terraform.io/providers/databricks/databricks/...
- 4665 Views
- 3 replies
- 19 kudos
Latest Reply
- 19 kudos
This is supported right now in the Azure, but not yet in AWS, but there is plan for AWS support as well.
- 19 kudos
- 5471 Views
- 3 replies
- 5 kudos
Resolved! Identity column definition lost using save as table
I found an issue:For a table with an identity column defined.when the table column is renamed using this method, the identity definition will be removed. That means using an identity column in a table requires extra attention to check whether the ide...
- 5471 Views
- 3 replies
- 5 kudos
Latest Reply
- 5 kudos
try to avoid reload table, I found we can upgrade table version, and use rename column commandALTER TABLE test_id2 SET TBLPROPERTIES ( 'delta.columnMapping.mode' = 'name', 'delta.minReaderVersion' = '2', 'delta.minWriterVersion' = '6')ALTER TABLE ...
- 5 kudos
- 8132 Views
- 3 replies
- 1 kudos
Resolved! unable to add new instance profile
Hi team, I want to start adding more instance profile per team . when adding it via TF , im getting the below error:Im able BTW to add and change other databricks resources via TF .This is my new code block:The new role created by the TF but wasnt ad...
- 8132 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Thank you @Vivian Wilfred​ all ok now, with the databricks console and the TFHave a graet day!
- 1 kudos
- 2323 Views
- 0 replies
- 1 kudos
Hi,I am a Solutions Architect at Databricks Northern Europe. I had a deep dive demo session at Data + AI Work Tour in London on Nov 2, 2022. The title...
Hi,I am a Solutions Architect at Databricks Northern Europe. I had a deep dive demo session at Data + AI Work Tour in London on Nov 2, 2022. The title was How to Build Your Modern Data Stack on Databricks to Solve Modern Problems.In the above-mention...
- 2323 Views
- 0 replies
- 1 kudos
- 3190 Views
- 4 replies
- 1 kudos
Certificate not received
I have passed Databricks Certified Data Engineer Associate on 29 October 2022 but still didn't receive the certificate, kindly help me on that.
- 3190 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Anandhakumar R​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue.If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please visit the ...
- 1 kudos
- 3162 Views
- 4 replies
- 0 kudos
I have passed my data analyst associate exam on 30th October , but still not received my certificate, its been more than than 4 days now , however Dat...
I have passed my data analyst associate exam on 30th October , but still not received my certificate, its been more than than 4 days now , however Databricks mentioned in an email that will deliver within 24 hours. Can anyone help on this ?
- 3162 Views
- 4 replies
- 0 kudos
Latest Reply
- 9851 Views
- 3 replies
- 7 kudos
Resolved! Converting a transformation written in Spark Scala to PySpark
Hello all,I've been tasked to convert a Scala Spark code to PySpark code with minimal changes (kinda literal translation).I've come across some code that claims to be a list comprehension. Look below for code snippet:%scala val desiredColumn = Seq("f...
- 9851 Views
- 3 replies
- 7 kudos
Latest Reply
- 7 kudos
Another follow-up question, if you don't mind. @Pat Sienkiewicz​ As I was trying to parse the name column into multiple columns. I came across the data below:("James,\"A,B\", Smith", "2018", "M", 3000)In order to parse these comma-included middle na...
- 7 kudos
- 5011 Views
- 4 replies
- 19 kudos
Resolved! How to prepare for Databricks Data Engineer Professional
Hi all,Could you please help suggest me some resource to prepare for " Databricks Data Engineer Professional" exam?I have also take the course in Databricks Accademy but seems not enough for this exam?Thank you so much!!!Best Regards,Nhan Nguyen
- 5011 Views
- 4 replies
- 19 kudos
Latest Reply
- 19 kudos
waitting some one post more details and experience road map to take exam
- 19 kudos
- 7276 Views
- 1 replies
- 3 kudos
How to resolve our of memory error?
Hi, I am working as azure support engineerI found this error while I am checking the pipeline failure, and showing below error"org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 72403.0 failed 4 times, most recent fail...
- 7276 Views
- 1 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi @mahesh bmk​ ,It would be nice to see the sql_query.is there some window function used? You might try to run this on bigger cluster.
- 3 kudos
- 8265 Views
- 0 replies
- 4 kudos
Export commands and output of the Databricks Notebook to MS Excel
I have a Databricks notebook and I have several headers, SQL commands and their output. I am currently copying the output and SQL commands manually to excel for a report. How can I reduce the manual work of copy pasting from Notebook to excel and au...
- 8265 Views
- 0 replies
- 4 kudos
- 4336 Views
- 2 replies
- 5 kudos
Hello, I'm trying to use Databricks on Azure with a Spark structured streaming job and an having very mysterious issue. I boiled the job down it i...
Hello,I'm trying to use Databricks on Azure with a Spark structured streaming job and an having very mysterious issue.I boiled the job down it it's basics for testing, reading from a Kafka topic and writing to console in a forEachBatch.On local, ever...
- 4336 Views
- 2 replies
- 5 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
