cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

rams
by Contributor
  • 3861 Views
  • 3 replies
  • 4 kudos

Rollback error - Configuring Databricks lakehouse platform with AWS account

I have logged in databricks account and while creating the workspace i have chosen quickstart approach to configure the databricks with AWS. During the quickstart process the databricks page will redirect to aws cloudformation stack page where the ac...

  • 3861 Views
  • 3 replies
  • 4 kudos
Latest Reply
Debayan
Databricks Employee
  • 4 kudos

Hi @rams shonu​ , The error looks like it is due to the length of the roleName. AWS IAM Role names are limited to 64 characters. Could you please try to edit the default roleName and try to append?

  • 4 kudos
2 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 2570 Views
  • 1 replies
  • 29 kudos

The New DBR LTS runtime is here. Databricks Runtime 11.3 is set to be LTS!   

The New DBR LTS runtime is here. Databricks Runtime 11.3 is set to be LTS! 

11.3 LTS
  • 2570 Views
  • 1 replies
  • 29 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 29 kudos

Great post, thank you for sharing this @Hubert Dudek​ 

  • 29 kudos
Jyo
by New Contributor II
  • 2681 Views
  • 2 replies
  • 0 kudos

Rollback error - during deploying the workspace through quickstart

These are the steps I followed:1) Under quickstart-> added workspace name-> selected N-virginia (us-east-1), and quick start2) next step: except the password, I haven't edited any of the below:stackname- (default) databricks-workspace-stackparameters...

  • 2681 Views
  • 2 replies
  • 0 kudos
Latest Reply
rams
Contributor
  • 0 kudos

Hi Jyo, I followed the same "quickstart" approach to configure the databricks with aws account and got the same errors. Please can you tell us how to solve the issue if you have already solved. Thank you.

  • 0 kudos
1 More Replies
Viren123
by Contributor
  • 12724 Views
  • 2 replies
  • 3 kudos

Resolved! Error : MALFORMED_REQUEST

Hello,I get error for below json. Can you please advice what am I missing here?{    "error_code": "MALFORMED_REQUEST",    "message": "Invalid JSON given in the body of the request - failed to parse given JSON"_________________________________________...

  • 12724 Views
  • 2 replies
  • 3 kudos
Latest Reply
Viren123
Contributor
  • 3 kudos

Thank you Pat. I just realised I had copy pasted the code and it didnt quite copy 1:1. In copy paste double quotes were appeared to be copied but behind the scene, it was some other character. Nothing wrong with the code. It works

  • 3 kudos
1 More Replies
User16752245636
by New Contributor II
  • 1024 Views
  • 0 replies
  • 1 kudos

Sergio Copy Data World tour Orchestration_v02.pptx (4)

Databricks Workflows: Reliable orchestration for data, analytics, and AIHello, I am a Solutions Architect at Databricks; and I recently presented at the Data and AI World Tour in London Databricks Workflows. You can see some of the presented slides a...

  • 1024 Views
  • 0 replies
  • 1 kudos
al_joe
by Contributor
  • 5933 Views
  • 5 replies
  • 1 kudos

Resolved! Databricks Academy LMS - shortcomings

I am using Databricks Academy extensively and see a few shortcomings in the (new) LMS -- Feedback:- Videos do not have captions/transcript to make them accessible to all audience including non-native English speakers.- "FullScreen" icon does not work...

image
  • 5933 Views
  • 5 replies
  • 1 kudos
Latest Reply
User16847923431
Databricks Employee
  • 1 kudos

Hi @al_joe! My name is Astrid, and I’m on the curriculum team here at Databricks. I was able to review your original post in this thread and wanted to update you on some of the input and questions you provided: Videos do not have captions/transcript ...

  • 1 kudos
4 More Replies
Shan3009
by New Contributor III
  • 5923 Views
  • 3 replies
  • 5 kudos

The transaction log has failed integrity checks. We recommend you contact Databricks support for assistance. Failed verification at version 48 of:

I am trying to write data frame data into delta table. Previously it was working fine but now it is throwing "Log has failed integrity checks"

  • 5923 Views
  • 3 replies
  • 5 kudos
Latest Reply
jcasanella
New Contributor III
  • 5 kudos

@Shanmuganathan Jothikumar​ I've the same exception after upgrading into unity catalog. Need to investigate a little more but adding the following setting, it works:spark.conf.set("spark.databricks.delta.state.corruptionIsFatal", False)

  • 5 kudos
2 More Replies
patojo94
by New Contributor II
  • 4379 Views
  • 3 replies
  • 4 kudos

Resolved! pyspark streaming failed for now reason

Hi everyone, I have a pyspark streaming reading from an aws kinesis that suddenly failed for no reason (I mean, we did not make any changes in the last time).It is giving the following error: ERROR MicroBatchExecution: Query kinesis_events_prod_bronz...

  • 4379 Views
  • 3 replies
  • 4 kudos
Latest Reply
jcasanella
New Contributor III
  • 4 kudos

@patricio tojo​ I've the same problem, however in my case is after migrating into unity catalog. Need to investigate a little more but adding this to my spark job, it works:spark.conf.set("spark.databricks.delta.state.corruptionIsFatal", False)

  • 4 kudos
2 More Replies
absolutelyRice
by New Contributor III
  • 11618 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks Terraform Provider Issues Passing Providers to Child Modules

I have been following the documentation on the terraform databricks documentation in order to provision account level resources on AWS. I can create the workspace fine, add users, etc... However, when I go to use the provider in non-mws mode, I am re...

  • 11618 Views
  • 5 replies
  • 2 kudos
Latest Reply
absolutelyRice
New Contributor III
  • 2 kudos

So the answer to this was that you need to explicitly pass the provider argument to each of the data resources blocks. The docs should be updated to accommodate that. ​i.e. data "databricks_spark_version" "latest" { provider = databricks.workspace ...

  • 2 kudos
4 More Replies
deficiant_codge
by Contributor II
  • 2534 Views
  • 1 replies
  • 7 kudos

Delta Live tables support for UNITY CATALOG

Is there any upcoming update in which UC will support DLT? if yes any expected ETA?

  • 2534 Views
  • 1 replies
  • 7 kudos
Latest Reply
Pat
Esteemed Contributor
  • 7 kudos

Hi @Rahul Mishra​,I think you need to contact your comapny databricks representative for this.The last time I heard about the ETA It was end of the November I believe.You might try to join the Databricks Office Hours tomorrow and ask the question or ...

  • 7 kudos
Ranjeeth
by New Contributor
  • 2099 Views
  • 1 replies
  • 2 kudos
  • 2099 Views
  • 1 replies
  • 2 kudos
Latest Reply
Pat
Esteemed Contributor
  • 2 kudos

Hi @Ranjeeth Rikkala​ ,you can check this:https://docs.databricks.com/optimizations/index.html#optimization-recommendations-on-databricksTBH, It’s nit enough info. Make sure you have partioned data, Files are nit too small, try to use bigger cluster,...

  • 2 kudos
HariSelvarajan
by Databricks Employee
  • 1114 Views
  • 0 replies
  • 5 kudos

DAIWT22_RadicalSpeenInLakehouse_Photon

Topic: Radical Speed on the Lakehouse: Photon under the hoodI am Hari and I works as a Specialist Solutions Architect at Databricks. I specialise in Data engineering and Cloud platforms problems helping client in EMEA.Purpose:I recently presented a t...

  • 1114 Views
  • 0 replies
  • 5 kudos
Sourav_Gulati
by Databricks Employee
  • 1185 Views
  • 0 replies
  • 7 kudos

This post is regarding Data Streaming on the Lakehouse' session in Data + AI World Tour 2022 in London. I am Resident Solutions Architect at Datab...

This post is regarding Data Streaming on the Lakehouse' session in Data + AI World Tour 2022 in London. I am Resident Solutions Architect at Databricks. I specialise in data engineering.In this session, I talked about how to leverage real time data t...

  • 1185 Views
  • 0 replies
  • 7 kudos
Digan_Parikh
by Valued Contributor
  • 17402 Views
  • 2 replies
  • 3 kudos

Resolved! Default Query Limit 1000

By default, we return back up to 1000 query results when a user runs a cell in Databricks. E.g., if you run display(storeData) and you have ten million customers, the UI will show the first 1000 results. If you graph that by age of customer, similarl...

  • 17402 Views
  • 2 replies
  • 3 kudos
Latest Reply
User16805453151
New Contributor III
  • 3 kudos

This is simple in Databricks SQL, just uncheck LIMIT 1000 in the drop down.

  • 3 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels