cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
martcerv
by • New Contributor II
  • 2899 Views
  • 4 replies
  • 2 kudos

Cloud provider launch failure

When I want to create a cluster a get this error message:DetailsAWS API error code: InvalidGroup.NotFoundAWS error message: The security group 'sg-0ded75eefd66bf421' does not exist in VPC 'vpc-0ec7da3d5977f6ec9'And when I inspect the security groups ...

Data Engineering
Reply
  • 2899 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
AminChad_22427
New Contributor II
  • 2 kudos

Hi, I am running into a similar issue. but in my case, the security has been deleted by mistake.Is there a way to make Databricks recreate the missing group ?@Kaniz Fatma​ , where can the CreateSecurityGroup command be ran ? Does it change the securi...

  • 2 kudos
3 More Replies
sudhanshu1
by • New Contributor III
  • 639 Views
  • 0 replies
  • 0 kudos

Structured Streaming

I need some solution for below problem.We have set of json files which are keep coming to aws s3, these files contains details for a property . please note 1 property can have 10-12 rows in this json file. Attached is sample json file.We need to read...

Data Engineering
Reply
  • 639 Views
  • 0 replies
  • 0 kudos
KVNARK
by • Honored Contributor II
  • 3255 Views
  • 4 replies
  • 13 kudos

Resolved! To practice Databricks SQL

Is there any sand box kind of thing where we can do some hands-on on Databricks SQL/run the Note books attaching to the Clusters apart from the free trial provided by Databricks.

Data Engineering
Reply
  • 3255 Views
  • 4 replies
  • 13 kudos
View Replies
Latest Reply
Harun
Honored Contributor
  • 13 kudos

Databricks SQL workspace will be available only for Databricks Premium service. If you have Azure Pass subscription, then you can able to get it for practicing it.

  • 13 kudos
3 More Replies
avidex180899
by • New Contributor II
  • 10220 Views
  • 3 replies
  • 3 kudos

Resolved! UUID/GUID Datatype in Databricks SQL

Hi all,I am trying to create a table with a GUID column.I have tried using GUID, UUID; but both of them are not working.Can someone help me with the syntax for adding a GUID column?Thanks!

Data Engineering
Reply
  • 10220 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

Hey @Avinash Narasimhan​ , What is the exact problem you are getting can you please share it is working fine for meThanksAviral Bhardwaj

  • 3 kudos
2 More Replies
Jyo777
by • Contributor
  • 1583 Views
  • 4 replies
  • 0 kudos

Hi, Has anyone cleared professional DE? please advise on professional data engineer exam. will advance DE learning path be sufficient? Or need to fol...

Hi,Has anyone cleared professional DE? please advise on professional data engineer exam. will advance DE learning path be sufficient? Or need to follow some other resource as well.

Data Engineering
Reply
  • 1583 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Hello Have a look at this link http://msdatalab.net/how-to-pass-the-professional-databricks-data-engineering/

  • 0 kudos
3 More Replies
architect
by • New Contributor
  • 1531 Views
  • 1 replies
  • 0 kudos

Does Databricks provide a mechanism to have rate limiting for receivers?

from pyspark.sql import SparkSession   scala_version = '2.12' spark_version = '3.3.0'   packages = [ f'org.apache.spark:spark-sql-kafka-0-10_{scala_version}:{spark_version}', 'org.apache.kafka:kafka-clients:3.2.1' ]   spark = SparkSession.bui...

Data Engineering
Reply
  • 1531 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Rajani
Contributor II
  • 0 kudos

hi @Software Architect​  i dont think so

  • 0 kudos
Pranjan
by • New Contributor II
  • 3092 Views
  • 7 replies
  • 1 kudos

Resolved! Badge Not Received for - Databricks Lakehouse Fundamentals Accreditation (V2)

Hi TeamI have passed the Databricks Lakehouse Fundamentals Accreditation (V2) on Dec 8th.Still have not received the Badge in credentials or any email of that kind.Please have a look.@Kaniz Fatma​ ​ 

Data Engineering
Reply
  • 3092 Views
  • 7 replies
  • 1 kudos
View Replies
Latest Reply
Tromen026
New Contributor II
  • 1 kudos

I wonder how much attempt you set to create this type of excellent informative web site.marco's pizza starr avedomino's pizza price

  • 1 kudos
6 More Replies
Smitha1
by • Valued Contributor II
  • 1856 Views
  • 3 replies
  • 2 kudos

December exam free voucher for Databricks Certified Associate Developer for Apache Spark 3.0 exam.

Dear @Vidula Khanna​  Hope you're having great day. This is of HIGH priority for me, I've to schedule exam in December before slots are full.I gave Databricks Certified Associate Developer for Apache Spark 3.0 exam on 30th Nov but missed by one perc...

Data Engineering
Reply
  • 1856 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

hey @Smitha Nelapati​ ,you can attend the below webinars and get the 75% off in Jan ​ 

  • 2 kudos
2 More Replies
KasimData
by • New Contributor III
  • 2666 Views
  • 3 replies
  • 6 kudos

Unable to signup to a Databricks community edition account

As you can see, I get the error underneath the big orange button. This is after I click the link at the bottom to try the community edition. I have tried a couple of locations since I am currently based in South Korea but I am actually from the UK. T...

image.png
Data Engineering
Reply
  • 2666 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Muhammad Ali​ Just a friendly follow-up. Are you able to log in to your Community-Edition account? If yes, then mark the answer as best or if you need further assistance kindly let me know. Thanks and Regards

  • 6 kudos
2 More Replies
sudhanshu1
by • New Contributor III
  • 2490 Views
  • 1 replies
  • 0 kudos

Write streaming output to DynamoDB

Hi All,I am trying to write a streaming DF into dynamoDB with below code.tumbling_df.writeStream \  .format("org.apache.spark.sql.execution.streaming.sinks.DynamoDBSinkProvider") \  .option("region", "eu-west-2") \  .option("tableName", "PythonForeac...

Data Engineering
Reply
  • 2490 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
LandanG
Databricks Employee
  • 0 kudos

Hi @SUDHANSHU RAJ​ ,I can't seem to find much on the "DynamoDBSinkProvider" source. Have you checked out the link for the streaming to DynamoDB documentation?

  • 0 kudos
Chris_Shehu
by • Valued Contributor III
  • 2802 Views
  • 3 replies
  • 3 kudos

Resolved! Is there a way to specify a header, set the delimiter, etc...in DLT?

I was looking forward to using the Data Quality features that are provided with DLT but as far as I can the ingestion process is more restrictive than other methods. It doesn't seem like you can do much as far as setting delimiter type, headers or an...

Data Engineering
Reply
  • 2802 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

DLT uses Autoloader to ingest data. With autoloader, you can provide read options for the table. https://docs.databricks.com/ingestion/auto-loader/options.html#csv-options has the docs on CSV. I attached a picture of an example.

  • 3 kudos
2 More Replies
pkgltn
by • New Contributor III
  • 2088 Views
  • 2 replies
  • 2 kudos

Resolved! Load an Excel File (located in Databricks Repo connected to Azure DevOps) into a dataframe

Hi, How can I load an Excel File (located in Databricks Repo connected to Azure DevOps) into a dataframe? When I pass the full path into the load method, it displays an error.java.io.FileNotFoundException Has someone done it previously?

Data Engineering
Reply
  • 2088 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
pkgltn
New Contributor III
  • 2 kudos

Hi,Just managed to do it.Upgraded the cluster to the latest version because Files in Repos only works in most recent versions of the cluster.When loading the dataframe, specify the path as follows: file:/Workspace/Repos/user@email.com/filepath/filena...

  • 2 kudos
1 More Replies
hf_santos
by • New Contributor III
  • 6952 Views
  • 4 replies
  • 2 kudos

Resolved! Error when importing PyDeequ package

Hi everyone,I want to do some tests regarding data quality and for that I pretend to use PyDeequ on a databricks notebook. Keep in mind that I'm very new to databricks and Spark.First I created a cluster with the Runtime version "10.4 LTS (includes A...

Data Engineering
Reply
  • 6952 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
hf_santos
New Contributor III
  • 2 kudos

I assumed I wouldn't need to add the Deequ library. Apparently, all I had to do was add it via Maven coordinates and it solved the problem.

  • 2 kudos
3 More Replies
db-avengers2rul
by • Contributor II
  • 1348 Views
  • 1 replies
  • 0 kudos

Jupyter notebooks import in databricks notebooks

Dear Team,Is it possible to import jupyter notebooks in databricks community edition ? if yes will there be any formatting issues ?

Data Engineering
Reply
  • 1348 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
db-avengers2rul
Contributor II
  • 0 kudos

if yes is there any limit ? , what is the difference or advantage using juypter notebooks over databricks notebooks

  • 0 kudos
db-avengers2rul
by • Contributor II
  • 1192 Views
  • 2 replies
  • 0 kudos

What is the underlying database used in data bricks community edition in sql

Dear DB Experts,I am reaching out to check whether can i still use postgressql in notebooks with notebook as sql and try postgresql , as far as i know from reading the back end db is mysql correct me my understanding ?

Data Engineering
Reply
  • 1192 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
db-avengers2rul
Contributor II
  • 0 kudos

are there any setting i have to update ?

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels