cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Red_blue_green
by New Contributor III
  • 4292 Views
  • 3 replies
  • 0 kudos

Databricks: Change the existing schema of columns to non-nullable for a delta table using Pyspark?

Hello,I have currently a delta folder as a table with several columns that are nullable. I want to migrate data to the table and overwrite the content using Pyspark, add several new columns and make them not nullable. I have found a way to make the c...

  • 4292 Views
  • 3 replies
  • 0 kudos
Latest Reply
kanjinghat
New Contributor II
  • 0 kudos

Not sure if you found a solution, you can also try as below. In this case you pass the full path to the delta not the table itself.spark.sql(f"ALTER TABLE delta.`{full_delta_path}` ALTER column {column_name} SET NOT NULL") 

  • 0 kudos
2 More Replies
venkata_kishore
by New Contributor
  • 969 Views
  • 1 replies
  • 1 kudos

delta live tables - oracle connectivity

Is delta live tables/pipelines support oracle or external database connectivity ? i am getting Oracle Driver not found error. dlt not supporting maven install through asset bundles. ERRORs: 1) py4j.protocol.Py4JJavaError: An error occurred while call...

Data Engineering
Delta Live Tables
dlt
oracle
pipelines
  • 969 Views
  • 1 replies
  • 1 kudos
Latest Reply
RamGoli
New Contributor II
  • 1 kudos

Hi @venkata_kishore  , As of now, DLT does not support Oracle, and one cannot install third-party libraries and JARs. https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#limitationsIf Lakehouse Federation has support for Oracle, then ...

  • 1 kudos
dprutean
by New Contributor III
  • 795 Views
  • 1 replies
  • 0 kudos

JDBC Driver Error

connect to databricks unity catalog and meet this error java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 59. Background thread ID: 61. Error caught: null. at com.databricks.cli...

  • 795 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @dprutean, Thank you for providing the details about the error you’re encountering while connecting to the Databricks Unity Catalog using the Databricks JDBC driver. Let’s troubleshoot this step by step: Check your connection string: The conn...

  • 0 kudos
AdityaM
by New Contributor II
  • 779 Views
  • 2 replies
  • 0 kudos

Creating external tables using gzipped CSV file - S3 URI without extensions

Hi Databricks community,Hope you are doing well.I am trying to create an external table using a Gzipped CSV file uploaded to an S3 bucket.The S3 URI of the resource doesn't have any file extensions, but the content of the file is a Gzipped comma sepa...

  • 779 Views
  • 2 replies
  • 0 kudos
Latest Reply
AdityaM
New Contributor II
  • 0 kudos

Hey  , thanks for your response. I tried using a Serde(I think the OpenCSVSerde should work for me) but unfortunately im getting the below from the Unity Catalog:[UC_DATASOURCE_NOT_SUPPORTED] Data source format hive is not supported in Unity Catalog....

  • 0 kudos
1 More Replies
KrishnaK135
by New Contributor
  • 453 Views
  • 1 replies
  • 0 kudos

Advanced Data Engineering

Just finished the final day of training. Great content and delivery!

  • 453 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @KrishnaK135, That's wonderful to hear! We're thrilled that you found the content and delivery of the training at DAIS 2023 to be excellent. Your positive feedback means a lot to us! We also wanted to share some exciting news with you all. The Dat...

  • 0 kudos
Databricks143
by New Contributor III
  • 7749 Views
  • 7 replies
  • 0 kudos

Recrusive cte in databrick sql

Hi Team,How to write recrusive cte in databricks SQL.Please let me know any one have solution for this 

  • 7749 Views
  • 7 replies
  • 0 kudos
Latest Reply
EWhitley
New Contributor III
  • 0 kudos

Weirdly, if you ask the Databricks assistant, it tells you it does support recursive CTE and will give you sample code for it. If you follow up and press it on details it doubles down and says it supports it, but tell you the runtime version informat...

  • 0 kudos
6 More Replies
JoshGallant
by New Contributor
  • 438 Views
  • 1 replies
  • 0 kudos

Delta Live Tables Best Practices

Delta Live Tables Best Practices session today was fantastic.

  • 438 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @JoshGallant, We're thrilled to hear that you found the Delta Live Tables Best Practices session fantastic! Your enthusiasm and engagement at DAIS 2023 were truly appreciated. We wanted to let you know that the Databricks Community team will be ba...

  • 0 kudos
EricIto
by New Contributor
  • 385 Views
  • 1 replies
  • 0 kudos

Data Engineering Associate Training

Props to the training team for the Data Engineering track at AI Summit, learned a ton and got some great experience!

  • 385 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @EricIto, We're thrilled to hear that you had such a positive experience at DAIS 2023, especially with the Data Engineering track! Your feedback is greatly appreciated, and we're glad that you found the training valuable. We're excited to announce...

  • 0 kudos
Peston
by New Contributor
  • 1952 Views
  • 1 replies
  • 0 kudos

DataAI Summit

Having a great time at the summit and learning about the advances with AI. The advantages of adding the ability to use english as a new programming language and the advantage that it will bring to the companies that adapt to the future of Databricks....

  • 1952 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Peston, We're thrilled to hear that you had a great experience at the summit and found the sessions on AI advancements and the use of English as a new programming language insightful. It's exciting to see the potential benefits these advancements...

  • 0 kudos
MohammadWasi
by New Contributor II
  • 924 Views
  • 6 replies
  • 1 kudos

i can list out the file using dbutils but can not able to read files in databricks

I can list out the file using dbutils but can not able to read files in databricks. PFB in screenshot. I can able to see the  file using dbutils.fs.ls but when i try to read this file using read_excel then it is showing me an error like "FileNotFound...

MohammadWasi_0-1715064354700.png
Data Engineering
Databricks
  • 924 Views
  • 6 replies
  • 1 kudos
Latest Reply
MohammadWasi
New Contributor II
  • 1 kudos

Hi @Kaniz_Fatma I changed file in .xlsx format but again got the same error as above.

  • 1 kudos
5 More Replies
ashraf1395
by Contributor
  • 1106 Views
  • 5 replies
  • 4 kudos

Resolved! Starting Serverless sql cluster on GCP

Hello there,I am trying to start a serverless databricks SQL cluster in GCP. I am following this databricks doc: https://docs.gcp.databricks.com/en/admin/sql/serverless.htmlI have checked that all my requirements are fulfilled for activating the clus...

Screenshot 2024-05-07 113120.png Screenshot 2024-05-07 113137.png
  • 1106 Views
  • 5 replies
  • 4 kudos
Latest Reply
ashraf1395
Contributor
  • 4 kudos

I had another question. Though not related to this thread.Do databricks has any plan for startups, like they have normal free trial

  • 4 kudos
4 More Replies
Keyurishah_
by New Contributor
  • 527 Views
  • 1 replies
  • 0 kudos

Excited for lakehouse iq

So excited to get my hands on lakehouse iq

Data Engineering
Summit 2023
  • 527 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Keyurishah_, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Communit...

  • 0 kudos
mickniz
by Contributor
  • 1134 Views
  • 3 replies
  • 0 kudos

Connect to Databricks from PowerApps

Hi All,Currently I trying to connect databricks Unity Catalog from Powerapps Dataflow by using spark connector specifying http url and using databricks personal access token as specified in below screenshot: I am able to connect but the issue is when...

mickniz_0-1714487746554.png mickniz_1-1714487891958.png
  • 1134 Views
  • 3 replies
  • 0 kudos
Latest Reply
mickniz
Contributor
  • 0 kudos

thanks for replying @Kaniz_Fatma . I am using my user person token to connect and I have all access on catalog,schema and tables. I am able to view in Databricks Sql editor but not via Spark connector in power apps. It still shows me Is there anythin...

  • 0 kudos
2 More Replies
Nastia
by New Contributor III
  • 583 Views
  • 1 replies
  • 0 kudos

Resolved! I am getting NoneType error when running a query from API on cluster

When I am running a query on Databricks itself from notebook, it is running fine and giving me results. But the same query when executed from FastAPI (Python, using databricks library) is giving me "TypeError: 'NoneType' object is not iterable".I can...

  • 583 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Nastia, The “TypeError: ‘NoneType’ object is not iterable” error typically occurs when you try to iterate over a variable that has a value of None. Let’s explore some possible solutions to address this issue: Check for None before Iterating: ...

  • 0 kudos
jitesh
by New Contributor
  • 341 Views
  • 1 replies
  • 0 kudos

Code reusability for silver table transformations

How/how many databricks notebooks should be created to populate multiple silver delta tables, all having different and complex transformations ? What's the best practice -1. create a notebook each for a silver table ?2. push SQL transformation logic ...

  • 341 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @jitesh, When organizing your Databricks Notebooks for multiple silver Delta tables with different and complex transformations, it’s essential to follow best practices. Here are some recommendations: Separate Notebooks for Each Layer: Bronze L...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels