- 327 Views
- 8 replies
- 0 kudos
I need help with migrating from dbfs on databricks to workspace. I am new to databricks and am struggling with what is on the links provided.My workspace.yml also has dbfs hard-coded. Included is a full deployment with great expectations.This was don...
- 327 Views
- 8 replies
- 0 kudos
Latest Reply
One of the other suggestions is to use Lakehouse Federation. It is possible it may be a driver issue (we will get to know from the logs)
7 More Replies
- 2013 Views
- 3 replies
- 0 kudos
Hello,I have currently a delta folder as a table with several columns that are nullable. I want to migrate data to the table and overwrite the content using Pyspark, add several new columns and make them not nullable. I have found a way to make the c...
- 2013 Views
- 3 replies
- 0 kudos
Latest Reply
Not sure if you found a solution, you can also try as below. In this case you pass the full path to the delta not the table itself.spark.sql(f"ALTER TABLE delta.`{full_delta_path}` ALTER column {column_name} SET NOT NULL")
2 More Replies
- 61 Views
- 1 replies
- 1 kudos
Is delta live tables/pipelines support oracle or external database connectivity ? i am getting Oracle Driver not found error. dlt not supporting maven install through asset bundles. ERRORs: 1) py4j.protocol.Py4JJavaError: An error occurred while call...
- 61 Views
- 1 replies
- 1 kudos
Latest Reply
Hi @venkata_kishore ,
As of now, DLT does not support Oracle, and one cannot install third-party libraries and JARs.
https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#limitationsIf Lakehouse Federation has support for Oracle, then ...
- 36 Views
- 0 replies
- 0 kudos
Hi,We have a DLT pipeline that has been running for a while with a Hive Metastore target that has stored billions of records. We'd like to move the data to a Unity Catalog. The documentation says "Existing pipelines that use the Hive metastore cannot...
- 36 Views
- 0 replies
- 0 kudos
by
Phani1
• Valued Contributor
- 26 Views
- 0 replies
- 0 kudos
Hi Team,Can you provide information about topology, centralized, and federated workspaces in databricks and how they are used?Regards,Janga
- 26 Views
- 0 replies
- 0 kudos
- 76 Views
- 0 replies
- 1 kudos
Hello,We are using delta live tables to ingest data from multiple business groups, each with different input file formats and parsing requirements. The input files are ingested from azure blob storage. Right now, we are only servicing three busines...
- 76 Views
- 0 replies
- 1 kudos
- 452 Views
- 5 replies
- 4 kudos
When running my notebook using personal compute with instance profile I am indeed able to readStream from kinesis. But adding it as a DLT with UC, while specifying the same instance-profile in the DLT pipeline setting - causes a "MissingAuthenticatio...
- 452 Views
- 5 replies
- 4 kudos
Latest Reply
@Mathias_Peters , Thanks for the details. Curious how make the roleAan part work , we are able to make it work only with passing accessKey and Secret key, not with roleArn. if you are using SQL based DLT tables , Could you please share some code samp...
4 More Replies
- 578 Views
- 1 replies
- 0 kudos
connect to databricks unity catalog and meet this error java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 59. Background thread ID: 61. Error caught: null. at com.databricks.cli...
- 578 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @dprutean, Thank you for providing the details about the error you’re encountering while connecting to the Databricks Unity Catalog using the Databricks JDBC driver.
Let’s troubleshoot this step by step:
Check your connection string:
The conn...
- 129 Views
- 2 replies
- 0 kudos
Hi Databricks community,Hope you are doing well.I am trying to create an external table using a Gzipped CSV file uploaded to an S3 bucket.The S3 URI of the resource doesn't have any file extensions, but the content of the file is a Gzipped comma sepa...
- 129 Views
- 2 replies
- 0 kudos
Latest Reply
Hey , thanks for your response. I tried using a Serde(I think the OpenCSVSerde should work for me) but unfortunately im getting the below from the Unity Catalog:[UC_DATASOURCE_NOT_SUPPORTED] Data source format hive is not supported in Unity Catalog....
1 More Replies
- 237 Views
- 1 replies
- 0 kudos
Just finished the final day of training. Great content and delivery!
- 237 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @KrishnaK135, That's wonderful to hear! We're thrilled that you found the content and delivery of the training at DAIS 2023 to be excellent. Your positive feedback means a lot to us!
We also wanted to share some exciting news with you all. The Dat...
- 4447 Views
- 7 replies
- 0 kudos
Hi Team,How to write recrusive cte in databricks SQL.Please let me know any one have solution for this
- 4447 Views
- 7 replies
- 0 kudos
Latest Reply
Weirdly, if you ask the Databricks assistant, it tells you it does support recursive CTE and will give you sample code for it. If you follow up and press it on details it doubles down and says it supports it, but tell you the runtime version informat...
6 More Replies
- 262 Views
- 1 replies
- 0 kudos
Delta Live Tables Best Practices session today was fantastic.
- 262 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @JoshGallant, We're thrilled to hear that you found the Delta Live Tables Best Practices session fantastic!
Your enthusiasm and engagement at DAIS 2023 were truly appreciated. We wanted to let you know that the Databricks Community team will be ba...
- 283 Views
- 1 replies
- 0 kudos
Props to the training team for the Data Engineering track at AI Summit, learned a ton and got some great experience!
- 283 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @EricIto, We're thrilled to hear that you had such a positive experience at DAIS 2023, especially with the Data Engineering track! Your feedback is greatly appreciated, and we're glad that you found the training valuable.
We're excited to announce...
- 556 Views
- 1 replies
- 0 kudos
Having a great time at the summit and learning about the advances with AI. The advantages of adding the ability to use english as a new programming language and the advantage that it will bring to the companies that adapt to the future of Databricks....
- 556 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Peston, We're thrilled to hear that you had a great experience at the summit and found the sessions on AI advancements and the use of English as a new programming language insightful. It's exciting to see the potential benefits these advancements...
- 82 Views
- 0 replies
- 1 kudos
I have a storage account dexflex and two containers source and destination. Source container has directory and files as below:results
search
03
Module19111.json
Module19126.json
04
Module11291...
- 82 Views
- 0 replies
- 1 kudos