- 9028 Views
- 1 replies
- 0 kudos
DLT SQL
What is best practise to implement parameterization in SQL DLT (specifically) pipelines so that it's easy and no manual intervention would be potentially required to migrate from dev_region to prod_region
- 9028 Views
- 1 replies
- 0 kudos
- 0 kudos
I would love to see a sample implementation of this config table.
- 0 kudos
- 11285 Views
- 3 replies
- 0 kudos
Unable to use unity catalog in notebook
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: org.apache.spark.sql.connector.catalog.CatalogNotFoundException: Catalog 'uc-dev' plugin class not found: spark.sql.catalog.uc-dev is not defined ....I get the above when ...
- 11285 Views
- 3 replies
- 0 kudos
- 0 kudos
I had the same error plugin class not found: spark.sql.catalog... is not defined immediately after attaching the workspace into Unity catalog.The error was resolved by restarting SQL Warehouse.It seems that if SQL Warehouse (or any cluster) is runnin...
- 0 kudos
- 12743 Views
- 9 replies
- 5 kudos
Resolved! pyspark dataframe empties after it has been saved to delta lake.
Hi, I am facing a problem that I hope to get some help to understand. I have created a function that is supposed to check if the input data already exist in a saved delta table and if not, it should create some calculations and append the new data to...
- 12743 Views
- 9 replies
- 5 kudos
- 5 kudos
Hi,im also having similar issue ..does creating temp view and reading it again after saving to a table works?? /
- 5 kudos
- 14404 Views
- 3 replies
- 3 kudos
Resolved! What's the best practice on running ANALYZE on Delta Tables for query performance optimization?
- 14404 Views
- 3 replies
- 3 kudos
- 3 kudos
Super write-up; very useful in understanding how the Delta and non-Delta approaches have evolved.
- 3 kudos
- 33806 Views
- 10 replies
- 5 kudos
Not able to read text file from local file path - Spark CSV reader
We are using Spark CSV reader to read the csv file to convert as DataFrame and we are running the job on yarn-client, its working fine in local mode. We are submitting the spark job in edge node. But when we place the file in local file path instead...
- 33806 Views
- 10 replies
- 5 kudos
- 2330 Views
- 1 replies
- 0 kudos
Databricks Connection to Redash
Hello,I am trying to connect my Redash account with Databricks so that my organization can run queries on the data in Unity Catalog from Redash.I followed through the steps in the documentation and managed to connect successfully. However, I am only ...
- 2330 Views
- 1 replies
- 0 kudos
- 0 kudos
it looks like the Redash connector for Databricks is hard-coded to run `SHOW DATABASES`, which only shows `hive_metastore` by default. This probably needs to be updated to run `SHOW CATALOGS` and then `SHOW SCHEMAS in <catalog_name>` for each of thos...
- 0 kudos
- 7546 Views
- 6 replies
- 0 kudos
Possible false positive warning on DLT pipeline
I have a DLT pipeline script that starts by extracting metadata on the tables it should generate from a delta table. Each record returned from the table should be a dlt table to generate, so I use .collect() to turn each row into a list and then iter...
- 7546 Views
- 6 replies
- 0 kudos
- 0 kudos
Thanks for the reply. Based on that response though, it seems like the warning itself is a bug in the DLT implementation. Per the docs "However, you can include these functions outside of table or view function definitions because this code is run on...
- 0 kudos
- 2536 Views
- 0 replies
- 0 kudos
Delta table cannot be reached with INTERNAL_ERROR
Hi all!I've been dropping and recreating delta tables at the new location. For one table something went wrong and now I cannot nor DROP nor recreate it. It is visible in catalog, however, when I click on the table I see message: [INTERNAL_ERROR] The ...
- 2536 Views
- 0 replies
- 0 kudos
- 1688 Views
- 1 replies
- 0 kudos
How to extend free trial period or enter free startup tier to complete our POC for a client.
We are a data consultancy. Our free trial period is currently getting over and we are still doing POC for one of our potential clients and focusing on providing expert services around databricks.1. Is there a possibility that we can extend the free t...
- 1688 Views
- 1 replies
- 0 kudos
- 0 kudos
hey @ashraf1395, I suggest you contact your databricks representative or account manager.
- 0 kudos
- 2452 Views
- 3 replies
- 0 kudos
CICD for Work Flows
HiI am facing issues when deploying work flows to different environment. The same works for Notebooks and Scripts, when deploying the work flows, it failed with "Authorization Failed. Your token may be expired or lack the valid scope". Anything shoul...
- 2452 Views
- 3 replies
- 0 kudos
- 2560 Views
- 1 replies
- 0 kudos
Is it possible to migrate data from one DLT pipeline to another?
Hi,We have a DLT pipeline that has been running for a while with a Hive Metastore target that has stored billions of records. We'd like to move the data to a Unity Catalog. The documentation says "Existing pipelines that use the Hive metastore cannot...
- 2560 Views
- 1 replies
- 0 kudos
- 0 kudos
@MarkD good day! I'm sorry, but according to the description, existing pipelines using the Hive metastore cannot be upgraded to use Unity Catalog. To migrate an existing pipeline that writes to Hive metastore, you must create a new pipeline and re-in...
- 0 kudos
- 6929 Views
- 3 replies
- 3 kudos
Resolved! Single-Node cluster works but Multi-Node clusters do not read data.
I am currently working with a VNET injected databricks workspace. At the moment I have mounted a the databricks cluster on an ADLS G2 resource. When running notebooks on a single node that read, transform, and write data we do not encounter any probl...
- 6929 Views
- 3 replies
- 3 kudos
- 3 kudos
@TheDataDexter Did you find a solution to your problem? I am facing the same issue
- 3 kudos
- 19170 Views
- 3 replies
- 0 kudos
Databricks: Change the existing schema of columns to non-nullable for a delta table using Pyspark?
Hello,I have currently a delta folder as a table with several columns that are nullable. I want to migrate data to the table and overwrite the content using Pyspark, add several new columns and make them not nullable. I have found a way to make the c...
- 19170 Views
- 3 replies
- 0 kudos
- 0 kudos
Not sure if you found a solution, you can also try as below. In this case you pass the full path to the delta not the table itself.spark.sql(f"ALTER TABLE delta.`{full_delta_path}` ALTER column {column_name} SET NOT NULL")
- 0 kudos
- 3104 Views
- 1 replies
- 1 kudos
delta live tables - oracle connectivity
Is delta live tables/pipelines support oracle or external database connectivity ? i am getting Oracle Driver not found error. dlt not supporting maven install through asset bundles. ERRORs: 1) py4j.protocol.Py4JJavaError: An error occurred while call...
- 3104 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @venkata_kishore , As of now, DLT does not support Oracle, and one cannot install third-party libraries and JARs. https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#limitationsIf Lakehouse Federation has support for Oracle, then ...
- 1 kudos
- 1512 Views
- 0 replies
- 0 kudos
topology, centralized, and federated workspaces
Hi Team,Can you provide information about topology, centralized, and federated workspaces in databricks and how they are used?Regards,Janga
- 1512 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
