- 3120 Views
- 6 replies
- 6 kudos
DBFS File browser option is not visible in Databricks Community Edition
I am not able to see the DBFS File Browser option to enable it. Can someone help how to get that option?
- 3120 Views
- 6 replies
- 6 kudos
- 688 Views
- 2 replies
- 0 kudos
Need help with DLT
I have a DLT pipeline on databricks that has been running since months and just now I found out that there has been an issue with a logic in silver layer and as a result, the tables in my silver schema has faulty records now. Silver layer tables are ...
- 688 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Fatimah-Tariq ,What about defining your DLT pipeline as below.This way you will create a stream table that reads from your silver, apply all the needed changes and then write back to your silver.%sql -- Read from the streaming table in the silver...
- 0 kudos
- 4032 Views
- 6 replies
- 1 kudos
Resolved! Delta table in catalogue are showing but DO NOT exist
I am just working with Databricks, and have come across an issue where delta tables have been created in the catalogue but do not actually exist. see screenshot for script ive been running and error messagesis this a bug, or am i missing something o...
- 4032 Views
- 6 replies
- 1 kudos
- 1 kudos
@RobDineenCan you try to refresh your table (REFRESH TABLE your_catalog.your_schema.your_table) and followed by spark.catalog.clearCache().Then try the drop operation: table_path = "dbfs:/user/hive/warehouse/season" dbutils.fs.rm(table_path, recurse=...
- 1 kudos
- 1458 Views
- 1 replies
- 2 kudos
Programatically remove external table completely (Azure)
Hi!I have created external tables with data stored in an Azure storage account.Is there a way to not only drop the tables but also remove the underlying folder in the storage account which contains the table's data? I want to do this from Databricks ...
- 1458 Views
- 1 replies
- 2 kudos
- 2 kudos
@johnb1 You can achieve this with the code below. Please review.table_name = "table_name" location = "abfss://container@storage-account.dfs.core.windows.net/path/to/table/data/" spark.sql(f"DROP TABLE IF EXISTS {table_name}") dbutils.fs.rm(location...
- 2 kudos
- 1673 Views
- 2 replies
- 0 kudos
read_files to create streaming table using Databricks SQL
I follow the same syntax from documentation for create streaming table and it was last week and not working nowEx query:CREATE OR REFRESH STREAMING TABLE`ax`.`db`.`healthex`AS SELECT * FROM STREAMread_files("/Volumes/ax/db/dlt-test/", -- The file pat...
- 1673 Views
- 2 replies
- 0 kudos
- 0 kudos
I followed the syntax in this documentation link https://docs.databricks.com/en/tables/streaming.htmlI get this error if the STREAM keyword is excluded Please add the STREAM keyword to your FROM clause to turn this relation into a streaming query. SQ...
- 0 kudos
- 8502 Views
- 9 replies
- 3 kudos
Resolved! How to enable Delta live tables serverless in Databricks?
I am trying to enable the Serverless mode in the Delta Live Tables, based on what the official Databricks channel YouTube video "Delta Live Tables A to Z: Best practices for Modern Data Pipelines".And I cannot find it in my UI. Could you help me with...
- 8502 Views
- 9 replies
- 3 kudos
- 3 kudos
Hey Kols,Unity catalog is enabled as well, as you can see in the screenshot. To open a support ticket I need dedicated Databricks support which our company doesn't have.
- 3 kudos
- 2269 Views
- 5 replies
- 2 kudos
Resolved! bundle run fails with [UC_NOT_ENABLED]
Hi all,I am trying to run my first databricks asset bundle but it always fails with [UC_NOT_ENABLED] Unity Catalog is not enabled on this cluster. SQLSTATE: 56038 Does anyone ecountered this error when running asset bundles or does anyone know how t...
- 2269 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @T0M ,Do you have Unity Catalog enabled workspace? The error indicates that you're trying to use DAB on workspace without UC enabled.
- 2 kudos
- 2053 Views
- 2 replies
- 1 kudos
Databricks and oracle
Does databricks has an oracle connector or it uses odbc?
- 2053 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, the DataBricks connector has been added to the Oracle Analytics Product Roadmap.You can find more information in the Oracle Analytics Community and ask questions.
- 1 kudos
- 2601 Views
- 3 replies
- 0 kudos
SparkRuntimeException: Sent message larger than max (10701549 vs. 10485760)
Hey Databricks team,I have been facing a weird error when i upgrade to use Unity Catalog. Actually where is the limit 10485760 (10MB) coming from?I have spark.sql.autoBroadcastJoinThreshold set to -1 already, and I can't find out any other spark conf...
- 2601 Views
- 3 replies
- 0 kudos
- 7023 Views
- 4 replies
- 2 kudos
Databricks asset bundles passing parameters using bundle run which are not declared
Hi,We recently decided to move to databricks asset bundles, one scenario that we are dealing with is we have different parameters passed to the same job which are handled in the notebook. With bundles when I try to pass parameters at runtime(which ar...
- 7023 Views
- 4 replies
- 2 kudos
- 2 kudos
Following syntax can be used.databricks bundle run -t ENV --params Param1=Value1,Param2=Value2 Job_NameJob Definition Parameter may look like this.
- 2 kudos
- 789 Views
- 2 replies
- 0 kudos
Hi i have uploaded excel file in databricks but it shows different language.
Hi,I have attach one excel file in data bricks edition but unfortunately it shows a diiferent langaue in ouput whice i wrote display(df). below im attaching the screenshot please let me now thanking you in advance.
- 789 Views
- 2 replies
- 0 kudos
- 0 kudos
CSV and Excel are not the same datatype. You can load the excel data into a pandas dataframe and then convert it to a pyspark dataframe.first, you have to install the openpyxl library %pip install openpyxl Then import PySpark Pandas: import pyspark.p...
- 0 kudos
- 2381 Views
- 1 replies
- 0 kudos
How Install Pyrfc into AWS Databrick using Volumes
I'm trying to install Pyrfc in a Databricks Cluster (already tried in r5.xlarge, m5.xlarge, and c6gd.xlarge). I'm following these link.https://community.databricks.com/t5/data-engineering/how-can-i-cluster-install-a-c-python-library-pyrfc/td-p/8118Bu...
- 2381 Views
- 1 replies
- 0 kudos
- 0 kudos
More details about the errorLibrary installation attempted on the driver node of cluster 0000-000000-00000 and failed. Please refer to the following error message to fix the library or contact Databricks support. Error code: DRIVER_LIBRARY_INSTALLATI...
- 0 kudos
- 4361 Views
- 4 replies
- 4 kudos
Resolved! Database creation error
I am trying to create database with external location abfss but facing the below error.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs....
- 4361 Views
- 4 replies
- 4 kudos
- 4 kudos
Changing it to a CLUSTER level for OAuth authentication helped me solve the problem.I wish the notebook AI bot could tell me the solution.before the changes, my configraiotn was at the notebook leve.and it has below errorsAnalysisException: org.apac...
- 4 kudos
- 957 Views
- 4 replies
- 4 kudos
Resolved! code execution from Databrick folder
We are able to run a notebook that references Python code using import statements from a Databricks repowith the source code checked out. However, we encounter a ModuleNotFoundError when executing the same code from a folder.Error: ModuleNotFoundErro...
- 957 Views
- 4 replies
- 4 kudos
- 4 kudos
Below worked as you have suggested.import os, sysproject_path = os.path.abspath("/Workspace/<folder-name-1>/<folder-name-2>/<top-level-code-folder>")if project_path not in sys.path: sys.path.append(project_path)
- 4 kudos
- 1287 Views
- 3 replies
- 0 kudos
S3 Data access through unity
Hey All I have the following issue : I've connected a s3 bucket through unity catalog as an external source. I perfectly see the files of my s3 bucket when i scroll through the catalog using the user interface. However when I try to connect through a...
- 1287 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey Chandra thank you for your answer. The path is a volume path indeed, : /Volumes/my_path_in_volume
- 0 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
