Hello,I am trying to upload the file from local linux machine to dbfs using request.post(<URI>,<Headers>, params={"path": "dbfs:/tmp", "contents": local_path}) and getting the error b'{"error_code":"INVALID_PARAMETER_VALUE","message":"You must provid...
We've had users make comments on tables/columns throughout Databricks SQL using the Data Explorer UI. I'm looking for a way to backup these comments, but when I run DESCRIBE TABLE, the comment column is always null despite being non-null in Data Expl...
@804082 - Markdown does not render when returned by DESCRIBE statements. we can view them on the Data Explorer UI.Reference: https://docs.databricks.com/en/data/markdown-data-comments.html#document-data-with-markdown-comments
/api/2.1/jobs/runs/list Currently returns all jobs which got executed within specified time lines which we provide as input. Is There any way where we can get only specific jobs as per their status(only success)?
@venkat94 thanks for the feedback. We are working on updating the Jobs Runs API so you can filter runs by status e.g. only success. Stay tuned in the next couple of months.
I am using DLT and unity catalog and using managed tables. The first table in this pipeline is a live streaming table. I first did this in the SQL editor:DELETE FROM my_table;This appears to have deleted all the records, which I wanted since now when...
Hi guys I have a question regarding this merge step and I am a new beginner for Databricks, trying to do some study in data warehousing, but couldn't figure it out by myself. need your help with it. Appreciate your help in advance. I got this questi...
When I try to use a third party JAR on an Azure shared cluster - which is installed via Maven and I can successfully import - , I get the following message: py4j.security.Py4JSecurityException: Method public static org.apache.spark.sql.Column com.da...
Thanks Kaniz.I must use a shared cluster because I'm reading from a DLT table stored in a Unity Catalog.https://docs.databricks.com/en/data-governance/unity-catalog/compute.htmlMy understanding is that shared clusters are enforcing the Py4J policy I ...
I try to build a DLT in UC with Kinesis as producer.My first table looks like: @dlt.create_table( table_properties={ "pipelines.autoOptimize.managed": "true" }, spark_conf={"spark.databricks.delta.schema.autoMerge.enabled": "true"},)def feed_chu...
If you use the "Preview" Channel in the "Advanced" section of the DLT Pipeline, this error should resolve itself. This fix is planned to make it into the "Current" channel by Aug 31, 2023
I have a DLT that runs every day and an automatically executed maintenance job that runs within 24 hours every day. The maintenance operations are costly, is it possible to change the schedule to once a week or so?
In my DLT pipeline outlined below which generically cleans identifier tables, after successfully creating initial streaming tables from the append-only sources, fails when trying to create the second cleaned tables witht the following:It'**bleep** cl...
Hi @scvbelle The error message you're seeing is caused by an IllegalArgumentException error due to the restriction in Azure Blob File System (ABFS) that does not allow files or directories to end with a dot. This error is thrown by the trailingPeriod...
Dear Databricks Expert,I got some doubts when dealing with DBFS and Local File System.Case01: Copy a file from ADLS to DBFS. I am able to do so through the below python codes:#spark.conf.set("fs.azure.account.auth.type", "OAuth") spark.conf.set("fs.a...
Hi @KS LAU​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your q...
I am trying to follow along Apache Spark Programming training module where the instructor creates events table from a parquet file like this:%sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/mnt/training/ecommerce/events/events.par...
@Retired_mod Thanks for your response. I didn't provide cloud file system scheme in the path while creating the table using DataFrame API, but I was still able to create the table. %python # File location and type file_location = "/mnt/training/ecom...
If I try to create a Volume, I get this error:Failed to access cloud storage: AbfsRestOperationException exceptionTraceId=fa207c57-db1a-406e-926f-4a7ff0e4afddWhen i try to create a table, I get this error:Error creating table[RequestId=4b8fedcf-24b3-...
How do I change my Databricks Community user name?
Hi Sujitha,Thanks for the quick help! I cleaned the caches, and it works now.
I have a few Databricks clusters, some share a single Hive Metastore (HMS), call them PROD_CLUSTERS, and an additional cluster, ADHOC_CLUSTER, which has its own HMS. All my data is stored in S3, as Databricks delta tables: PROD_CLUSTERS have read-wri...
Something went wrong there, here's the last sentence: I expected "location" will be the s3 path but it's not always so (elaborated in the original posting). Thanks!
We are in the process of implementing data mesh in our organization. When trying to help the different teams produce raw data, the absolute majority want to do this through their APIs. We tried an implementation where we did web-requests directly fro...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group