I have a DLT that runs every day and an automatically executed maintenance job that runs within 24 hours every day. The maintenance operations are costly, is it possible to change the schedule to once a week or so?
In my DLT pipeline outlined below which generically cleans identifier tables, after successfully creating initial streaming tables from the append-only sources, fails when trying to create the second cleaned tables witht the following:It'**bleep** cl...
Hi @scvbelle The error message you're seeing is caused by an IllegalArgumentException error due to the restriction in Azure Blob File System (ABFS) that does not allow files or directories to end with a dot. This error is thrown by the trailingPeriod...
Dear Databricks Expert,I got some doubts when dealing with DBFS and Local File System.Case01: Copy a file from ADLS to DBFS. I am able to do so through the below python codes:#spark.conf.set("fs.azure.account.auth.type", "OAuth") spark.conf.set("fs.a...
Hi @KS LAU Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your q...
I am trying to follow along Apache Spark Programming training module where the instructor creates events table from a parquet file like this:%sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/mnt/training/ecommerce/events/events.par...
@Retired_mod Thanks for your response. I didn't provide cloud file system scheme in the path while creating the table using DataFrame API, but I was still able to create the table. %python # File location and type file_location = "/mnt/training/ecom...
How do I change my Databricks Community user name?
Hi Sujitha,Thanks for the quick help! I cleaned the caches, and it works now.
I have a few Databricks clusters, some share a single Hive Metastore (HMS), call them PROD_CLUSTERS, and an additional cluster, ADHOC_CLUSTER, which has its own HMS. All my data is stored in S3, as Databricks delta tables: PROD_CLUSTERS have read-wri...
Something went wrong there, here's the last sentence: I expected "location" will be the s3 path but it's not always so (elaborated in the original posting). Thanks!
We are in the process of implementing data mesh in our organization. When trying to help the different teams produce raw data, the absolute majority want to do this through their APIs. We tried an implementation where we did web-requests directly fro...
Hi,Let's assume I have these things:Binary column containing protobuf-serialized dataThe .proto file including message definitionWhat different approaches have Databricks users chosen to deserialize the data? Python is the programming language that...
We've now added a native connector with parsing directly with Spark Dataframes. https://docs.databricks.com/en/structured-streaming/protocol-buffers.htmlfrom pyspark.sql.protobuf.functions import to_protobuf, from_protobuf schema_registry_options = ...
I have a main databricks notebook that runs a handful of functions. In this notebook, I import a helper.py file that is in my same repo and when I execute the import everything looks fine. Inside my helper.py there's a function that leverages built-i...
Hi,i 'm facing similiar issue, when deploying via dbx.I have an helper notebook, that when executing it via jobs works fine (without any includes)while i deploy it via dbx (to same cluster), the helper notebook results withdbutils.fs.ls(path)NameEr...
Hello there,I have successfully created a databricks account and went to login to the community edition with the exact same login credentials as my account, but it tells me that the email/password are invalid. I can login with these same exact creden...
I have the same issue, and I have previously logged in and only a few months ago, it started the problem. Can anyone help?
Hi there, I read data from Azure Event Hub and after manipulating with data I write the dataframe back to Event Hub (I use this connector for that): #read data df = (spark.readStream .format("eventhubs") .options(**ehConf) ...
I had the same problem when starting with databricks. As outlined above, it is the shuffle partitions setting that results in number of files equal to number of partitions. Thus, you are writing low data volume but get taxed on the amount of write (a...
HelloI suspect that this can´t be done out of the box and want to know a way of doing this. I am trying to do it without success. So far I have tried this:Based on this link, I have created a Class and an object (companion and not, both ways) for cip...
I found a workaround for the problem to be able to use the secrets from the KeyVault in all the execturos. I only tested so far this in the notebooks, I want to try later in a JAR job.First here is a link to the official documentation that highlights...
Hi, Cluster Configuration details:RDS Configuration Details:I have 30 files, each file having 540000 recordsI read all files and created one dataframe.When i write dataframe(16,200,000 records) to a table it take more time nearly more than 1 hour (so...
Hi, I have Databricks cluster earlier connected to hive metastore and we have started migrating to Glue catalog.I'm facing an issue while creating table,Path must be absolute: <table-name>-__PLACEHOLDER__We have provided full access to glue and s3 in...
Hello,I am creating a Devops pipeline to run unit tests in my notebooks using the Nutter library. When a commit is pushed to a branch, I have a pipeline that triggers, and it should update my repo in a folder Staging (/Repos/Staging/MyRepo)For that I...
Hello, Thanks for contacting Databricks Support.The error message indicates that there is an issue with the URL or endpoint you are using with the Databricks repos update command. It appears that one or more required parameters are not being set corr...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
