- 8694 Views
- 5 replies
- 2 kudos
High cost of storage when using structured streaming
Hi there, I read data from Azure Event Hub and after manipulating with data I write the dataframe back to Event Hub (I use this connector for that): #read data df = (spark.readStream .format("eventhubs") .options(**ehConf) ...
- 8694 Views
- 5 replies
- 2 kudos
- 2 kudos
I had the same problem when starting with databricks. As outlined above, it is the shuffle partitions setting that results in number of files equal to number of partitions. Thus, you are writing low data volume but get taxed on the amount of write (a...
- 2 kudos
- 17745 Views
- 1 replies
- 1 kudos
Resolved! Access Azure KeyVault from all executors in Databricks
HelloI suspect that this can´t be done out of the box and want to know a way of doing this. I am trying to do it without success. So far I have tried this:Based on this link, I have created a Class and an object (companion and not, both ways) for cip...
- 17745 Views
- 1 replies
- 1 kudos
- 1 kudos
I found a workaround for the problem to be able to use the secrets from the KeyVault in all the execturos. I only tested so far this in the notebooks, I want to try later in a JAR job.First here is a link to the official documentation that highlights...
- 1 kudos
- 3029 Views
- 0 replies
- 0 kudos
Writing data to RDS table taking more time
Hi, Cluster Configuration details:RDS Configuration Details:I have 30 files, each file having 540000 recordsI read all files and created one dataframe.When i write dataframe(16,200,000 records) to a table it take more time nearly more than 1 hour (so...
- 3029 Views
- 0 replies
- 0 kudos
- 4767 Views
- 1 replies
- 1 kudos
Error while creating table with Glue catalog
Hi, I have Databricks cluster earlier connected to hive metastore and we have started migrating to Glue catalog.I'm facing an issue while creating table,Path must be absolute: <table-name>-__PLACEHOLDER__We have provided full access to glue and s3 in...
- 4767 Views
- 1 replies
- 1 kudos
- 3007 Views
- 1 replies
- 0 kudos
Cannot update databricks repos in Devops Pipeline
Hello,I am creating a Devops pipeline to run unit tests in my notebooks using the Nutter library. When a commit is pushed to a branch, I have a pipeline that triggers, and it should update my repo in a folder Staging (/Repos/Staging/MyRepo)For that I...
- 3007 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, Thanks for contacting Databricks Support.The error message indicates that there is an issue with the URL or endpoint you are using with the Databricks repos update command. It appears that one or more required parameters are not being set corr...
- 0 kudos
- 8151 Views
- 1 replies
- 0 kudos
Libraries Not Working on Shared Cluster 13.3 LTS
I am facing this error while installing the spark-excel library into the cluster. Does anyone know how to add library into artifact allowlist?Jars and Maven Libraries on Shared Clusters must be on the allowlist. Failed Libraries: com.crealytics:spark...
- 8151 Views
- 1 replies
- 0 kudos
- 0 kudos
You can add the jar followed below steps:How to add items to the allowlistYou can add items to the allowlist with Data Explorer or the REST API.To open the dialog for adding items to the allowlist in Data Explorer, do the following:In your Databricks...
- 0 kudos
- 11812 Views
- 1 replies
- 2 kudos
Handling GDPR requests in databricks
When dealing with GDPR requests in databricks, there are some essential things to keep in mind:- Use a low retention period to ensure you don't keep table delta version history for tables with personal information.- Use APPLY CHANGES to handle Slowly...
- 11812 Views
- 1 replies
- 2 kudos
- 2 kudos
Thank you for sharing this information @Hubert-Dudek!!!!
- 2 kudos
- 26460 Views
- 1 replies
- 2 kudos
Checking that spark dataframe is empty
#databricks #spark 3.3 has introduced a simple yet powerful isEmpty() function for DataFrames. Gone are the days of using count() to check for empty DataFrames — now it'**bleep** as easy as calling df.isEmpty().
- 26460 Views
- 1 replies
- 2 kudos
- 2 kudos
Thank you for sharing this @Hubert-Dudek !!!
- 2 kudos
- 2170 Views
- 0 replies
- 0 kudos
Regarding databricks community edition **bleep** issue
Hi team,I had created a databricks community edition account. I am trying to **bleep** into it and its showing error. I tried changing password but it still doesn't work. Please let me know where the problem is.Thanks & Regards
- 2170 Views
- 0 replies
- 0 kudos
- 11765 Views
- 6 replies
- 10 kudos
Databricks JDBC Driver v2.6.29 Cloud Fetch failing for Windows Operating System
Hi everyone, I've been stuck for the past two days on this issue with my Databricks JDBC driver and I'm hoping someone can give me more insight into how to troubleshoot. I am using the Databricks JDBC driver in RStudio and the connection was working ...
- 11765 Views
- 6 replies
- 10 kudos
- 10 kudos
@Debbie Ng​ From your message I see there was a windows update and this failure started. based on the conversation you tried latest version of the driver and still you face the problem. I believe this is something related to the Java version compatib...
- 10 kudos
- 3597 Views
- 0 replies
- 0 kudos
how to build data warehouses and data marts with Python
I don't know how to build data warehouses and data marts with Python. My current development environment is storing data in AWS Redshift, and I can run queries from Databricks against the stacked tables in Redshift.Can you show me some simple code?
- 3597 Views
- 0 replies
- 0 kudos
- 12519 Views
- 7 replies
- 2 kudos
Delta live table generate unique integer value (kind of surrogate key) for combination of columns
Hi,we are in process of moving our Datawarehouse from sql server to databricks. we are in process of testing our Dimension Product table which has identity column for referencing in fact table as surrogate key. In Databricks Apply changes SCD type 2 ...
- 12519 Views
- 7 replies
- 2 kudos
- 2 kudos
Hey. Yep, xxhash64 (or even just hash) generate numerical values for you. Combine with abs function to ensure the value is positive. In our team we used abs(hash()) ourselves... for maybe a day. Very quickly I observed a collision, and the data s...
- 2 kudos
- 7134 Views
- 4 replies
- 0 kudos
Resolved! udf not admin user
java.lang.SecurityException: User does not have permission SELECT on anonymous function.
- 7134 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @data engineer​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 0 kudos
- 6069 Views
- 0 replies
- 0 kudos
Custom logging using Log4J to a file
Hello,I would like to ask for help setting up the log4j.I want to use log4j (log4j2) to generate custom log messages in my notebook when running.This message would be generated like this: logger.info("some info message") but using log4j not python lo...
- 6069 Views
- 0 replies
- 0 kudos
- 2144 Views
- 1 replies
- 1 kudos
Design Question
we have an application that takes in raw metrics data like key-value pairs. then we split them into four different table like below`key1, min, max, average`Those four tables are later used for dashboard. What are the design recommendations to this? S...
- 2144 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey,I can totally relate to the challenges Frank is facing with this application'**bleep** data processing. It'**bleep** frustrating to deal with delays, especially when dealing with real-time metrics. I've had a similar experience where optimizing d...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
