- 1534 Views
- 1 replies
- 0 kudos
Mounting adls gen2 from databricks RBAC issue
I am trying to mount my Gen 2 storage account in databricks. I have added permission of Storage blob data contributor to the storage account. But I am getting below error:Invalid permissions on the specified KeyVault https://kvmigrationnew.vault.azu...
- 1534 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @keer1392, The error you are encountering is due to the Azure Databricks application not having the appropriate permissions to access the secrets in the Azure Key Vault. The error message indicates that the Azure Databricks application is not aut...
- 0 kudos
- 7811 Views
- 1 replies
- 0 kudos
Left join with merging two streaming tables returns inner join instead
Hi all, Im trying to merge two streaming tables together with a left (outer) join, but it seems to somehow return all values from the left table that have matching values from the right table, instead of also appending the NULL values if there is no ...
- 7811 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Brammer88, In your code, you have applied the watermark on the vw_inventdim view but not on the vw_block view. According to Databricks documentation, watermark and event-time constraints must be specified for left and right outer joins. This is ...
- 0 kudos
- 29862 Views
- 8 replies
- 6 kudos
Get up to speed on Generative AI with this free on-demand training
Build foundational knowledge of generative AI, including large language models (LLMs), with 4 short videos. Here is how it works: Watch 4 short tutorial videosPass the knowledge testEarn a badge for Generative AI Fundamentals you can share on your ...
- 29862 Views
- 8 replies
- 6 kudos
- 1918 Views
- 1 replies
- 0 kudos
Cluster can't connect to redshift
I'm trying to connect to redshift with redshift-connector module, but it get 'Connection refused' error, redshift cluster and ec2 cluster are in the same VPC, security group of redshift cluster is open with inbound from VPC's IP range, i can connect ...
- 1918 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Danh_Hoa , There could be several reasons why you are unable to connect to your Redshift cluster from your Databricks cluster: 1. **VPC Peering**: It might be possible that VPC peering is misconfigured. You need to validate the peering configurat...
- 0 kudos
- 6936 Views
- 3 replies
- 0 kudos
Databricks Certified Data Engineer Associate exam
My Databricks Certified Data Engineer Associate exam got timed out as I was uninstalling the Antivirus, it took a while to set up my device. By that time it got timed out to start the exam. So please reschedule my exam.webassessor id: anjaliisanvaga...
- 6936 Views
- 3 replies
- 0 kudos
- 0 kudos
Dear Support Team, I apologize for the inconvenience caused. Unfortunately, due to unforeseen circumstances with my antivirus uninstallation, I was unable to start the exam on time. I kindly request you to reschedule my Databricks Certified Data Engi...
- 0 kudos
- 6779 Views
- 3 replies
- 0 kudos
Data Engineer Associate V3 exam got suspended
Hi team,My Databricks Certified Data Engineer Associate exam got suspended within 40 minutes. I had also shown my exam room to the proctor. My exam got suspended due to eye movement. I was not moving my eyes away from laptop screen. Some questions ar...
- 6779 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @vivek2612 the support team did receive your case and is working on it now. Good luck on your exam!
- 0 kudos
- 2276 Views
- 2 replies
- 0 kudos
Resolved! Ydata-profiling on Spark cluster -working for you?
I am trying to profile my dataset using ydata-profiling.I constantly run into errors, even with simple datasets on my spark cluster.Profiling in Spark cluster erroring out · Issue #1350 · ydataai/ydata-profiling (github.com)Has anyone got ydata-profi...
- 2276 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @manojvas, Databricks has a built-in data profiling feature that you can use. When you use the display(<dataframe>) command in Scala or Python or run a SQL query, the results pane shows a new tab, "Data Profiles," that presents an interactive ta...
- 0 kudos
- 1415 Views
- 3 replies
- 0 kudos
Mosaic library not supported in db13 and unity catalog
I want to use gepatial functions which uses mosaic library. How can I use it without using mosaic library and import functions available ?Example st_aswkt
- 1415 Views
- 3 replies
- 0 kudos
- 5496 Views
- 2 replies
- 2 kudos
Using Azure Event Grid for structured streaming
Can anyone point me to any Databricks documentation (or other resources) for configuring structured streaming to use Azure Event Grid for a source/sink? I found examples for Kafka and EventHubs but Azure Event Grid is different than Azure Event Hubs....
- 5496 Views
- 2 replies
- 2 kudos
- 2 kudos
I must be missing something. I don't see how the examples in the referenced document can be applied to Azure Event Grid.Is there another example that shows how to subscribe to an Event Grid topic?
- 2 kudos
- 760 Views
- 1 replies
- 0 kudos
sql end point and jdbc driver
When we try and connect to a sql warehouse endpoint with the databricks jdbc driver our query is failing if we use first_value(). We've rewritten the query to use limit 1, but we would like to understand if this is a gap in the simba/databricks driv...
- 760 Views
- 1 replies
- 0 kudos
- 0 kudos
A sample error message when using first_value() is:An error occurred while calling o132.csv. [Databricks][JDBC](10140) Error converting value to BigDecimal.
- 0 kudos
- 917 Views
- 1 replies
- 0 kudos
Databricks certified Data engineer Associate exam got suspended_need immediate help(10/09/2023)
Hi team,I've scheduled my exam on 10th September,2023 at 15:15hrs Asia Calcutta timing but my exam got suspended stating no proper environment by the proctor.Please check this issue as I didn't attempt one single question too, this is not fair as I l...
- 917 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sirishavemula20 We have addressed the case. If the issue is not resolved, please reply back to the case notes.
- 0 kudos
- 373 Views
- 0 replies
- 0 kudos
Hive Metastore permission on DBX 10.4
I've been working on creating a schema in the Hive Metastore using the following command:spark.sql(f'CREATE DATABASE IF NOT EXISTS {database}')The schema or database is successfully created, but I encountered an issue where it's only accessible for m...
- 373 Views
- 0 replies
- 0 kudos
- 2645 Views
- 2 replies
- 1 kudos
Change default catalog
It seems that when I am connecting to Databricks Warehouse, it is using the default catalog which is hive_metastore. Is there a way to define unity catalog to be the default?I know I can run the queryUSE CATALOG MAINAnd then the current session will ...
- 2645 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks Brian2. Is there an equivalent config parameter for a SQL Warehouse?
- 1 kudos
- 366 Views
- 0 replies
- 0 kudos
ronaldo is back
create table SalesReport(TerritoryName NVARCHAR(50), ProductName NVARCHAR(100), TotalSales DECIMAL(10,2), PreviousYearSales DECIMAL(10,2), GrowthRate DECIMAL(10,2)); create table ErrorLog( ErrorID int, ErrorMessage nvarchar(max),ErrorDate datetime);...
- 366 Views
- 0 replies
- 0 kudos
- 748 Views
- 1 replies
- 1 kudos
Save dataframe to the same variable
I would like to know if there is any difference if I save dataframe during tranformation to itself as first code or to new dataframe as second example.Thankslog_df = log_df.withColumn("process_timestamp",from_utc_timestamp(lit(current_timestamp()),"E...
- 748 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @alesventus, When saving a DataFrame after transformation, there is no difference between saving it to itself or a new DataFrame. Both approaches will result in the same output. Sources:- [Docs: dataframes-python](https://docs.databricks.com/getti...
- 1 kudos
