- 17818 Views
- 5 replies
- 3 kudos
Is there a way to use wild card expressions inside dbutils functions
I am working on a migration project to migrate the HDFS command executions I do inside my Python code via os.system() function by replacing it with dbutils functions.dbutils functions are working as expected if I pass the fully qualified path of a fi...
- 17818 Views
- 5 replies
- 3 kudos
- 3 kudos
As I mentioned in my problem statement, I have already implemented the required functionality with alternative approaches (AWS S3 API and BOTO3 API).Still, it is an outstanding issue with DBUTILS.
- 3 kudos
- 2221 Views
- 0 replies
- 0 kudos
Mounting adls gen2 from databricks RBAC issue
I am trying to mount my Gen 2 storage account in databricks. I have added permission of Storage blob data contributor to the storage account. But I am getting below error:Invalid permissions on the specified KeyVault https://kvmigrationnew.vault.azu...
- 2221 Views
- 0 replies
- 0 kudos
- 3985 Views
- 1 replies
- 0 kudos
Ydata-profiling on Spark cluster -working for you?
I am trying to profile my dataset using ydata-profiling.I constantly run into errors, even with simple datasets on my spark cluster.Profiling in Spark cluster erroring out · Issue #1350 · ydataai/ydata-profiling (github.com)Has anyone got ydata-profi...
- 3985 Views
- 1 replies
- 0 kudos
- 2540 Views
- 2 replies
- 0 kudos
Mosaic library not supported in db13 and unity catalog
I want to use gepatial functions which uses mosaic library. How can I use it without using mosaic library and import functions available ?Example st_aswkt
- 2540 Views
- 2 replies
- 0 kudos
- 4930 Views
- 5 replies
- 1 kudos
Access RDS Postgres DB via SSH Tunnel
Hello, How can I configure a foreign catalog connection to use SSH tunneling? I want to be able to use unity catalog
- 4930 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi, in addition to our previous message, you can try https://docs.databricks.com/en/query-federation/foreign-catalogs.html and https://grant-6562.medium.com/connecting-to-sql-server-through-an-ssh-tunnel-with-python-17de859caca5. Also please tag @Deb...
- 1 kudos
- 7214 Views
- 1 replies
- 2 kudos
Using Azure Event Grid for structured streaming
Can anyone point me to any Databricks documentation (or other resources) for configuring structured streaming to use Azure Event Grid for a source/sink? I found examples for Kafka and EventHubs but Azure Event Grid is different than Azure Event Hubs....
- 7214 Views
- 1 replies
- 2 kudos
- 2 kudos
I must be missing something. I don't see how the examples in the referenced document can be applied to Azure Event Grid.Is there another example that shows how to subscribe to an Event Grid topic?
- 2 kudos
- 2726 Views
- 1 replies
- 0 kudos
Unity Catalog - Invalid configuration value detected for fs.azure.account.key
Hi there,I am having issue with writing a df to a table or display it. I have three dataframes that I have unioned and after I have done the union I cannot display the dataframe.df_table1 = spark.sql(f'SELECT * FROM {sql_full_name}')df_table2 = ...df...
- 2726 Views
- 1 replies
- 0 kudos
- 1560 Views
- 1 replies
- 0 kudos
sql end point and jdbc driver
When we try and connect to a sql warehouse endpoint with the databricks jdbc driver our query is failing if we use first_value(). We've rewritten the query to use limit 1, but we would like to understand if this is a gap in the simba/databricks driv...
- 1560 Views
- 1 replies
- 0 kudos
- 0 kudos
A sample error message when using first_value() is:An error occurred while calling o132.csv. [Databricks][JDBC](10140) Error converting value to BigDecimal.
- 0 kudos
- 1748 Views
- 1 replies
- 0 kudos
Databricks certified Data engineer Associate exam got suspended_need immediate help(10/09/2023)
Hi team,I've scheduled my exam on 10th September,2023 at 15:15hrs Asia Calcutta timing but my exam got suspended stating no proper environment by the proctor.Please check this issue as I didn't attempt one single question too, this is not fair as I l...
- 1748 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sirishavemula20 We have addressed the case. If the issue is not resolved, please reply back to the case notes.
- 0 kudos
- 1399 Views
- 0 replies
- 0 kudos
Use a single cluster policy for multiple teams and capture team name via custom tags
I am planning to add team names in custom tags and was hoping can do it with allowList and then have the user choose from the list. I am trying to avoid having multiple policy files per team.Has anybody found a good way to do this? May be using globa...
- 1399 Views
- 0 replies
- 0 kudos
- 2379 Views
- 2 replies
- 0 kudos
Cannot see 'Databricks Lakehouse Platform" on prod AWS Cost Explorer
Hi, Currently, we have two different AWS accounts: dev and prod. We also have two different workspaces: one for dev and another for prod. The strange thing is that prod costs are being added to the dev account on AWS Cost Explorer ("Databricks Lakeho...
- 2379 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks uses tags and AWS CloudTrail logs to connect and report costs to AWS.Tags can be used to monitor costs and attribute Databricks usage to different business units and teams.AWS CloudTrail logs can be used to calculate the exact cost of API ...
- 0 kudos
- 2475 Views
- 1 replies
- 0 kudos
Snowflake filter Query giving empty results in Databricks, same query working in Snowflake.
I am trying to fetch filter data based on date format on a date column. Below is the query forming in Databricks. In Databricks giving empty results.SELECT * FROM Test.TestSchema.Address where TO_DATE(TO_VARCHAR(MODIFIEDDATE,'YYYY-MM-DD')) = '2023-09...
- 2475 Views
- 1 replies
- 0 kudos
- 1188 Views
- 1 replies
- 0 kudos
Sql warehouse setting permission
How do you set permission to just execute a query but cannot modify settings on the sql warehouse
- 1188 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Have you tried this: https://docs.databricks.com/en/security/auth-authz/access-control/sql-endpoint-acl.html Please let us know if this helps. Thanks!
- 0 kudos
- 2365 Views
- 1 replies
- 0 kudos
Resolved! Structured streaming from Azure Event Hub, authenticating without SAS keys
Using SAS keys is a security issue that we would like to avoid, how do we utilize structured streaming from Event Hub while authenticating to Azure AD (client_id and secret).We know that we can use pythons Event Hub library, but that will make have t...
- 2365 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please try structured streaming event hubs integration? https://docs.databricks.com/en/_extras/notebooks/source/structured-streaming-event-hubs-integration.html
- 0 kudos
- 1572 Views
- 1 replies
- 1 kudos
How to connect to database in azure databricks from vs2022 console application to access the data
Hi Team,Am new to databricks and please find my below question and do the favor.I have created a cluster then database, tables and data also inserted, Now I want to access this table data from my dotnet console application which is in 6.0 framework(v...
- 1572 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, Databricks extension will help you to do so. https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext Please let us know if this helps. Thanks!
- 1 kudos
