Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
I am trying to setup a Databricks SQL endpoint to connect to an external hive metastore. I have replicated my metastore spark config in the SQL endpoint, with the addition of adding the below to the configuration:spark.sql.hive.metastore.jars maven I...
Hi @SETH JENSEN​ , Just checking if you were able to solve this issue? If no, Can you try to modify the connection string and make sslMode=Require and see if it works?
Hi Everyone,This may sound bit odd as I haven't used Databricks much.I have a use case where I want to union data from two separate Databricks workspaces (under the same account) into a single SQL view. (Basically 2-2 tables from each workspaces and ...
Hey there @Bindi P​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? It would be really helpful for the other members too.Cheers!
I'm unable to run any command that queries data from the unity catalog within a function that executes in the event of an ipywidget button click. Code block below. I cannot do queries such as spark.sql(f"SHOW SCHEMAS;") or spark.sql(f"select * from d...
can you try to println out? val databricksApiTokenKey = CredentialContext.INHERITED_PROPERTY_DATABRICKS_API_TOKEN val databricksApiCredentialOpt = CredentialContext.getCredential(databricksApiTokenKey) val rawUrlProp = spark.sparkContext.get...
As we are starting to build our Lakehouse solution on Databricks, we need ACLs to be active. So far I have found two options:via UI or terraform: create a high-concurrency cluster and enable table access control for python and SQL. In terraform this ...
Privileges are additive by design.Let's say I as a user belong to 2 groups, foo1, foo2.The admin has run the following SQL:GRANT SELECT on table bar to `foo1`This means that I'll be able to select from the table bar because there's a grant.Let's say ...
I have created a premium cluster on Azure site. There were no problem Data Science and Engineering (DSAE) while i was binding Postgre SQL hive metastore. I have done all settings via Global init Scripts from Admin Console.However when I try to adjust...
Find soluttion :spark.hadoop.javax.jdo.option.ConnectionURL jdbc:postgresql://#########.postgres.database.azure.com:5432/databricks_metastore?ssl=trueneeds to be spark.hadoop.javax.jdo.option.ConnectionURL jdbc:postgresql://#########.postgres.databas...
Your guide to data and AI governance sessions at Data + AI Summit 2022We are so excited to welcome you to Data + AI Summit this year. This year’s hybrid conference is jam packed with exciting product announcements, live keynotes, breakout sessions wi...
Hi allWanted to check if anyone has made an attempt to exploit the Hive Metastore of Databricks for lineage?For example, I loaded metadata of 2 databricks databases using the Collibra Marketplace provided Databricks driver. Here is the scenario -Data...
@Chetan Kardekar​ @Kaniz Fatma​ yes, I still need a standard way (through SQL) to access Hive Metastore.
hi Team ,We are using a shared hive metastore for connecting to dev and prod as we need cross DB access for certain ML use-cases.Currently we are having issues as "DROP TABLE COMMAND" directly drops the table without checking the underlying locationH...
Hi All,I am facing the GC metadata issue while performing distributed computing on Spark. 2022-01-13T22:02:28.467+0000: [GC (Metadata GC Threshold) [PSYoungGen: 458969K->18934K(594944K)] 458969K->18958K(1954816K), 0.0144028 secs] [Times: user=0.05 sy...
Hi @Chandan Angadi​ Hope you are doing well now.Thanks for getting back to us and sending in your solution. Would you like to mark an answer as best?Thanks!
Databricks Runtime 10.5 (Beta) Auto Loader: new SQL function CLOUD_FILES_STATEYou can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream. Delta Lake: new maxRecordsPerFile option for maximum records written t...
anyone that got CLOUD_FILES_STATE to work on 10.5 Beta/Azure. Syntax:%sqlSELECT * FROM cloud_files_state('abfss://test@test.dfs.core.windows.net/test/checkpoint/'); i get this error: com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecu...
I am adding a credential to access MongoDB through the Python MongoDB Spark connector. It is stored as a secret and the `users` group has `MANAGE` ACL privileges. I turned on table ACLs and created a `High Concurrency` compute cluster. I then run a s...
@Erik Louie​ looks the issue is resolved. Please let me know if you need further help on this. Thanks!
IDM Serial Key with Internet Download Manager (IDM) is a tool to increase download speeds, resume, and schedule downloads. https://assadpc.com/idm-serial-number/
Hi, I would like to use Delta Sharing but I have not understood how to configure the Delta Sharing Server. Is that fully managed by Databricks ? If so I tried to share a table :%sql create share first_shareand I got the following error message with t...
Hi @Bertrand BURCKER​ ,This feature is not available yet. If you would like to test it or explore it, then you will need to sign up to get more details once the feature is available. In addition, you could reach out to your account team (in case you...
I found a page that tells the compatibility between Databricks and Hive metastore versions: https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastoreBut the information is only up to Databricks 7.1. Anywhere I can find...
it still should work with 7.x +. version. Are you seeing any issue on your cluster ? @Jeffrey Mak​
