- 745 Views
- 2 replies
- 0 kudos
Error connecting to MySQL from Databricks: (2003, "Can't connect to MySQL server")
Hello Community,I'm facing an issue connecting to a MySQL database hosted on AWS RDS from within a Data bricks notebook. My Python script to connect to MySQL works fine locally, but when I run it in Data bricks, I receive this error:Error connecting ...
- 745 Views
- 2 replies
- 0 kudos
- 2064 Views
- 2 replies
- 3 kudos
Resolved! Asset Bundles pause_status Across Different Environments
HiQuestion probably around best practices, but curious if someone else has dealt with a similar situation. I have 2 Databricks workspaces - one for Dev and one for Prod. Had to be two workspaces because Azure Landing Zones had to be air gapped from e...
- 2064 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @RangaSarangan ,We have faced same issue and solved using databricks workflow API and json file for job metadata that consist job and thier respective status for each env.You can create azure devops that run after your cicd pipeline and change the...
- 3 kudos
- 567 Views
- 1 replies
- 0 kudos
CREATED WIDGET - SQL syntax - how do I specify a label?
What is the syntax in SQL for creating a widget in a notebook with a label?This documentation says "The last argument is label, an optional value for the label shown over the widget text box or dropdown."The one example provided on that page doesn't ...
- 567 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @eriodega I do not know how it works with sql but a possibility is to use a python cell in your SQL notebook and create it like this:%python dbutils.widgets.text("some_name", "a great default", "some label/description")All the best
- 0 kudos
- 11210 Views
- 3 replies
- 1 kudos
Schedule a Notebook Dashboard
Hey all,I have a workflow that updates a delta table, and then runs a notebook that generates a dashboard. I was hoping that by adding this second step that the dashboard would get updated to show the most current data, instead of the user needing to...
- 11210 Views
- 3 replies
- 1 kudos
- 1 kudos
@ehpogue at the end of the day I just used chatgpt to rewrite a bunch of python code into SQL and mash together all of the temporary views into one giant query. hacky but it did work
- 1 kudos
- 942 Views
- 1 replies
- 0 kudos
Managed Table
Hi Experts,I have a workspace created and associated a metastore with it, the metastore points to a storage location USDATA and then I create two catalogs in the workspace and one is using default meta store as the external storage location and other...
- 942 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mani2105,if i create a table in the sales catalog without specifiying any external location, will the tables created be managed and will go to the Sales storage account Yes, if you create a table in the sales catalog without specifying any exter...
- 0 kudos
- 3119 Views
- 2 replies
- 1 kudos
Resolved! Unity Catalog Metastore Details
hi,I would like to seek response to my following questions regarding Unity Catalog Metastore's path.While configuring metastore, designating a metastore storage account (in case of Azure, it's ADLS Gen2) seems to be an optional thing. In case I confi...
- 3119 Views
- 2 replies
- 1 kudos
- 1 kudos
The storage container you configure for the metastore will contain the files of managed tables and volumes. The metadata is stored in a database of the Databricks control plane.
- 1 kudos
- 5005 Views
- 1 replies
- 0 kudos
DLT full refresh
Running a task with full refresh in delta live tables removes existing data and reloads it from scratch. We are ingesting data from an event hub topic and from files. The event hub topic stores messages for seven days after arrival. If we would run a...
- 5005 Views
- 1 replies
- 0 kudos
- 0 kudos
I know it's a bit after the fact, but in case you didn't solve it, I came across this article in the Databricks documentation. You can set pipelines.reset.allowed to false on a table to prevent a full refresh of a table. Ref: https://docs.databrick...
- 0 kudos
- 881 Views
- 1 replies
- 0 kudos
Resolved! can't create cluster
Hi,I am trying to learn databricks and created a student account In Azure.currently I have 100$ available with me.But when i try to create a basic cluster with 1DBU/hour I am getting warning and not able to create.This account may not have enough CP...
- 881 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi This is because new accounts on azure have lower quotas: https://learn.microsoft.com/en-us/azure/quotas/quotas-overviewTry to choos another Node-Type like Standard_4Fs, which has less CPU. And you can increase th quota: https://learn.microsoft.com...
- 0 kudos
- 2515 Views
- 7 replies
- 2 kudos
Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' :
Hi there,My cluster version is 15.4 LTS, and the workspace has UC enabled. When I used the initialization script to install ODBC Driver 17 for SQL Server, there were no errors and the cluster started successfully. But when I use ODBC Driver 17 for SQ...
- 2515 Views
- 7 replies
- 2 kudos
- 2 kudos
Also is INIT script route is the only route? or is there any other option available?the reason I am asking, some time back we had a call with DBK and then they mentioned like usage of INIT script is not the right way or so.. cant recall exact explana...
- 2 kudos
- 1520 Views
- 4 replies
- 3 kudos
Resolved! Workflow concurrent runs not working as expected
Hello All,I am trying to fetch data from different sources for tables driven by a metadata table. Data will get fetched from sources using jdbc connector for each table mentioned in the metadata table. A scheduled job is responsible for fetching the ...
- 1520 Views
- 4 replies
- 3 kudos
- 3 kudos
Soo.. You use a loop to go through metadata table and then retrieve and ingest files using JDBC?If so, then the concurrent runs won't be helpful. Concurrent runs means the number of how many runs of that job can be ran side by side. For you, this wou...
- 3 kudos
- 1184 Views
- 4 replies
- 2 kudos
Resolved! Restricting access to secrets
Hi. I want to restrict access to secrets to a security group, as the secrets can be used to retrieve sensitive data only a few people should see. Up until now, we have been using KV-backed secret scopes, but as it's sufficient that Databricks has the...
- 1184 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi Johan, this should work for restriction: https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secrets.Fine granulat access based on secrets is currently not possible.BR
- 2 kudos
- 563 Views
- 1 replies
- 0 kudos
In Unity Catalog repartition method issue
We are in the process of upgrading our notebooks to Unity Catalog. Previously, I was able to write data to an external Delta table using df.repartition(8).write. Save('path'), which correctly created multiple files. However, during the upgrade, in te...
- 563 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Shiva3,Maybe you can try this option in Delta Lake in Unity Catalog may have optimizedWrites enabled by default, which can reduce the number of files by automatically coalescing partitions during writes. # Disable auto-compaction and optimized wr...
- 0 kudos
- 14822 Views
- 5 replies
- 2 kudos
databricks single user cluster is not able to assign service principals
I want to set the databricks cluster Access mode to single user and assign the service principal account to the user. In other words, after creating a single mode cluster, how can I access only servcie principals?
- 14822 Views
- 5 replies
- 2 kudos
- 2 kudos
So, here is an alternative to either the UI (doesn't work actually; not possible) or the CLI. Use the Databricks API.endpoint_change = f"{databricksURL}/api/2.1/clusters/edit" # change single ownerpayload_change = { "cluster_id": cluster_id ,"clu...
- 2 kudos
- 3729 Views
- 4 replies
- 0 kudos
Databricks Runtime 13.3 - can I use Databricks Connect without Unity Catalog?
Hello all,The official documentation for Databricks Connect states that, for Databricks Runtime versions 13.0 and above, my cluster needs to have Unity Catalog enabled for me to use Databricks Connect, and use a Databricks cluster through an IDE like...
- 3729 Views
- 4 replies
- 0 kudos
- 0 kudos
Did you configure databricks connect without UnitCatalog at the end? If you managed to do that can you share with me how?
- 0 kudos
- 9113 Views
- 2 replies
- 2 kudos
Identity column value of Databricks delta table is not started with 0 and increaed by 1. It always started with something like 1 or 2 and increased by 2. Below is the sample code and any logical input here is appreciated
spark.sql("CREATE TABLE integrated.TrailingWeeks(ID bigint GENERATED BY DEFAULT AS IDENTITY (START WITH 0 increment by 1) ,Week_ID int NOT NULL) USING delta OPTIONS (path 'dbfs:/<Path in Azure datalake>/delta')")
- 9113 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi,When you define an identity column in Databricks with GENERATED BY DEFAULT AS IDENTITY (START WITH 0 INCREMENT BY 1), it is expected to start at 0 and increment by 1. However, due to Databricks' distributed architecture, the values may not be str...
- 2 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
