- 2745 Views
- 8 replies
- 1 kudos
Unable to run sql alert task using databricks job using service principal
I am trying to run a SQL alert task in data bricks job, Service principal is the run_as as set to the job, and task fails with below error message. I also checked if SPN can be granted permission on SQL alert and SQL query, turns out only user id or...
- 2745 Views
- 8 replies
- 1 kudos
- 1 kudos
unfortunately, only these options are available in my UI, is it the new UI?
- 1 kudos
- 793 Views
- 1 replies
- 0 kudos
Resolved! Declative Pipeline: Can pipeline or job be deployed run_as using asset bundle
Hi, I have very interesting scenario. I am trying to use Declarative pipelines for first time. The platform team has made workspace artefacts as devops based deployment [infra as code], meaning, I cannot create compute. I have to create compute with ...
- 793 Views
- 1 replies
- 0 kudos
- 0 kudos
@crami wrote:Hi, I have very interesting scenario. I am trying to use Declarative pipelines for first time. The platform team has made workspace artefacts as devops based deployment [infra as code], meaning, I cannot create compute. I have to create ...
- 0 kudos
- 2562 Views
- 2 replies
- 0 kudos
Resolved! Update jobs parameter, when running, from API
Hi, When a Job is running, I would like to change the parameters with an API call.I know that I can set parameters value from API when I start a job from API, or that I can update the default value if the job isn't running, but I didn't find an API c...
- 2562 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Alessandro, I am trying to set job parameters value from API when I start a job from API call within python notebook, however, it has never succeeded, the job can be triggered, but always with job parameters' default values instead values from AP...
- 0 kudos
- 1299 Views
- 2 replies
- 1 kudos
How to enable Liquid Clustering on an existing Delta Live Table (DLT) and syntax for enabling it
Hi all,I’m working with Delta Live Tables (DLT) and want to enable Liquid Clustering on an existing DLT table that was already created without it.Could someone please clarify:How can I enable Liquid Clustering on an existing DLT table (without recre...
- 1299 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Sakthi0311 ,For SQL language you can enable LC for materialized views and streaming tables. So the syntax looks following:If you want to use automatic clustering then use CLUSTER BY AUTO.
- 1 kudos
- 5171 Views
- 6 replies
- 3 kudos
Resolved! How do use Databricks Lakeflow Declarative Pipeline on AWS DMS data?
Hi!I am trying to replicate an AWS RDS PostgreSQL database in Databricks. I have successfully manage to enable CDC using AWS DMS that writes an initial load file and continuous CDC files in parquet.I have been trying to follow the official guide Repl...
- 5171 Views
- 6 replies
- 3 kudos
- 3 kudos
hey @excavator-matt Let's remember that the Bronze layer is for mere raw ingestion; this provides a baseline for auditing and to start applying transformations based on the different use cases you need to serve. Systems and their requirements change...
- 3 kudos
- 4709 Views
- 4 replies
- 5 kudos
Resolved! Quota Limit Exhausted Error when Creating Data Ingestion with SQL Server Connector (Azure)
Good Day all,I am having an issue with our first Data Ingestion Pipelines, I am wanting to connect to our Azure SQL Server with our Unity Connector (I can access the data in Unity Catalog). When I am on Step 3 of the process (Source) when it is scann...
- 4709 Views
- 4 replies
- 5 kudos
- 18474 Views
- 5 replies
- 3 kudos
Resolved! How does partition pruning work on a merge into statement?
I have a delta table that is partitioned by Year, Date and month. I'm trying to merge data to this on all three partition columns + an extra column (an ID). My merge statement is below:MERGE INTO delta.<path of delta table> oldData using df newData ...
- 18474 Views
- 5 replies
- 3 kudos
- 3 kudos
Isn't the suggested idea only filtering the input dataframe (resulting in a smaller amount of data to match across the whole delta table) rather than prune the delta table for relevant partitions to scan?
- 3 kudos
- 1738 Views
- 3 replies
- 3 kudos
Resolved! Does Autoloader supports loading PDF files?
I need to process PDF files already ingested. Based on the documentation, Autoloader does not support PDFs - or am I missing something?Also, I've found this sparkPDF library in other discussions in the community, but from what I see it's only for bat...
- 1738 Views
- 3 replies
- 3 kudos
- 11160 Views
- 7 replies
- 0 kudos
How to Assign User Managed Identity to DBR Cluster so I can use it for quering ADLSv2?
Hi,I'm trying to figure out if we can switch from Entra ID SPN's to User Assigned Managed Indentities and everything works except I can't figure out how to access the lake files from python notebook.I've tried with below code and was running it on a ...
- 11160 Views
- 7 replies
- 0 kudos
- 0 kudos
Besides, this only works in dedicated clusters, non working on shared ones. Why? No idea at all. Latest case, IMDS (Internal Metadata Service) used by Azure to inject token endpoint inside resources as a unique secure and valid channel to get tokens ...
- 0 kudos
- 4092 Views
- 1 replies
- 0 kudos
Genie Integrating with streamlit
Hi All,What are the best practices to follow while integrating with Genie and streamlit , and are there any limitations?how best way to present in UI level on user perceptive ?Regards,Phani
- 4092 Views
- 1 replies
- 0 kudos
- 0 kudos
This should help you get started. Please let us know if you've any specific question after you've looked at the links below. https://blog.streamlit.io/best-practices-for-building-genai-apps-with-streamlit/ https://databrickster.medium.com/call-genie-...
- 0 kudos
- 3771 Views
- 3 replies
- 3 kudos
Resolved! Hive Metastore End of Life
Hello everyone,is there a rough date on which the Hive Metastore will be deactivated?In the end, I ask the question again, which was already asked 2 years ago:Solved: Hive metastore table access control End of Support - Databricks Community - 50487We...
- 3771 Views
- 3 replies
- 3 kudos
- 3 kudos
HI @julius_bkr I do not think Databricks has any plans to retire HMA completely. However, Unity Catalog is the strategic direction and the recommended approach for new deployments and migrating existing data governance. Databricks is investing heavil...
- 3 kudos
- 1740 Views
- 6 replies
- 3 kudos
Resolved! [NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION] The -12874815911431.6200000000 rounded half up from
I am using dataricks version 15.4 and getting below error whicle reading from jdbc and writing to aws S3 location: [NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION] The -12874815911431.6200000000 rounded half up from -12874815911431.6200000000 cannot b...
- 1740 Views
- 6 replies
- 3 kudos
- 2287 Views
- 4 replies
- 2 kudos
Are there SQL linters for Databricks GUI queries and notebook %sql cells?
I'm looking for SQL linters in two different contexts within Databricks GUI:SQL queries in Databricks SQL Editor (GUI) - Is there a built-in linter for writing SQL queries in the Databricks SQL workspace?%sql magic cells in Databricks notebooks (GUI...
- 2287 Views
- 4 replies
- 2 kudos
- 2 kudos
in databricks notebook is possible to connect own linters but I don't think it will work in SQL editor.
- 2 kudos
- 2397 Views
- 4 replies
- 1 kudos
Resolved! Notebook Session Has Crashed
I am getting a pop-up error message in the right hand corner of my Databricks session, "Your notebook session has crashed." This is a notebook I've been working in for a while.The only line of code I am running is pip listThe error at the bottom of ...
- 2397 Views
- 4 replies
- 1 kudos
- 1 kudos
From the screen, it looks like an interactive cluster—maybe try rebooting it.
- 1 kudos
- 811 Views
- 1 replies
- 1 kudos
Resolved! Looking for CLI-based SQL formatter for Databricks: Alternative to gethue/sql-formatter for .
I'm looking for a SQL formatter (CLI) for two different contexts within Databricks.1. SQL queries in .sql files - What tool does Databricks use by default to format SQL?2. %sql magic cells in notebooks (.ipynb files) - What tool does Databricks use b...
- 811 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Lenoq ,As far as I know, currently there is no tool that meets your given criteria by databricks/labs. The closest that I can see is the tool you gave as an example, but as you said its not that good. what I would do maybe is an intermedial optio...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
