- 1493 Views
- 2 replies
- 2 kudos
Resolved! Lakehouse monitoring dashboard shows no data
Hello, I am replicating the demo for Lakehouse monitoring found here: https://notebooks.databricks.com/demos/lakehouse-monitoring/index.htmlFor some reason, my dashboards show empty, i.e., they say 'no data' - like nothing fits the criteria from the ...
- 1493 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @heli123 Can you share the image again? looks like didnt upload well.
- 2 kudos
- 2691 Views
- 3 replies
- 3 kudos
Databricks model serving endpoint returns 403 Unauthorized access to workspace when using service
I deployed a simple Iris model in Databricks Model Serving and exposed it as an endpoint. I’m trying to query the endpoint using a service principal. I can successfully fetch the access token with the following databricks_token() function:def databri...
- 2691 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @szymon_dybczak, Thanks for your comment.One of the admins in this workspace tried using the token generated via client id and secret, and were able to successfully get a response from the serving endpoint using this same above mentioned code.Coul...
- 3 kudos
- 1212 Views
- 2 replies
- 0 kudos
How to switch serverless dlt pipeline to cost optimized mode from performance optimized
We have a few serverless dlt pipelines that we want to optimize for cost as we are ok with an increased latency. Where can I change the pipeline to run on cost optimized mode. I dont see this option in UI or API
- 1212 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello!To enable cost-optimized mode for your serverless DLT pipeline, switch it to Triggered mode and edit the schedule trigger—there you’ll find the option to disable “Performance optimized.” This setting isn’t visible in the main UI or API unless t...
- 0 kudos
- 44384 Views
- 13 replies
- 6 kudos
SQL Stored Procedure in Databricks
Hello. Is there an equivalent of SQL stored procedure in Databricks? Please note that I need a procedure that allows DML statements and not only Select statement as a function provides.Thank you in advance
- 44384 Views
- 13 replies
- 6 kudos
- 6 kudos
how to connect the stored procedures to power bi report builder, seems like it is not working
- 6 kudos
- 1602 Views
- 3 replies
- 0 kudos
Convert a Managed Table to Streaming Table
HiI have applied transformations on a set of streaming tables and saved it as a managed table....How can i change the Managed table to a Streaming table with minimal changesRegardsZD
- 1602 Views
- 3 replies
- 0 kudos
- 0 kudos
I am just writing the dataframe to delta table.....Are you suggesting me to first define a STREAMING TABLE (using the DLT definition) and then save the dataframe into that table?
- 0 kudos
- 3285 Views
- 4 replies
- 2 kudos
Databricks app with parameters from databricks asset bundle
HelloooI tried out setting up a Databricks App using asset bundle, where i was able to successfully parameterize the sql warehouse id which was specified on specific targets. However i was unable to get values of other variables from the targets, the...
- 3285 Views
- 4 replies
- 2 kudos
- 2 kudos
Found that this is an implementation in progress on the Databricks CLI. https://github.com/databricks/cli/issues/3679
- 2 kudos
- 2391 Views
- 2 replies
- 3 kudos
Resolved! handling both Pyspark and Python exceptions
In a Python notebook, I am using error handling according to the official documentation. try:[some data transformation steps]except PySparkException as ex:[logging steps to log the error condition and error message in a table]However, this catches o...
- 2391 Views
- 2 replies
- 3 kudos
- 3 kudos
To handle both PySpark exceptions and general Python exceptions without double-logging or overwriting error details, the recommended approach is to use multiple except clauses that distinguish the exception type clearly. In Python, exception handlers...
- 3 kudos
- 2903 Views
- 5 replies
- 1 kudos
Resolved! BUG in Job Task of Type DBT
Hi, just wanted to let the Databricks Team know, that there is a bug in the task ui.Currently it is not possible to save a task of "Type: dbt" if the "SQL Warehouse" is set to "None (Manual)".Some weeks ago this was possible, also the "Profiles Direc...
- 2903 Views
- 5 replies
- 1 kudos
- 1 kudos
I am facing this bug still. I don't want to select any SQL warehouse, what do I do? and from where can I pass the profiles directory
- 1 kudos
- 1131 Views
- 3 replies
- 1 kudos
Resolved! I recently made new account on databricks under Free edition
It by default made SQL warehouse compute, but I want all-purpose compute, as I want test and learn capabilities of PySpark and Databricks.I can't connect with the serverless compute in the notebook; it gives a mean error as follows: "An error occurr...
- 1131 Views
- 3 replies
- 1 kudos
- 1 kudos
In the free Databricks edition, to use PySpark you need to create and start a cluster, since the SQL Warehouse is only for SQL queries; go to Compute > Create Cluster, set up a free cluster, click Start, and then attach your notebook to it this will ...
- 1 kudos
- 2996 Views
- 2 replies
- 1 kudos
Unable to access metrics from Driver node on localhost:4040
Greetings,I am trying to setup monitoring in Grafana for all my databricks clustersI have added 2 things as part of thisUnder Compute > Configuration > Advanced > Spark > Spark Config, I have addedspark.ui.prometheus.enabled trueUnder init_scripts, I...
- 2996 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @vishal_balaji ,You're following guides that were prepared for OSS Apache Spark. For sure localhost won't work in this case because in Databricks all compute is cloud-based. Please follow below guide how to configure it properly on databricks:Data...
- 1 kudos
- 2724 Views
- 4 replies
- 3 kudos
Autoloader - File Notification Mode
Hello All,We have started to consume source messages/files via autoloader directory listing mode at the moment and want to convert this to file notification mode instead so consumption can be faster with no more entire directories/folder scanning. I ...
- 2724 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @K_Anudeep @szymon_dybczak how do i understand a situation when 100 jobs are running in parallel with minimal latency needed. does autoloader directly connect to the cloud queue service ? or databricks stores and manages detected files somewhere? ...
- 3 kudos
- 1137 Views
- 2 replies
- 0 kudos
Foreign Catalog Wrong Mapping - Azure SQL Database Binary Column
Hi all.I've used foreign catalog attached to azure sql databases and never had problems except in 2 situations:1) Foreign Catalogs don't support sql schemas/objects like [xxxx.yyyy].tablename. The workaround is creating views on sql database2) This i...
- 1137 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @RobsonNLPT One thing that might help to narrow this down: could you check whether the problem occurs for the entire column, or if some of the batches you receive actually contain the full (non-truncated) value?If some batches are complete but ...
- 0 kudos
- 3337 Views
- 5 replies
- 3 kudos
Schedule databricks job based on custom calendar
I want to schedule the databricks jobs based on the custom calender, like skip the job run on random days or holidays.#databricks @DataBricks @DATA
- 3337 Views
- 5 replies
- 3 kudos
- 3 kudos
Hello @DataDev! Did the suggestions shared above help address your question? If so, please consider marking one or more responses as the accepted solution. If you found another approach that worked for you, sharing it with the community would be real...
- 3 kudos
- 961 Views
- 3 replies
- 3 kudos
How to load all the previous day's data only into the newly added column of the existing delta table
How to load all the previous day's data only into the newly added column of the existing delta table? Is there any option available to do that without writing any logic?
- 961 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @shan-databricks! Did the suggestions shared above help resolve your concern? If so, please consider marking one of the responses as the accepted solution. If you found a different approach that worked for you, it would be great if you could sh...
- 3 kudos
- 6631 Views
- 8 replies
- 3 kudos
Resolved! How to create a managed iceberg table via REST catalog
We're iceberg's java lib to write managed iceberg tables in databricks. We actually can create these tables using databricks as iceberg REST catalog. But this only works when we provide a partitioning spec. This is then picked up as cluster_columns f...
- 6631 Views
- 8 replies
- 3 kudos
- 3 kudos
Why are you using the iceberg-core Java library instead of an existing open source Iceberg client (like Apache Spark)? Any of these can create a table with partitions when using Unity Catalog.
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
