- 701 Views
- 1 replies
- 0 kudos
Facing Some Issues with Tablet PC and Databricks Product – Any Advice?
Hello everyone,I’m having some trouble using Databricks SQL Analytics v2.1 on my tablet PC, and I was wondering if anyone here has had similar experiences or could offer some advice.The main issues I’m facing are:Performance Slowdowns: When I run com...
- 701 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jonathanjone , 1 - Performance Slowdowns could be because of the warehouse size and the query count, if there are many queries, the warehouse has a limit of 10 query in parallel, else you see the query being queued. You could also check if the q...
- 0 kudos
- 949 Views
- 1 replies
- 1 kudos
Resolved! Has the numUpdateRetryAttempts property been deprecated?
I noticed there is a numUpdateRetryAttempts property mentioned in the the document https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/properties used for configing the retry times of any DLT pipeline, but I cannot find it in the DL...
- 949 Views
- 1 replies
- 1 kudos
- 1 kudos
According to the Delta Live Tables properties reference, pipelines.numUpdateRetryAttempts is a recognized configuration parameter. It specifies the maximum number of attempts to retry an update before failing the update when a retryable failure occur...
- 1 kudos
- 2110 Views
- 3 replies
- 0 kudos
Implementing Conditional Logic for Dependent Tasks Using SQL Output and Task Values
Hi team,I'm working on setting up a workflow with task dependencies where a subsequent task should execute conditionally, based on the result of a preceding SQL task. Specifically, I need to evaluate an if/else condition on the output of the SQL quer...
- 2110 Views
- 3 replies
- 0 kudos
- 888 Views
- 1 replies
- 0 kudos
QGIS python command in Databricks notebook
Hello, I would like to run a qgis python script in a databricks notebook. Currently, databricks doesn't recognize the qgis package. For example, i'd like to run this small example : from qgis.core import *# Supply path to qgis install locationQgsAppl...
- 888 Views
- 1 replies
- 0 kudos
- 0 kudos
QGIS is not directly usable in Scala or Spark environments as it is a standalone Geographic Information System (GIS) application, installing and using it directly within Databricks may not be straightforward due to the specific environment and depend...
- 0 kudos
- 1323 Views
- 4 replies
- 0 kudos
MetaStore Issues
Hi,I am using Student Account in Azure and created databricks workspace. I am trying to locate Manage Account to create MetaStore but I am not successful. Would need your help.no Manage Account option available.If you see I am an Admin, Thanks for a...
- 1323 Views
- 4 replies
- 0 kudos
- 0 kudos
Databricks student account, may have certain limitations compared to a full Azure Databricks account. For student accounts, you might not have the necessary permissions to create a Unity Catalog metastore. Typically, creating and managing a metastore...
- 0 kudos
- 7387 Views
- 5 replies
- 1 kudos
Cannot create storage credential without Contributor role
Hello,I am trying to create a Storage Credential. I have created the access connector and gave the managed identity "Storage Blob Data Owner" permissions. However when I want to create a storage credential I get the following error:Creating a storage...
- 7387 Views
- 5 replies
- 1 kudos
- 1 kudos
Did someone find a solution?{ "error_code": "RESOURCE_DOES_NOT_EXIST", "message": "Refresh token not found for userId: Some(2302042022180399)", "details": [ { "@type": "type.googleapis.com/google.rpc.RequestInfo", "request_id": "d731471b-b...
- 1 kudos
- 968 Views
- 1 replies
- 0 kudos
Scala notebooks don't automatically print variables
It seems like with Scala notebook if I declare some variables or have important statements then the cell runs it will automatically print out the variables and import statements. Is there a way to disable this, so only explicit println are output?
- 968 Views
- 1 replies
- 0 kudos
- 0 kudos
There isn't a configuration that can be set to True/False and control this behavior for some statements. This output is part of Databrick's interactive notebook design, where all evaluated statements—such as imports, variable declarations, and expres...
- 0 kudos
- 6868 Views
- 7 replies
- 4 kudos
Resolved! Databricks SQL: catalog of each query
Currently, we are migrating from hive metastore to UC. We have several dashboards and a huge number of queries whose catalogs have been set to hive_metastore and using <db>.<table> access pattern.I'm just wondering if there's a way to switch catalogs...
- 6868 Views
- 7 replies
- 4 kudos
- 4 kudos
May you can also have a look here, if you need hot fix https://github.com/databrickslabs/ucx
- 4 kudos
- 1265 Views
- 2 replies
- 0 kudos
Glob pattern for copy into
I am trying to load some files in my Azure storage container using copy into method. The files have a naming convention of "2023-<month>-<date> <timestamp>".csv.gz. All the files are in one folder. I want to load only files for month 2. So I've used...
- 1265 Views
- 2 replies
- 0 kudos
- 0 kudos
TL;DR Try removing the trailing slash in the FROM value. The trailing slash in FROM confuses the URI parser, making it think that PATTERN might be an absolute path rather than a relative one. The error message points to a problem not with respect to ...
- 0 kudos
- 22190 Views
- 12 replies
- 2 kudos
Resolved! Dbfs init script migration
I need help with migrating from dbfs on databricks to workspace. I am new to databricks and am struggling with what is on the links provided.My workspace.yml also has dbfs hard-coded. Included is a full deployment with great expectations.This was don...
- 22190 Views
- 12 replies
- 2 kudos
- 4812 Views
- 5 replies
- 0 kudos
Default maximum spark streaming chunk size in delta files in each batch?
working with delta files spark structure streaming , what is the maximum default chunk size in each batch?How do identify this type of spark configuration in databricks?#[Databricks SQL]​ #[Spark streaming]​ #[Spark structured streaming]​ #Spark​
- 4812 Views
- 5 replies
- 0 kudos
- 0 kudos
doc - https://docs.databricks.com/en/structured-streaming/delta-lake.html Also, what is the challenge while using foreachbatch?
- 0 kudos
- 1181 Views
- 4 replies
- 1 kudos
workflow fails when ran using a job cluster but not in shared cluster
Hi,We have a workflow which calls 3 notebooks and when we run this workflow using shared cluster it runs fine, but when ran with job cluster , one of the notebooks fail.This notebook uses sql function Lpad and looks like it errors because of it. Has ...
- 1181 Views
- 4 replies
- 1 kudos
- 1887 Views
- 5 replies
- 0 kudos
Delta live type 2 scd Liquid clustering on Start and end dates
I've created a DLT pipeline that creates type 2 SCDs and often the __Start_at and __end_at columns are beyond the first 32 columns for stat collection.I'd like to add these columns to liquid clustering without increasing the number of columns in the ...
- 1887 Views
- 5 replies
- 0 kudos
- 0 kudos
Are these responses generated by chatgpt? They don't answer the question and very much have the tone of generative AI
- 0 kudos
- 1016 Views
- 2 replies
- 0 kudos
Error connecting to MySQL from Databricks: (2003, "Can't connect to MySQL server")
Hello Community,I'm facing an issue connecting to a MySQL database hosted on AWS RDS from within a Data bricks notebook. My Python script to connect to MySQL works fine locally, but when I run it in Data bricks, I receive this error:Error connecting ...
- 1016 Views
- 2 replies
- 0 kudos
- 3218 Views
- 2 replies
- 3 kudos
Resolved! Asset Bundles pause_status Across Different Environments
HiQuestion probably around best practices, but curious if someone else has dealt with a similar situation. I have 2 Databricks workspaces - one for Dev and one for Prod. Had to be two workspaces because Azure Landing Zones had to be air gapped from e...
- 3218 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @RangaSarangan ,We have faced same issue and solved using databricks workflow API and json file for job metadata that consist job and thier respective status for each env.You can create azure devops that run after your cicd pipeline and change the...
- 3 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 456 | |
| 349 | |
| 287 |
