- 690 Views
- 4 replies
- 0 kudos
MetaStore Issues
Hi,I am using Student Account in Azure and created databricks workspace. I am trying to locate Manage Account to create MetaStore but I am not successful. Would need your help.no Manage Account option available.If you see I am an Admin, Thanks for a...
- 690 Views
- 4 replies
- 0 kudos
- 0 kudos
Databricks student account, may have certain limitations compared to a full Azure Databricks account. For student accounts, you might not have the necessary permissions to create a Unity Catalog metastore. Typically, creating and managing a metastore...
- 0 kudos
- 4199 Views
- 5 replies
- 1 kudos
Cannot create storage credential without Contributor role
Hello,I am trying to create a Storage Credential. I have created the access connector and gave the managed identity "Storage Blob Data Owner" permissions. However when I want to create a storage credential I get the following error:Creating a storage...
- 4199 Views
- 5 replies
- 1 kudos
- 1 kudos
Did someone find a solution?{ "error_code": "RESOURCE_DOES_NOT_EXIST", "message": "Refresh token not found for userId: Some(2302042022180399)", "details": [ { "@type": "type.googleapis.com/google.rpc.RequestInfo", "request_id": "d731471b-b...
- 1 kudos
- 176 Views
- 1 replies
- 0 kudos
Scala notebooks don't automatically print variables
It seems like with Scala notebook if I declare some variables or have important statements then the cell runs it will automatically print out the variables and import statements. Is there a way to disable this, so only explicit println are output?
- 176 Views
- 1 replies
- 0 kudos
- 0 kudos
There isn't a configuration that can be set to True/False and control this behavior for some statements. This output is part of Databrick's interactive notebook design, where all evaluated statements—such as imports, variable declarations, and expres...
- 0 kudos
- 4746 Views
- 9 replies
- 4 kudos
Resolved! Databricks SQL: catalog of each query
Currently, we are migrating from hive metastore to UC. We have several dashboards and a huge number of queries whose catalogs have been set to hive_metastore and using <db>.<table> access pattern.I'm just wondering if there's a way to switch catalogs...
- 4746 Views
- 9 replies
- 4 kudos
- 4 kudos
May you can also have a look here, if you need hot fix https://github.com/databrickslabs/ucx
- 4 kudos
- 282 Views
- 2 replies
- 0 kudos
Glob pattern for copy into
I am trying to load some files in my Azure storage container using copy into method. The files have a naming convention of "2023-<month>-<date> <timestamp>".csv.gz. All the files are in one folder. I want to load only files for month 2. So I've used...
- 282 Views
- 2 replies
- 0 kudos
- 0 kudos
TL;DR Try removing the trailing slash in the FROM value. The trailing slash in FROM confuses the URI parser, making it think that PATTERN might be an absolute path rather than a relative one. The error message points to a problem not with respect to ...
- 0 kudos
- 12574 Views
- 9 replies
- 2 kudos
Kafka timout
Hello, I am trying to read topics from a kafaka stream but I am getting the time out error below.java.util.concurrent.ExecutionException: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: describeT...
- 12574 Views
- 9 replies
- 2 kudos
- 2 kudos
try this :.option('kafka.session.timeout.ms', 200000).option('group.max.session.timeout.ms', 7200000) kafka.session.timeout.ms: Specifies the timeout for detecting consumer failures.group.max.session.timeout.ms: Sets the maximum allowed session timeo...
- 2 kudos
- 214 Views
- 2 replies
- 0 kudos
FeatureEngineeringClient and Databricks Connect
Hello everyone, I want to use Databricks Connect to connect externally to my clusters and run code, and while Databricks connect works without any issue, like this: ```from databricks.sdk.core import Config config = Config(cluster_id="XXXX")spark = S...
- 214 Views
- 2 replies
- 0 kudos
- 12631 Views
- 12 replies
- 2 kudos
Resolved! Dbfs init script migration
I need help with migrating from dbfs on databricks to workspace. I am new to databricks and am struggling with what is on the links provided.My workspace.yml also has dbfs hard-coded. Included is a full deployment with great expectations.This was don...
- 12631 Views
- 12 replies
- 2 kudos
- 2671 Views
- 5 replies
- 0 kudos
Default maximum spark streaming chunk size in delta files in each batch?
working with delta files spark structure streaming , what is the maximum default chunk size in each batch?How do identify this type of spark configuration in databricks?#[Databricks SQL]​ #[Spark streaming]​ #[Spark structured streaming]​ #Spark​
- 2671 Views
- 5 replies
- 0 kudos
- 0 kudos
doc - https://docs.databricks.com/en/structured-streaming/delta-lake.html Also, what is the challenge while using foreachbatch?
- 0 kudos
- 269 Views
- 4 replies
- 1 kudos
workflow fails when ran using a job cluster but not in shared cluster
Hi,We have a workflow which calls 3 notebooks and when we run this workflow using shared cluster it runs fine, but when ran with job cluster , one of the notebooks fail.This notebook uses sql function Lpad and looks like it errors because of it. Has ...
- 269 Views
- 4 replies
- 1 kudos
- 465 Views
- 5 replies
- 0 kudos
Delta live type 2 scd Liquid clustering on Start and end dates
I've created a DLT pipeline that creates type 2 SCDs and often the __Start_at and __end_at columns are beyond the first 32 columns for stat collection.I'd like to add these columns to liquid clustering without increasing the number of columns in the ...
- 465 Views
- 5 replies
- 0 kudos
- 0 kudos
Are these responses generated by chatgpt? They don't answer the question and very much have the tone of generative AI
- 0 kudos
- 284 Views
- 2 replies
- 0 kudos
Error connecting to MySQL from Databricks: (2003, "Can't connect to MySQL server")
Hello Community,I'm facing an issue connecting to a MySQL database hosted on AWS RDS from within a Data bricks notebook. My Python script to connect to MySQL works fine locally, but when I run it in Data bricks, I receive this error:Error connecting ...
- 284 Views
- 2 replies
- 0 kudos
- 333 Views
- 2 replies
- 1 kudos
Resolved! Asset Bundles pause_status Across Different Environments
HiQuestion probably around best practices, but curious if someone else has dealt with a similar situation. I have 2 Databricks workspaces - one for Dev and one for Prod. Had to be two workspaces because Azure Landing Zones had to be air gapped from e...
- 333 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @RangaSarangan ,We have faced same issue and solved using databricks workflow API and json file for job metadata that consist job and thier respective status for each env.You can create azure devops that run after your cicd pipeline and change the...
- 1 kudos
- 175 Views
- 1 replies
- 0 kudos
CREATED WIDGET - SQL syntax - how do I specify a label?
What is the syntax in SQL for creating a widget in a notebook with a label?This documentation says "The last argument is label, an optional value for the label shown over the widget text box or dropdown."The one example provided on that page doesn't ...
- 175 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @eriodega I do not know how it works with sql but a possibility is to use a python cell in your SQL notebook and create it like this:%python dbutils.widgets.text("some_name", "a great default", "some label/description")All the best
- 0 kudos
- 10278 Views
- 3 replies
- 1 kudos
Schedule a Notebook Dashboard
Hey all,I have a workflow that updates a delta table, and then runs a notebook that generates a dashboard. I was hoping that by adding this second step that the dashboard would get updated to show the most current data, instead of the user needing to...
- 10278 Views
- 3 replies
- 1 kudos
- 1 kudos
@ehpogue at the end of the day I just used chatgpt to rewrite a bunch of python code into SQL and mash together all of the temporary views into one giant query. hacky but it did work
- 1 kudos