- 5126 Views
- 1 replies
- 1 kudos
Clusters are really slow
We have two environments for our Azure Databricks. Dev and Prod. We had clusters created and tested in Dev environment, then they were exported to the prod environment through APIs. The clusters in Dev are performing as expected. Whereas, the cluster...
- 5126 Views
- 1 replies
- 1 kudos
- 1 kudos
Both Prod and Dev are connect to unity catalog and I am working with the same table in both the envs. Can something done during the creation of workspace itself affect the performance of clusters? Do clusters update to latest Databricks runtime versi...
- 1 kudos
- 1043 Views
- 0 replies
- 0 kudos
How to check if autoloader processed new file
Hi,My scenario is this: autoloader loads data from ADLS once a day.data are merged into the bronze zone.bronze table has CDF turned on.read CDF and merge into the silver.all works fine, but when there is not new file for autoloader to process, CDF ta...
- 1043 Views
- 0 replies
- 0 kudos
- 1706 Views
- 2 replies
- 0 kudos
SQL wharehouse do not work with power bi online service
Whenever i try to use a SQL Wharehouse serverless cluster on a power bi dataset it does not refresh on the power bi online service. It does work normally for other types of databricks clusters. The catalog is being defined on the power query import.I...
- 1706 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi,We have the exact same issue, even if we specify the catalog in the connection parameters.However, Oauth authentication through a dataflow (instead of from Power Query Desktop) works fine. In Desktop we are in version 2.122.746.0, but the issue is...
- 0 kudos
- 694 Views
- 0 replies
- 0 kudos
we have enabled compute schema in system tables, not able to see cluster and node_type tables
we have enabled system tables, under system tables we are able to see compute schema, under that we are not able to see cluster and node_type in azure databricks. do we have any limitations for above tables
- 694 Views
- 0 replies
- 0 kudos
- 692 Views
- 1 replies
- 0 kudos
Issue: Verification email not received to sign up for Community Edition
Hello! I'm following this documentation : https://docs.databricks.com/en/getting-started/community-edition.htmlAnd I can't succeed the step 4) because I don't receive any email to verify my email address. I have checked the spam, no email.Is there so...
- 692 Views
- 1 replies
- 0 kudos
- 0 kudos
@farah please check the junk mail folder and also message quarantine if you are using M$ services.The verification mail goes out fine. However, it may show up in Junk | Quarantine.
- 0 kudos
- 4277 Views
- 0 replies
- 0 kudos
Job Clusters With Multiple Tasks
Hi all,I'm trying to do creating one job cluster with one configuration or specification which has a workflow and this workflow needs to have 3 dependent tasks as a straight line. For example, t1->t2->t3. In databricks there are some constraints also...
- 4277 Views
- 0 replies
- 0 kudos
- 789 Views
- 0 replies
- 0 kudos
Disable DLT notifications in development mode
Hello,We are using DLT pipelines for many of our jobs with notifications on failures to slack.Wondering if there is a clean way to disable the alerts when in development mode. It does make sense to have it turned off in dev, doesn't it?
- 789 Views
- 0 replies
- 0 kudos
- 2100 Views
- 2 replies
- 0 kudos
Auotoloader-"cloudFiles.backfillInterval"
1. How to use cloudFiles.backfillInterval option in a notebook?2. Does It need to be any set of the property?3. Where is exactly placed readstream portion of the code or writestream portion of the code?4. Do you have any sample code?5. Where we find ...
- 2100 Views
- 2 replies
- 0 kudos
- 0 kudos
1.Is the following code correct for specifying the .option("cloudFiles.backfillInterval", 300)?df = spark.readStream.format("cloudFiles") \.option("cloudFiles.format", "csv") \.option("cloudFiles.schemaLocation", f"dbfs:/FileStore/xyz/back_fill_opti...
- 0 kudos
- 1222 Views
- 0 replies
- 0 kudos
Unable to Login to community edition
I am unable to login in databricks community account, my email was "sureshsuthar1251@gmail.com". When i try to login it says account not found.
- 1222 Views
- 0 replies
- 0 kudos
- 1977 Views
- 1 replies
- 0 kudos
Can the same Databricks account be used for both AWS and Azure
I am testing Databricks with non-AWS S3 object storage and want to test it with Databricks on Azure and Databricks on AWS. My Databricks account is currently using Databricks on AWS with metadata and a single node compute running. Can the same accou...
- 1977 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you Kaniz for posting the link. Looking at that, I believe the answer is:This is not possible in Databricks for now.
- 0 kudos
- 2595 Views
- 3 replies
- 0 kudos
Resolved! Databricks data engineer associate Exam got suspended.
@Cert-Team I have registered for Databricks certified data engineer associate exam and I have done all biometric and other prerequisites required and launched my exam. While writing the exam, it's exited twice with some technical issue although I don...
- 2595 Views
- 3 replies
- 0 kudos
- 3052 Views
- 1 replies
- 0 kudos
Connecting to Databricks using Python(VS Code)
I'm trying to connect to tables/views in Databricks using Python(via VS Code). However, I'm getting the following error:- File "C:\Users\XXXXXXXX\AppData\Roaming\Python\Python311\site-packages\urllib3\util\retry.py", line 592, in incrementraise MaxRe...
- 3052 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm trying to connect to tables/views in Databricks using Python(via VS Code). However, I'm getting the following error:- File "C:\Users\XXXXXXXX\AppData\Roaming\Python\Python311\site-packages\urllib3\util\retry.py", line 592, in incrementraise MaxRe...
- 0 kudos
- 5493 Views
- 0 replies
- 0 kudos
Get Network usage read and write from a job cluster on databricks
Hi ! I am writing because I am trying to get the data from network usage from a databricks job cluster. I know that the data is on spark-ui / storage but when the job cluster is terminate I cannot acces to the information , and I trying to make some ...
- 5493 Views
- 0 replies
- 0 kudos
- 9243 Views
- 3 replies
- 3 kudos
Azure Databricks User Alerts: query_result_table
Hi,How can I set up the notification email to show all of rows from query_result_table, not only first 10?
- 9243 Views
- 3 replies
- 3 kudos
- 3 kudos
I see that my previous message has been cut. So, I just wanted to check where can I change email setting using the code above, because when I open User > Account Settings > Notifications I can see only option like this: there is no place to configur...
- 3 kudos
- 3756 Views
- 1 replies
- 0 kudos
Unable to use job cluster for task in workflows
Hi,I have a workflow setup in Databricks using 12.2 LTS ML.I am trying to use a job cluster for the task but i am getting the following error: Spark Conf: ‘spark.databricks.acl.enabled’ is not allowed when choosing an access modeAs a result I have to...
- 3756 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,I have a workflow setup in Databricks using 12.2 LTS ML.I am trying to use a job cluster for the task but i am getting the following error: Spark Conf: ‘spark.databricks.acl.enabled’ is not allowed when choosing an access modeAs a result I have to...
- 0 kudos
