- 14679 Views
- 5 replies
- 0 kudos
Databricks SQL Warehouse did not auto stop after specified 90 minute interval - why not?
In this specific case, we're running a 2XSmall SQL Warehouse on Databricks SQL.In looking at the SQL Warehouse monitoring log for this cluster, we noticed:final query executed by user at 10:26 on 2023-06-20no activity for some time, yet cluster remai...
- 14679 Views
- 5 replies
- 0 kudos
- 0 kudos
Also dealing with the same issue! Anybody has any idea how to check it? There is no sort of logs and no actual query happens
- 0 kudos
- 4610 Views
- 7 replies
- 0 kudos
Databricks Lakehouse Fundamentals BadgeI have successfully passed the test after completion of the course "Databricks Lakehouse Fundamentals"...
Databricks Lakehouse Fundamentals BadgeI have successfully passed the test after completion of the course "Databricks Lakehouse Fundamentals". But I have'nt recieved any badge. I have been provided with a certificate only. Please provide me with th...
- 4610 Views
- 7 replies
- 0 kudos
- 5893 Views
- 4 replies
- 2 kudos
Turn on full screen for databricks training videos
It seems full screen option for databricks training videos are turned off. How to turn it on ?
- 5893 Views
- 4 replies
- 2 kudos
- 2 kudos
It sounds like the full-screen option is disabled by the platform hosting the Databricks training videos. If that's the case, it may be out of your control. However, you could try these workarounds:Browser Zoom: Use the zoom feature (Ctrl + "+" on Wi...
- 2 kudos
- 3012 Views
- 2 replies
- 1 kudos
Resolved! Foreign key constraint in a dlt pipeline
As primary/foreign key constraints are now supported/available in Databricks, how are foreign key constraints handled in a dlt pipeline, i.e if a foreign key constraint is violated, is the record logged as a data quality issue and still added to the ...
- 3012 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Mario_D!While primary & foreign key constraints are generally available in Databricks Runtime 15.2 and above, they are strictly informational only.This means that a primary key will not prevent duplicates from being added to a table and a foreign...
- 1 kudos
- 16172 Views
- 5 replies
- 1 kudos
Running unit tests from a different notebook (using Python unittest package) doesn't produce output (can't discover the test files)
I have a test file (test_transforms.py) which has a series of tests running using Python's unittest package. I can successfully run the tests inside of the file with expected output. But when I try to run this test file from a different notebook (run...
- 16172 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello, I have exactly the same issue.In my case, using the ipytest library from Databricks clusters, this is the error that occurs when I try to run the tests:EEEEE [100%]============================================== ERRORS =========================...
- 1 kudos
- 4365 Views
- 1 replies
- 0 kudos
400 Unable to load OAuth Config
I've enabled SSO for my Databricks account with Okta as the identity provider and tested the integration is working. I'm now trying to implement an on-behalf-of token exchange so that my API can make authenticate requests to Databricks's API (e.g. ) ...
- 4365 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello ,Were able to fix the issue ? I'm trying to implement the same thing
- 0 kudos
- 5473 Views
- 3 replies
- 0 kudos
REST API for Pipeline Events does not return all records
I'm using the REST API to retrieve Pipeline Events per the documentation:https://docs.databricks.com/api/workspace/pipelines/listpipelineeventsI am able to retrieve some records but the API stops after a call or two. I verified the number of rows us...
- 5473 Views
- 3 replies
- 0 kudos
- 0 kudos
You can leverage this code base. It works as expected using "next_page_token" parameter-Don't forget to mark this solution as correct if this helped you import requests token = 'your token' url = 'your URL' params = {'expand_tasks': 'true'} header...
- 0 kudos
- 1912 Views
- 1 replies
- 1 kudos
Run now with different parameters doesn't pass parameter to pipeline tasks
I have a job with some tasks. Some of the tasks are pipeline_task's some are notebook_task's.When I run the job with "Run now with different parameters" and enter a new key-value, I see that the key-value is available in the notebook_task's with dbut...
- 1912 Views
- 1 replies
- 1 kudos
- 1 kudos
As per docs it seems that pipeline task type is currently not supported to pass parameters: https://docs.databricks.com/en/jobs/create-run-jobs.html#pass-parameters-to-a-databricks-job-taskYou could create a notebook task that runs before your pipeli...
- 1 kudos
- 5817 Views
- 3 replies
- 1 kudos
Resolved! Delta Live Tables + Databricks Assets Bundles
Hi,I'm trying to setup CICD pipeline for Delta Live Table jobs using Databricks Bundles. I have a problem with path to notebook in pipeline. According to this example:https://docs.databricks.com/en/delta-live-tables/tutorial-bundles.htmlYAML file sho...
- 5817 Views
- 3 replies
- 1 kudos
- 1 kudos
I had this error once.you need to specify the extension of your file. If you set the notebook to be python, then it must be .py at the end, likewise .sql if you used SQL libraries: - notebook: path: ${workspace.file_path}/datab...
- 1 kudos
- 1741 Views
- 2 replies
- 0 kudos
Excel Formula results
Hi all,Just wanted to raise a question regarding Databricks workbooks and viewing the results in the cells. For the example provided in the screenshot I want to view the results of an excel formula that has been applied to a cell in our workbooks. Fo...
- 1741 Views
- 2 replies
- 0 kudos
- 0 kudos
@Kurtis_R do you want to display the value of 45 or formula of how 45 is achieved.?
- 0 kudos
- 1173 Views
- 1 replies
- 0 kudos
I don't have any Last name. what should give in the mandatory last name field?
Hi, I was about to register for Databricks certified Data engineer Associate exam. while registering for the exam, it is asking for Last name which is the mandatory field. But none of my Government proofs have last name, only first name is there. wha...
- 1173 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,To proceed with the registration, please enter a space or a full stop in the last name field. This should allow you to continue with the process, feel free to reach out if you need any further assistance.Best Regards,Anushree
- 0 kudos
- 7171 Views
- 1 replies
- 0 kudos
Photon Benchmark
I'm conducting my own comparative study between a cluster with Photon enabled and a cluster without Photon to see what improvements occur. According to Databricks, there should be up to 12x better performance, but I'm only finding about a 20% improve...
- 7171 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Enrique1987 ,You can find more information about photon in below whitepaper:https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf
- 0 kudos
- 17977 Views
- 5 replies
- 6 kudos
Resolved! How to read JSON files embedded in a list of lists?
HelloI am trying to read this JSON file but didn't succeed You can see the head of the file, JSON inside a list of lists. Any idea how to read this file?
- 17977 Views
- 5 replies
- 6 kudos
- 6 kudos
The correct way to do this without using open, which will work only with local/mounted files is to read the files as binaryfile and then you will get the entire json string on each row, from there you can use from_json() and explode() to extract the ...
- 6 kudos
- 7583 Views
- 4 replies
- 0 kudos
Resolved! Unable to call UDF inside the Spark SQL: RuntimeError: SparkSession should be create
Here is how I define the UDF inside the file udf_define.py:from pyspark.sql.functions import length, udf from pyspark.sql.types import IntegerType from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() def strlen(s): ret...
- 7583 Views
- 4 replies
- 0 kudos
- 7053 Views
- 3 replies
- 2 kudos
Problems with Azure Databricks
Hi,I want to use Databricks the first time, and am having many problems and confusions. Please help me resolve them.1. I created a free Databricks Community account on Azure and get error when creating the cluster/compute"Azure Quota Exceeded Excepti...
- 7053 Views
- 3 replies
- 2 kudos
- 2 kudos
you must have created some resources with public IP addresses in your azure subscription, ie: storage account, etc...Try to avoid using public IPs as much as possible to secure your tenant/subscription.try to find which of your Azure resources are us...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
