- 368 Views
- 1 replies
- 0 kudos
Facilitate if/else condition in conjuction with parameters
The current state: I have a working workflow with 3 tasks with several parameters.The change: I want to modify the workflow to add 4 tasks - if/else so that based on one of the parameters (call it xyz) the workflow will not proceed after 1st task.The...
- 368 Views
- 1 replies
- 0 kudos
- 0 kudos
The logic should be simple if the xyz parameter equals 1000 then run the other 2 tasks otherwise, do not run the rest.
- 0 kudos
- 479 Views
- 2 replies
- 1 kudos
Is there a way to create a local CSV file by creating a local external table?
Hello,I have a user that would like to create a CSV file on their local file system by creating an external table (USING CSV) and specifying a local file for the path parameter using SQL. They will be running this command from a local client (DbVisua...
- 479 Views
- 2 replies
- 1 kudos
- 1 kudos
not sure if this would work, but you could run Unity Catalog locally (possible since last week) and define the csv file as a table in that local UC. then query it.
- 1 kudos
- 19885 Views
- 3 replies
- 2 kudos
Resolved! Executor heartbeat timed out
Hello, I'm trying to read a table that is located on Postgreqsl and contains 28 million rows. I have the following result:"SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in sta...
- 19885 Views
- 3 replies
- 2 kudos
- 2 kudos
Please also review the Spark UI to see the failed Spark job and Spark stage. Please check on the GC time and data spill to memory and disk. See if there is any error in the failed task in the Spark stage view. This will confirm data skew or GC/memory...
- 2 kudos
- 350 Views
- 1 replies
- 0 kudos
Cannot navigate to workspace directory in multi-node cluster
When I open a terminal on a multi-node cluster, I cannot navigate to the workspace directoryHowever, on a single node cluster, it works fineThanks in advance
- 350 Views
- 1 replies
- 0 kudos
- 0 kudos
If this cluster is backed by an AWS Graviton instance, there is currently a limitation with the web terminal not being able to interact with the Workspace Filesystem. Please give it a try in the notebook cell with the %sh magic command or switch to ...
- 0 kudos
- 32546 Views
- 5 replies
- 2 kudos
Driver is up but is not responsive, likely due to GC.
Hi all, "Driver is up but is not responsive, likely due to GC." This is the message in cluster event logs. Can anyone help me with this. What does GC means? Garbage collection? Can we control it externally?
- 32546 Views
- 5 replies
- 2 kudos
- 2 kudos
9/10 times GC is due to out of memory exceptions.@Jaron spark.catalog.clearCache() is not a configurable option, but rather a command to submit.
- 2 kudos
- 545 Views
- 2 replies
- 3 kudos
Resolved! Data prefixed by ' > '
HiI have a databricks table that has a column with a string datatypeWhen loading data, certain rows are prefixed by ' > 'Does anybody know what would cause this?It seems to be when the string is above a certain number of characters (around 200)Thanks
- 545 Views
- 2 replies
- 3 kudos
- 3 kudos
It looks like default string data length exceeded and due to that its showing this kind of symbol and extra new line in the column.
- 3 kudos
- 615 Views
- 2 replies
- 0 kudos
Service principal’s Microsoft Entra ID access token returns 400 when calling Databricks REST API
I'm using the following to call a Databricks REST API. When I use a PAT for access_token, everything works fine. When I use a Microsoft Entra ID access token, the response returns 400. The service principal has access to the workspace and is part of ...
- 615 Views
- 2 replies
- 0 kudos
- 0 kudos
hello, Try to print the repose and see are you table to see the access_token in the payload else looks like access issue.Try to go to the workspace setting and grant token access permission to the service principle.Workspace > Setting
- 0 kudos
- 3026 Views
- 5 replies
- 5 kudos
SQL IDE Support
How to use a SQL IDE with Databricks SQLDatabricks provides SQL IDE support using DataGrip and DBeaver with Databricks SQL.Let us know in the comments if you've used DataDrip or DBeaver with Databricks! Let us know if there are any other SQL IDEs you...
- 3026 Views
- 5 replies
- 5 kudos
- 5 kudos
dbeaver is perfectly working fine but I fount one issue it wont show the correct error for query.
- 5 kudos
- 1282 Views
- 4 replies
- 1 kudos
Resolved! Yml file replacing job cluster with all-purpose cluster when running a workflow
Hi all,I have been having some trouble running a workflow that consists of 3 tasks that run sequentially. Task1 runs on an all-purpose cluster and kicks off Task2 that needs to run on a job cluster. Task2 kicks off Task3 which also uses a job cluster...
- 1282 Views
- 4 replies
- 1 kudos
- 1 kudos
I don't know if you've cut off your yaml snippet, but your snippet doesn't show your job cluster with key job-cluster. Just to validate, your job cluster is also defined in your workflow yaml?Edit: Looking it it again and knowing the defaults, it loo...
- 1 kudos
- 2896 Views
- 7 replies
- 0 kudos
Resolved! How to import a function to another notebook?
Could you please provide guidance on the correct way to dynamically import a Python module from a user-specific path in Databricks Repos? Any advice on resolving the ModuleNotFoundError would be greatly appreciated.udf_check_table_exists notebook:fro...
- 2896 Views
- 7 replies
- 0 kudos
- 1345 Views
- 2 replies
- 0 kudos
Terraform Error: Cannot create sql table context deadline
I am currently trying to deploy external parquet tables to the Databricks UC using terraform. However, for some tables I get the following error:Error: cannot create sql table: Post "https://[MASKED]/api/2.0/sql/statements/": context deadline exceede...
- 1345 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @Kaniz_Fatma,thanks for your reply and sorry for the late reply from my side. I couldn't fix the problem with the databricks terraform provider unfortunately. I now switched to using liquibase to deploy tables to databricks.
- 0 kudos
- 9798 Views
- 4 replies
- 0 kudos
Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....
Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...
- 9798 Views
- 4 replies
- 0 kudos
- 0 kudos
@Debayan Mukherjee​ Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...
- 0 kudos
- 1012 Views
- 4 replies
- 0 kudos
CREATE WIDGETS in SQL Notebook attached to SQL Warehouse Doesn't Work.
I'm able to create and use widgets using the UI in my SQL notebooks, but they get lost quite frequently when the notebook is reset.There is documentation suggesting we can create widgets in code in SQL: https://learn.microsoft.com/en-us/azure/databri...
- 1012 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @JeremyH - can you please try adding like the below in your query and see if widgets are getting populated? {{parameter_name }}
- 0 kudos
- 768 Views
- 2 replies
- 0 kudos
Resolved! Unzipping with Serverless Compute
HiI have started using serverless compute but have come across the limitation that I cannot use the local filesystem for temporarily storing the files and directories before moving them to where they need to be in adls.Does anyone have a way of unzip...
- 768 Views
- 2 replies
- 0 kudos
- 0 kudos
Do you have the ability to make a Unity Catalog volume? You could use it as temporary storage before migrating the files to adls.
- 0 kudos
- 4407 Views
- 5 replies
- 3 kudos
How to fix plots and image color rendering on Notebooks?
I am currently running dark mode for my Databricks Notebooks, and am using the "new UI" released a few days ago (May 2023) and the "New notebook editor."Currently all plots (like matplotlib) are showing wrong colors. For example, denoting:```... p...
- 4407 Views
- 5 replies
- 3 kudos
- 3 kudos
Any updated on this issue? This is a huge drawback to use the dark theme
- 3 kudos