When I run notebooks from within a notebook using `dbutils.notebook.run`, I see a nice progress table that updates automatically, showing the execution time, the status, links to the notebook and it is seamless.My goal now is to execute many notebook...
Hi @Retired_mod I have seen numerous post by you. Thanks for continuously providing support. Can you or your colleagues help on this. We have a basic user which assumes a role with S3 policy to a specific bucket. When we try to read the bucket from D...
Py4JJavaError: An error occurred while calling o503.json. : java.nio.file.AccessDeniedException: s3a://xxxxxx.json: shaded.databricks.org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by AwsCredentialContextTokenProvid...
Data source errorDataSource.Error: ODBC: ERROR [42000] [Microsoft][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query.Error message from server: org.apache.hive.service.cli.HiveSQLException: Error running query: [DEL...
This view used to run till last week and suddenly started giving this error. So need to check what can be the reason for this issue
I recently faced an issue that took good hours to identify. I'm loading an environment variable with a secretENVVAR: {{secrets/scope/key}}The secret is loaded in my application, I could verify it's there, but its value is not correct. I realised tha...
Hi team, is there an update or fix for this?
The following code works perfectly fine: df = spark.createDataFrame([('A', 1), ('B', 2)]) df.createOrReplaceTempView('temp') spark.sql(""" SELECT IDENTIFIER(:col) FROM temp """, args={ "col": "_1" } ).display(...
If it helps anyone else I found this article that described a few limitations:https://community.databricks.com/t5/technical-blog/how-not-to-build-an-execute-immediate-demo/ba-p/82167
I've a job in Databricks running multiple tasks in parallel. Those tasks read parameters of the job using the utility of dbutils. I'm getting the following error when trying to read parameters in my different tasks:com.databricks.common.client.Databr...
Hi all,Our solution has been to use job parameters and dynamic value references. These are read using dbutils.widgets.get() instead of dbutils.jobs.taskValues.get(). Now, our ETL is working well again.Pass context about job runs into job tasks - Azur...
Hello,We have set up a lakehouse in Databricks for one of our clients. One of the features our client would like to use is the Unity Catalog data lineage view. This is a handy feature that we have used with other clients (in both AWS and Azure) witho...
Hello,It's been a few months since this exchange. The feature limitation is not documented anywhere - documents imply that this should be working in GCP:https://docs.gcp.databricks.com/en/data-governance/unity-catalog/data-lineage.htmlIs this feature...
Hello,Since a week ago, our notebook are stuck in running on the firsts cells which import python module from our github repository which is cloned in databricks.The cells stays in running state and when we try to manually cancel the jobs in databric...
Hello everyone, nice to greet you. I have a question about the data lifecycle in ADLS. I know ADLS has its own rules, but they aren't working properly because I have two ADLS accounts: one for hot data and another for cool storage where the informati...
Hello guys. I use pyspark in my daily life. A demand has arisen to collect information in Jira. I was able to do this via Talend ESB, but I wouldn't want to use different tools to get the job done. Do you have any example of how to extract data from ...
Hi,There is also a new Databricks for Jira add-on on the Atlassian Marketplace. It is easy to setup and exports are directly created within Jira. They can be one-time, scheduled, or real-time. It can also export additional Jira data such as Assets, C...
Hello,I am currently working through an issue I am seeing when querying a Lakehouse Federation UC table in a workflow. I am using pyspark to query a table through Lakehouse Federation which returns a result based on the query. When running it in a ce...
I hit a similar error when trying to view FC data using a cluster with a lower Databricks Runtime.Needs to be DBR 13.1 or higher.
We created a new Service Principal (SP) on Azure and would like to transfer the ownership of the databases and tables created with the old SP. The issue is that these databases and tables are not visible to the users using the new SP.I am using a Hiv...
Regarding the [PARSE_SYNTAX_ERROR] Syntax error at or near 'OWNER'.Remember to wrap the new owner name in the SQL statement with the Grave Accent (`) as the below sample. ALTER SCHEMA schema_name OWNER TO `new_oner_name`;
I'm constantly getting frozen "Cancelling" or "Running" cells since a few days when prototyping some stuff in notebooks. Examples:And the same for instantiating functions (should be instant), neither finishing or able to cancelThis button doesn't wor...
Seems to have resolved itself today. Can't reproduce it anymore
I know how to use Spark in Databricks to create a CSV, but it always has lots of side effects.For example, here is my code:file_path = “dbfs:/mnt/target_folder/file.csv”df.write.mode("overwrite").csv(file_path, header=True)Then what I got isA folder ...
Hi @guangyi ,To disable _commited_xxx, _started_xxx and _SUCCSSS you must set below spark options: spark.conf.set("spark.databricks.io.directoryCommit.createSuccessFile","false") spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "f...
I am trying to create a JAR for a Azure Databricks job but some code that works when using the notebook interface does not work when calling the library through a job. The weird part is that the job will complete the first run successfully but on an...
I am facing similar issue when trying to use from_utc_timestamp function. I am able to call the function from databricks notebook but when I use the same function inside my java jar and running as a job in databricks, it is giving below error. Analys...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group