spark 3.4 and databricks 13 introduce two new types of timestamps for handling time zone information:- TIMESTAMP WITH LOCAL TIME ZONE: This type assumes that the input data is in the session's local time zone and converts it to UTC before processing....
I am going to use the newly released DLT with UC.But it keeps getting access denied. As I keep tracking the reasons, it seems that an account. ID other than my account ID or Databricks account ID is being requested.I cannot use '*' in principal attri...
Every service on AWS, an SQS queue, and all the other services in your stack using that queue will be configured with minimal permissions, leading to access issues. So, make sure you get your IAM policies set up correctly before deploying to producti...
Thanks @Kaniz Fatma​ for selecting this as the best answer, Keep adding questions by that we can put our views and people get guidance And the databricks community can grow more.
From yesterday, suddenly clusters do not start and are in the pending state indefinitely (more than 30 minutes). From a previous post, I tried to add 443 port to the firewall but it doesn't help. In the clusters page, the message says: Finding instan...
I tried log in to > https://credentials.databricks.com/ with a registered email address and not able to find any Badge there. I need badge for the test i taken and passed, do needful.
Hello!The same situation is valid for me. I couldn't reach my badge from the following link. Could you help me too? @Vidula Khanna​ https://customer-academy.databricks.com/legacy/lms/index.php%3Fr%3DmyActivities/index%26tab%3Dbadges
New unified Databricks navigation - New Release to Public preview check it out :Databricks release a new navigation experience to public preview. The goal is to reduce clicks and context switches required to complete tasks. The new experience include...
Databricks MarketplaceDatabricks Marketplace, an open forum for exchanging data products. Databricks Marketplace takes advantage of Delta Sharing to give data providers the tools to share data products securely and data consumers the power to explore...
MessageCluster terminated. Reason: Spark Image Download FailureHelpFailed to set up spark container due to an image download failure: Exception when downloading spark image: Stdout: /usr/local/bin/fastar /usr/local/bin/fastarIhave tried enabling the ...
Hi, This error can be surfaced due to various reasons, such as networking , slowness etc.. Is there any other significant errors showing up in the driver logs? Is this for a single cluster? Is this a new workspace? You can start with checking on the ...
I configured databricks-connect locally and want to run spark code against a remote cluster.I verified `databricks-connect test` passes and it can connect to remote databricks cluster.However when I query a tables or read parquet from s3, it fails wi...
@Du Liu​ :The error message suggests that there is no API token found in the local properties. This could be the cause of the failure when trying to access the tables or read parquet files from S3.To fix this issue, you need to ensure that the API to...
I am trying to use the following code to create a deltatable%sqlCREATE TABLE rectangles(a INT, b INT, area INT GENERATED ALWAYS AS IDENTITY (START WITH 1, STEP BY 1))I don't know why but I am always getting the ParseException error.I tried all other ...
I am building a bronze table with CDF-enables in these steps:Initially, Reading json file from landing zone and write to table locationdf = spark.readStream.format("cloudFiles") \
.option("cloudFiles.schemaLocation", <schema_loc>) \
.option("clou...
HiOn December 28th ,I attempted the Databricks Certified Machine Learning Professional exam for 1st time , unfortunately I ended up by failing grade. My passing grade was 70%, and I received 68.33%.I am planning to reattempt the exam, Could you kindl...
Getting below error Context: Using Databricks shared interactive cluster for scheduled run multiple parallel jobs at the same time after every 5 mins. When I check Ganglia, driver node's memory reaches almost max and then restart of driver happens an...
I am using the databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure databricks.The connection needs to be routed through a HTTP proxy. I found parameters that can be configured for using the HTTP proxy:By pa...
Hi @Jonas Minning​ , actually I am also having the same issue and when i looked into the driver related documentation I found that the driver currently only supports SOCKS proxies and I believe this is the reason why we are getting this error. So, I ...
