cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

paras11
by New Contributor III
  • 917 Views
  • 3 replies
  • 1 kudos

Databricks data engineer associate exam Suspended

Hi Team,I recently had a disappointing experience while attempting my first Data bricks certification exam. During the exam, I was abruptly directed to Proctor Support. The proctor asked me to show my desk and the room I was in. I complied by showing...

Community Platform Discussions
@Cert-Bricks@Cert-Team@Cert-TeamOPS @Kaniz_Fatma
  • 917 Views
  • 3 replies
  • 1 kudos
Latest Reply
paras11
New Contributor III
  • 1 kudos

 @Kaniz_Fatma @Cert-Team @Cert-Bricks Requesting you to please look into this and update me since it has not been resolved yet and I am not able to reschedule my exam. 

  • 1 kudos
2 More Replies
TinaN
by New Contributor III
  • 1868 Views
  • 3 replies
  • 3 kudos

Resolved! Extracting 'time' from a 'timestamp' datatype in Databricks

We are loading a data source to Databricks that contains columns with 'Time' datatype.  Databricks converts this to 'Timestamp', so I am researching for a way to extract time only. This is what I came up with, but the result isn't quite right.  Is th...

  • 1868 Views
  • 3 replies
  • 3 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 3 kudos

Hi @TinaN ,I check it in the evening, but try below: SELECT date_format(timestamp_column, 'HH:mm:ss') AS time_partFROM your_table

  • 3 kudos
2 More Replies
Prasad_Koneru
by New Contributor III
  • 1004 Views
  • 4 replies
  • 0 kudos

Deployment of tables and views in unity catalog and Repo structuring for catalogs objects.

we want to create the CI/CD Pipeline for deploying Unity catalog objects inorder to enhance the deployment ability.but How to maintain a repository of Tables/views/ or any other objects created in the catalogs and schema.Is this possible to do just l...

  • 1004 Views
  • 4 replies
  • 0 kudos
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 0 kudos

Hi @Prasad_Koneru , Thank you for reaching out to our community! We're here to help you.  To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your f...

  • 0 kudos
3 More Replies
icyapple
by New Contributor
  • 694 Views
  • 3 replies
  • 0 kudos

hive_metastore schema access control

We are trying to control access to schemas under hive_metastore, only allowing certain users to access the tables under a schema (via SQL, Pyspark, and Python...), we have follow steps in a testing schema:1. Enable workspace table access control2. Ru...

  • 694 Views
  • 3 replies
  • 0 kudos
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 0 kudos

Hi @icyapple , Thank you for reaching out to our community! We're here to help you.  To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedba...

  • 0 kudos
2 More Replies
SHASHANK2
by New Contributor III
  • 847 Views
  • 3 replies
  • 1 kudos

cluster termination

Hello All,when I am creating all purpose cluster I am getting an idle time of 3 days, my cluster is terminating after 3days, I want to make my cluster terminate in 60 min of idle time, i want to do it globally so that in future any cluster created by...

  • 847 Views
  • 3 replies
  • 1 kudos
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 1 kudos

Hi @SHASHANK2 , Thank you for reaching out to our community! We're here to help you.  To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedb...

  • 1 kudos
2 More Replies
VPara
by New Contributor
  • 2206 Views
  • 0 replies
  • 0 kudos

DBRX: Serving Endpoint Not Ready: Failing to Load the Model within 300 seconds

Hi Team,I created a serving endpoint using the mlflow.deployments -> get_deploy_client -> create_endpoint. The endpoint was created successfully but it was not going to the ready state as the update failed with the error message being "Exceeded maxim...

  • 2206 Views
  • 0 replies
  • 0 kudos
Yashodip
by New Contributor II
  • 588 Views
  • 1 replies
  • 0 kudos

My Databricks Professional Data Engineer exam has suspended , Need help Urgently (17/07/2024)

Hello Team, I encountered Pathetic experience while attempting my Professional Data Engineer DataBricks certification. Abruptly, Proctor asked me to show my desk, after 30 mins of exam showing he/she asked multiple times.. wasted my time and then sus...

  • 588 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello Team, I encountered Pathetic experience while attempting my Professional Data Engineer DataBricks certification. Abruptly, Proctor asked me to show my desk, after 30 mins of exam showing he/she asked multiple times.. wasted my time and then sus...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
elsirya
by New Contributor III
  • 718 Views
  • 2 replies
  • 2 kudos

Resolved! unit testing

Currently I am creating unit tests for our ETL scripts although the test is not able to recognize sc (SparkContext).Is there a way to mock SparkContext for a unit test? Code being tested: df = spark.read.json(sc.parallelize([data])) Error message rec...

  • 718 Views
  • 2 replies
  • 2 kudos
Latest Reply
elsirya
New Contributor III
  • 2 kudos

Was able to get this to work.What I had to do was instantiate the "sc" variable in the PySpark notebook.PySpark code:"sc = spark.SparkContext"Then in the PyTest script we add a "@patch()" statement with the "sc" variable and create a "mock_sc" variab...

  • 2 kudos
1 More Replies
aparna_526
by New Contributor
  • 371 Views
  • 0 replies
  • 0 kudos

Databricks exam got suspended

My Databricks Certified data engineer associate exam got suspended on 18 July 2024 .I was continuously in front of the camera and an alert appeared and then my exam resumed. Then later a support person told me that your exam got suspended. I Don't kn...

  • 371 Views
  • 0 replies
  • 0 kudos
aditi06
by New Contributor III
  • 415 Views
  • 1 replies
  • 1 kudos

Issue while launching the data engineer exam

I started my exam but it said due to a technical issue it had been suspended though I checked all prerequisites and system checks.I have already raised the ticket please resolve this issue as early as possible.Ticket no.:  #00504757#reschedule #issue...

  • 415 Views
  • 1 replies
  • 1 kudos
Latest Reply
aditi06
New Contributor III
  • 1 kudos

@Retired_mod , Please resolve this issue as early as possible. I have already raised the ticket

  • 1 kudos
Simon_T
by New Contributor III
  • 1675 Views
  • 2 replies
  • 1 kudos

Resolved! Databricks Bundle Error

I am running this command: databricks bundle deploy --profile DAVE2_Dev --debug And I am getting this error: 10:13:28 DEBUG open dir C:\Users\teffs.THEAA\OneDrive - AA Ltd\Databricks\my_project\dist: open C:\Users\teffs.THEAA\OneDrive - AA Ltd\Databr...

  • 1675 Views
  • 2 replies
  • 1 kudos
Latest Reply
Simon_T
New Contributor III
  • 1 kudos

So I found a link to a page that said that the databricks bundle command is expecting python3.exe instead of python.exe. So I took a copy of python.exe and renamed it to python3.exe and that seems to work. Thanks for investigating though.

  • 1 kudos
1 More Replies
Puneet096
by New Contributor II
  • 793 Views
  • 3 replies
  • 1 kudos

how to implement delta load when table only has primary columns

I have a table where there are two columns and both are primary key, I want to do delta load when taking data from source to target. Any idea how to implement this? 

  • 793 Views
  • 3 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

But that shouldn't be a problem. In merge condition you check both keys as in example above. If combination of two keysb already exists in the table then do nothing. If there is new combination of key1 and key2 just insert it into target table.It's t...

  • 1 kudos
2 More Replies
InquisitiveGeek
by New Contributor II
  • 725 Views
  • 3 replies
  • 0 kudos

how can I store my cell output as a text file in my local drive?

I want to store the output of my cell as a text file in my local hard drive.I'm getting the json output and I need that json in my local drive as a text file. 

  • 725 Views
  • 3 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @InquisitiveGeek ,You can do this following below approach: https://learn.microsoft.com/en-us/azure/databricks/notebooks/notebook-outputs#download-results

  • 0 kudos
2 More Replies
himanmon
by New Contributor III
  • 590 Views
  • 2 replies
  • 1 kudos

Can I move a single file larger than 100GB using dbtuils fs?

Hello. I have a file over 100GB. Sometimes this is on the cluster's local path, and sometimes it's on the volume.And I want to send this to another path on the volume, or to the s3 bucket. dbutils.fs.cp('file:///tmp/test.txt', '/Volumes/catalog/schem...

himanmon_0-1721181332995.png himanmon_1-1721185042042.png
  • 590 Views
  • 2 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Hi @himanmon ,This is caused because of S3 limit on segment count. The part files can be numbered only from 1 to 10000After Setting spark.hadoop.fs.s3a.multipart.size to 104857600. , did you RESTART the cluster? Because it'll only work when the clust...

  • 1 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors