cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
iwan_aucamp
by • New Contributor III
  • 1591 Views
  • 2 replies
  • 2 kudos

Are there any python API bindings for the Databricks Account API?

All I could find in terms of API bindings for python is https://pypi.org/project/databricks-cli/, and this does not include the Account API and it is also not official.I will just use the OpenAPI spec, but just want to be sure I'm not doing unnecessa...

Data Engineering
Reply
  • 1591 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Iwan Aucamp​ : Yes, there are Python API bindings available for the Databricks Account API.For Databricks Account API with Python, please refer to the Databricks documentation: https://docs.databricks.com/dev-tools/api/latest/accounts.html#python-ap...

  • 2 kudos
1 More Replies
youssefmrini
by • Databricks Employee
  • 2024 Views
  • 4 replies
  • 0 kudos

Resolved! Where Can I find the latest Databricks releases regarding the features and updates ?

Data Engineering
Reply
  • 2024 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Youssef Mrini​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...

  • 0 kudos
3 More Replies
jairomonassa
by • New Contributor
  • 2358 Views
  • 4 replies
  • 2 kudos

Resolved! where can I find my certification id in order for apply in rewards from community ?

Data Engineering
Reply
  • 2358 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @jairo neder monassa moreira​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear...

  • 2 kudos
3 More Replies
Azure_databric1
by • New Contributor II
  • 1906 Views
  • 1 replies
  • 2 kudos

How to find the road distance between two cities? We can use Azure databricks and azure map.

We will be given an excel file, in which we can see column sender_city and destination_city. We have to find the distance between these two cities and the distance calculated should be written in a column total_distance. All these processes should be...

Data Engineering
Reply
  • 1906 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
sher
Valued Contributor II
  • 2 kudos

heywithout using latitude and longitude it is hard to find out but you can try some distance-based algorithm

  • 2 kudos
Oliver_Floyd
by • Contributor
  • 2542 Views
  • 4 replies
  • 6 kudos

Where to find documentation about : spark.databricks.driver.strace.enabled

Hello ,For a support request, Microsoft support ask me to add spark.databricks.driver.strace.enabled trueto my cluster configuration.MS was not able to send me a link to the documentation and I did not find it on the databricks website.Can someone he...

Data Engineering
Reply
  • 2542 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
Oliver_Floyd
Contributor
  • 6 kudos

Yes no problem. I have a python program, called "post ingestion", that run on a databricks job cluster during the night and consist of :inserting data to a deltalake tableexecuting an optimize command on that tableexecuting a vacuum command on that t...

  • 6 kudos
3 More Replies
StephanieAlba
by • Databricks Employee
  • 2278 Views
  • 1 replies
  • 1 kudos

Resolved! How to find my workspace id?

 My Solutions Architect is asking for my workspaceID. I do not know where to look for it. All I see is the user settings menu.

Data Engineering
Reply
  • 2278 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
PeteStern
Databricks Employee
  • 1 kudos

The workspaceID is usually in the URL of the workspace. For example:https://myworkspace.com/?o=12345The workspace id in this case is 12345. You can also just share the URL with your solutions architect.

  • 1 kudos
Sugumar_Sriniva
by • New Contributor III
  • 7604 Views
  • 11 replies
  • 5 kudos

Resolved! Data bricks cluster creation is failing while running the Cron job scheduling script through init script method from azure data bricks.

Dear connections,I'm unable to run a shell script which contains scheduling a Cron job through init script method on Azure Data bricks cluster nodes.Error from Azure Data bricks workspace:"databricks_error_message": "Cluster scoped init script dbfs:/...

Data Engineering
Reply
  • 7604 Views
  • 11 replies
  • 5 kudos
View Replies
Latest Reply
User16764241763
Honored Contributor
  • 5 kudos

Hello @Sugumar Srinivasan​  Could you please enable cluster log delivery and inspect the INIT script logs in the below path dbfs:/cluster-logs/<clusterId>/init_scripts path.https://docs.databricks.com/clusters/configure.html#cluster-log-delivery-1

  • 5 kudos
10 More Replies
brickster_2018
by • Databricks Employee
  • 3250 Views
  • 1 replies
  • 0 kudos

Resolved! I can't find my cluster

I had a cluster that I used in the past. I do not see the cluster any longer. I checked with the admin and my team and everyone confirmed that there no user deletion. 

Data Engineering
Reply
  • 3250 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

If the cluster is unsued for 30 days, Databricks removes the cluster. This is a general clean-up policy. It's possible to whitelist a cluster from this clean-up by Pinning the cluster. https://docs.databricks.com/clusters/clusters-manage.html#pin-a-c...

  • 0 kudos
User16826987838
by • Contributor
  • 1888 Views
  • 1 replies
  • 0 kudos

How do I find the users in workspaces

Looking to pull a list of all the users in their workspaces (including the ones who have never done anything), is there a way to do that? This for AWS

Data Engineering
Reply
  • 1888 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

You can use the SKIM APIs. Endpoint: https://docs.databricks.com/dev-tools/api/latest/scim/scim-users.html#get-users Or you can use the Workspace API. The workspace API does not have a direct list users command, but you can use the workspace API to l...

  • 0 kudos
Anonymous
by • Not applicable
  • 883 Views
  • 1 replies
  • 0 kudos

Where can I find a best practice guide on Delta Lake Structured Streaming with Amazon Kinesis?

Data Engineering
Reply
  • 883 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

Hi we have best practices around streaming with Kinesis and you can find them here .https://docs.databricks.com/spark/latest/structured-streaming/kinesis-best-practices.html

  • 0 kudos
User16860826802
by • New Contributor III
  • 6344 Views
  • 1 replies
  • 1 kudos

Resolved! Why does my cluster keeps disappearing?

My team and I were using a cluster for some days and it disappeared without any apparent reason. I recreate the cluster, but after some days it disappeared again. Do you know why my cluster disappeared? how to avoid that?

Data Engineering
Reply
  • 6344 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
User16860826802
New Contributor III
  • 1 kudos

A cluster is deleted after 30 days after a cluster is terminated. To keep an all-purpose cluster configuration even after a cluster has been terminated for more than 30 days, an administrator can pin the cluster. Up to 70 clusters can be pinned.To av...

  • 1 kudos
Labels