Databricks is excited to announce the general availability of Databricks Workflows to you, our community. Databricks Workflows is the fully managed lakehouse orchestration service for all your teams to build reliable data, analytics, and AI workflow...
I am trying to run the Workflow Pipeline with smaple code shared in getting start.. and getting the below error :DataPlaneException: Failed to start the DLT service on cluster 0526-084319-7hucy1np. Please check the stack trace below or driver logs fo...
Hi everyone,I can't seem to find Alerts in the sidebar, also my data-explorer looks different from what I see in the videos. Do I need to upgrade my environment? Thanks,K
Hi group,After speaking with my rep, it appears that Databricks ALERTS is only for premium members even though that is not what is advertised on the site or in the documentation. This is unfortunate as data-quality is a concern for us and we don't fe...
Hi,I am using the logger to log some parameters in my code and I want to save the file under DBFS. But for some reason the file is not getting created under DBFS. If I clear the state of the notebook and check the DBFS dir then file is present. Pleas...
I have a scenario where one source is unstructered pdf files and another source is semi-structered JSON files. I get files from these two sources on a daily basis into an ADLS storage. What is the best way to load this into a medallion structure by s...
Hi all,I've recently checked my bucket size on AWS and saw that it's size doesn't make much sense. I decided to vacuum each delta table with 2 weeks of retention. That shrunk the data from 30TB to around 5TB, though I was wondering, shouldn't default...
Hi @Merca Ovnerud​ , We haven't heard from you since my last response, and I was checking to see if my suggestions helped you. Also, please don't forget to click the "Select as Best" button" whenever the information provided helps resolve your questi...
What are the best practices for the isolation of different environments in Databricks?I am trying to find out the best practice around Databricks environments creation like - dev, stage, and prod. Should it be:1. Single Databricks account with multip...
Hey @Abhijit Rai​ Does @Atanu Sarkar​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?Thank you!
Configured DBeaver to work with either databricks latest driver or simba. I can connect and see databases, schemas, tables and columns. However, when a select statement is executed 30-40 seconds go by before I get the following error message: SQL...
Has this issue been resolved? @aravhish solution did not help me. Any other options?I am experiencing the exact same issue with the same configuration on a Mac. Much help would be appreciated.
Hello all, I have been not using databricks CE for 4-5 months but today I've tried to create the default cluster, as always did, and I've got the error "Backend services unavailable". Is this a temporary problem or am I doing something wrong. Thanks ...
There's currently an outage of the community edition. Please follow this link for status:https://status.databricks.com/pages/incident/5cf02dde58a00904bda41926/62cd7677db464e053416a89c
My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endpoint via API/SDK
Hi @Harish S​ , We haven't heard from you on the last response from @Prabakar Ammeappin​ , me, and @Jose Gonzalez​ , and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can b...
We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?
Hi @Bhanu Patlolla​ ​, We haven’t heard from you on the last response from @Werner Stinckens​ and @Hubert Dudek​ and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be h...
Is there a way to compare a time stamp within on field/column for an individual ID? For example, if I have two records for an ID and the time stamps are within 5 min of each other....I just want to keep the latest. But, for example, if they were an h...
Hi @Cory Bullard​, We haven't heard from you on the last response from @Merca Ovnerud​, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwis...
I'm using databricks connect to talk to a cluster on Azure. When doing a count on a dataframe I sometimes get this error message. Once I've gotten it once I don't seem to be able to get rid of it even if I restart my dev environment. ----------------...
Hi @Johan Rex​ , We haven't heard from you on my last response, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with ...
