The recommendation before dropping a table is to do a DELETE then VACUUM RETENTION 0 (recommended in DEV).If you DROP the table without doing a DELETE|VACUUM, your table will be soft deleted with your entire data (permanently deletedin 30 days) and y...
The table property dataSkippingNumIndexedCols that gets statistics for a table starts from left to right. I am wondering what will happen to the statistics for both new and old records if we add a column in between using FIRST|AFTER identifier.
What is the status of bamboolib? I understand that it is public preview but I'm unable to find any support references. I am getting error below. I've tried installing in a notebook, on a cluster, creating a pandas dataframe and running bam, etc. ...
Hi Databricks community team,I have code as below"""df = spark.readStream \.format("kinesis") \.option("endpointUrl", endpoint_url) \.option("streamName", stream_name) \.option("initialPosition", "latest") \.option("consumerMode", "efo") \.option("ma...
Assuming you have a catalog "my_catalog" and a schema "my_schema", the following code is not working : full_table_location = "`my_catalog`.`my_schema`.`my_table_hourl`" spark.conf.set("fullTableName", full_table_location) spark.sql("""SELECT * FRO...
hello,i'm new to Databricks (community edition account) and encountered a problem just now.When creating a new cluster (default 10.4 LTS) it fails with the following error: Backend service unavailable.I've tried a different runtime > same issue.I've ...
Hey mbvb_py,I'm sorry to hear you're facing this "Backend service unavailable" issue with Databricks. I've encountered similar problems in the past, and it can be frustrating. Don't worry; you're not alone in this!From my experience, this error can o...
Hi All Enthusiasts !As per my understanding when a user submits an application in spark cluster it specifies how much memory, executors etc. it would need . But in Data bricks notebooks we never specify that anywhere. If we have submitted the noteboo...
@DBEnthusiast great question! Today, with Job Clusters, you have to specify this. As @btafur note, you do this by setting CPU, memory etc. We are in early preview of Serverless Job Clusters where you no longer specify this configuration, instead Data...
While scheduling the Databricks job using continuous mode - what will happen if the job is configured to run with Job cluster.At the end of each run will the cluster be terminated and re-created again for the next run? The official documentation is n...
Hello @youssefmrini So how is the DBU calculated? As the cluster is reused, the DBU should be calculated per hour on all the jobs run in an hour correct? Or will it be calculated based on each run?I would like to know the cost calculation when runnin...
Streamline Data Modeling Normalization with Databricks Delta Live Tables in Just a Few Steps:- Use the "Apply changes" function to populate tables with slowly changing dimensions using auto-increment IDs.- Register SQL mapping functions to associate ...
Thank you for sharing this @Hubert-Dudek !!!
Hi - I have tried my level best to go through both elasticsearch documentation as well as Databricks documentation to get an answer for my question - is it possible to connect to AWS elasticsearch of a different AWS account from Databricks? I did no...
When importing a .csv file with leading and/or trailing empty spaces around the separators, the output results in strings that appear to be trimmed on the output table or when using .display() but are not actually trimmed.It is possible to identify t...
Hi,There's a way to make work the Simba ODBC Driver for M1 Macbook Pros?I find myself able to run on an old intel version of Macbook easily, but now every time I even test the connection with the iODBC Manager fails.Definitely, the issue is around no...
Things seem to be mostly working for me now. I've added a bit more detail on my connection steps and process in case it's helpful for anyone on Stack Overflow: https://stackoverflow.com/questions/76407426/connecting-rstudio-desktop-to-databricks-comm...
I am hoping someone can help me remove the WHERE 1=0 that is constantly getting added onto the end of my Query (see below). Please let me know if I can provide more info here.This is running a notebook, in Azure Databricks against a cluster that has...
Trying to determine a root cause of UDFException that occurs when returning a variable length ArrayType. If I hardcode the data returned from the UDF to a fixed length, say 19, the error does not occur. Setup codesplit_runs_UDF = udf(split_runs_udf, ...
After further investigation, It reproduces slightly differently on single user mode.Single user mode: runs foreverShared: gives the above messageI've determined that there was a corner case in the dataset which lead to UDF never returning. I am am as...
I need to retrieve the event logs of deleted All Purpose clusters of a certain workspace.databricks list API ({workspace_url}/api/2.0/clusters/list) provides me with the list of all active/terminated clusters but not the clusters that are deleted. I ...
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
