- 41075 Views
- 13 replies
- 3 kudos
Resolved! displayHTML can't seem to be used from Python code, only hand typed into a cell???
Trying to use displayHTML from w/in a Python module gets a Python exception:NameError: name 'displayHTML' is not definedand I've found no way around this. It seems to be something at the UI layer or something, not a Python function that can be refere...
- 41075 Views
- 13 replies
- 3 kudos
- 3 kudos
Holy Guacamole Batman! It works finally!!!! Wow, thanks @ptweir That's awesome! I can go back and update my doc (and code, to just use databricks the same, now, and Jupyter!) and it'll work by default. It's great they fixed it, shame they never told ...
- 3 kudos
- 7513 Views
- 2 replies
- 0 kudos
Resolved! Invalid configuration value detected for fs.azure.account.key only when trying to save RDD
Hello,We have encountered a weird issue in our (old) set-up that looks like a bug in the Unity Catalog. The storage account which we are trying to persist is configured via External Volumes.We have a pipeline that gets XML data and stores it in an RD...
- 7513 Views
- 2 replies
- 0 kudos
- 0 kudos
I will post here what worked resolving this error for us, in case someone else in the future encounters this.It turns out that this error appears in this case, when we were using the below command while the directory 'staging2' already exists. To avo...
- 0 kudos
- 14473 Views
- 3 replies
- 1 kudos
Resolved! How to kill the execution of a notebook on specyfic cell?
Let's say I want to check if a condition is false then stop the execution of the rest of the script. I tried with two approaches:1) raising exceptionif not data_input_cols.issubset(data.columns): raise Exception("Missing column or column's name mis...
- 14473 Views
- 3 replies
- 1 kudos
- 1 kudos
In Jupyter notebooks or similar environments, you can stop the execution of a notebook at a specific cell by raising an exception. However, you need to handle the exception properly to ensure the execution stops. The issue you're encountering could b...
- 1 kudos
- 10795 Views
- 9 replies
- 2 kudos
Resolved! How to version your workflows/jobs
WE would like to version control workflows/jobs over git, not the underlying notebooks but the job logic itselfis it possible?
- 10795 Views
- 9 replies
- 2 kudos
- 5706 Views
- 1 replies
- 0 kudos
Move Files from S3 to Local File System with Unity Catalog Enabled
Dear Databricks Community Experts,I am working on databricks on AWS with unity catalog.One usecase for me is to uncompress files with many extensions there on S3 Bucket.Below is my strategy:-Move files from S3 to Local file system (where spark driver...
- 5706 Views
- 1 replies
- 0 kudos
- 0 kudos
did you find a solution to this?
- 0 kudos
- 5422 Views
- 3 replies
- 0 kudos
Version control jobs
How do engineering teams out there version control their jobs? If there is a production issue, can I revert to an older version of the job?
- 5422 Views
- 3 replies
- 0 kudos
- 0 kudos
You can use version controlled source code for you databricks job and each time you need to rollback to older version of your job you need just to move to older version code. For version controlled source code you have multiple choises:- Use a noteb...
- 0 kudos
- 2989 Views
- 2 replies
- 2 kudos
Stream-stream window join after time window aggregation not working in 13.1
Hey,I'm trying to perform Time window aggregation in two different streams followed by stream-stream window join described here. I'm running Databricks Runtime 13.1, exactly as advised.However, when I'm reproducing the following code:clicksWindow = c...
- 2989 Views
- 2 replies
- 2 kudos
- 2 kudos
Hey,I'm currently facing the same problem, so I would to know if you've made any progress in resolving this issue.
- 2 kudos
- 5054 Views
- 1 replies
- 0 kudos
Combining DLT and workflow - MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED
Hello everyone !I currently have a DLT pipeline that loads into several Delta LIVE tables (both streaming and materialized view).The end table of my DLT pipeline is a materialized view called "silver.my_view".In a later step I need to join/union/merg...
- 5054 Views
- 1 replies
- 0 kudos
- 10897 Views
- 2 replies
- 0 kudos
Divide a dataframe into multiple smaller dataframes based on values in multiple columns in Scala
I have to divide a dataframe into multiple smaller dataframes based on values in columns like - gender and state , the end goal is to pick up random samples from each dataframeI am trying to implement a sample as explained below, I am quite new to th...
- 10897 Views
- 2 replies
- 0 kudos
- 0 kudos
@raela I also have similar usecase. I am writing data to different databricks tables based on colum value.But I am getting insufficient disk space error and driver is getting killed. I am suspecting df.select(colName).distinct().collect()step is taki...
- 0 kudos
- 8199 Views
- 1 replies
- 2 kudos
IDENTITY columns generating every other number when merging
Hi,I'm doing merge to my Delta Table which has IDENTITY column:Id BIGINT GENERATED ALWAYS AS IDENTITYInserted data has in the id column every other number, like this:Is this expected behavior? Is there any workaround to make number increasing by 1?
- 8199 Views
- 1 replies
- 2 kudos
- 2 kudos
Were you able to resolve it? I'm facing the same issue.
- 2 kudos
- 5583 Views
- 0 replies
- 0 kudos
Merge delta tables with data more than 200 million
HI Everyone,Im trying to merge two delta tables who have data more than 200 million in each of them. These tables are properly optimized. But upon running the job, the job is taking a long time to execute and the memory spills are huger (1TB-3TB) rec...
- 5583 Views
- 0 replies
- 0 kudos
- 1473 Views
- 0 replies
- 0 kudos
Issue with MERGE INTO for first batch
I have source data with multiple rows and columns, 1 of column is city. I want to get unique city into other table by stream data from source table. So I trying to use merge into and foreachBatch with my merge function. My merge condition is : On so...
- 1473 Views
- 0 replies
- 0 kudos
- 1688 Views
- 0 replies
- 0 kudos
cursor type\loop question
Hello:In my Hive Metastore, I have 35 tables in database that I want to export in excel. I need help on query that can loop one table at a time export one table to excel.Any help is appreciated.Thanking in advance for your kind help.
- 1688 Views
- 0 replies
- 0 kudos
- 10037 Views
- 1 replies
- 0 kudos
Unable to start Cluster in Databricks because of `BOOTSTRAP_TIMEOUT`
Unable to start the Cluster in AWS-hosted Databricks because of the below reason{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-0634ee9c2d420edc8), status: INSTANCE_INITIALIZIN...
- 10037 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Sahha: Thanks for contacting Databricks Support. This is the common type of error, which indicates that the bootstrap failed due to a misconfigured data plane network. Databricks requested EC2 instances for a new cluster, but encountered a long ...
- 0 kudos
- 1572 Views
- 0 replies
- 0 kudos
Geoparquet support with Use Photon Acceleration enabled
Hi Experts,Recently our team noticed that when we are using Aparch Sedona to create the parquet file with Geoparquet format, the geo metedata was not created inside the parquet file. But if we turn off the Photon setting, everything was working as ex...
- 1572 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 518 | |
| 349 | |
| 287 |
