- 6612 Views
- 2 replies
- 2 kudos
Execute Pyspark cells concurrently
Hi Team,Hi Team,Is it feasible to run pyspark cells concurrently in databricks notebooks? If so, kindly provide instructions on how to accomplish this. We aim to execute the intermediate steps simultaneously.The given scenario entails the simultaneou...
- 6612 Views
- 2 replies
- 2 kudos
- 2 kudos
Databricks also supports executing SQL cells in parallel. While a command is running and your notebook is attached to an interactive cluster, you can run a SQL cell simultaneously with the current command. The SQL cell is executed in a new, parallel ...
- 2 kudos
- 4313 Views
- 1 replies
- 0 kudos
Data transfer from AWS/Databricks to GEO repository via FTP
Does anyone have a Python script that runs in Databricks to transfer RNAseq data stored in AWS bucket to a public repository (GEO)? All my attempts failed, it looks like the connection between dbx and geo isn't working as expected.
- 4313 Views
- 1 replies
- 0 kudos
- 0 kudos
What is the exact error that you face? We can debug from there. I see there are some steps shared on GEO submissions https://www.ncbi.nlm.nih.gov/geo/info/submissionftp.html
- 0 kudos
- 5117 Views
- 1 replies
- 0 kudos
Bootstrap cluster timeout for job pipeline - databricks bug?
From time to time we have these erors in scheduled PROD runs. It happens when job starts and tries to create one time cluster. It happens 1 time from 10-20 runs and we are not able to identify the root cause, as all network connectivity is fine, some...
- 5117 Views
- 1 replies
- 0 kudos
- 0 kudos
The error message "BOOTSTRAP_TIMEOUT (SERVICE_FAULT)" indicates that the cluster was terminated because it took too long to initialize. This can happen due to various reasons, including network connectivity issues between the data plane and the contr...
- 0 kudos
- 1218 Views
- 1 replies
- 2 kudos
Resolved! Retrive inforrmation about table clustering from information_schema
Hi guys,I wonder if and when it will be possible to extract from information_schema how the table is clustered. I know that analogous information can be obtained when a table is partitioned using this query: SELECT * FROM cdl_dev.information_schema.c...
- 1218 Views
- 1 replies
- 2 kudos
- 2 kudos
Iit is not possible to extract clustering information of a table directly from the information_schema . The information_schema.columns table can provide details about partitioning, but similar information for clustering is not available through the i...
- 2 kudos
- 5176 Views
- 1 replies
- 0 kudos
Maven libs often failed during installation
Dear Community,I have a Databricks compute where I added 2 Maven libs using a custom repository from Nexus (because of a company policy, Databricks cannot communicate with the public internet, so I must use a private Nexus repo using a firewall). Sin...
- 5176 Views
- 1 replies
- 0 kudos
- 0 kudos
@Paxi Not sure if you are still looking for the solution. But we have Databricks Git server proxy that you can configure which enables you to proxy Git commands from Databricks Git folders to your on-premises Git repositories https://docs.databricks....
- 0 kudos
- 1998 Views
- 2 replies
- 1 kudos
Exam got suspended in the middle
My Databricks Certified Data Engineer Associate exam was suspended on 31st Jan'2025 and is currently in a "SUSPENDED" state.I remained in front of the camera throughout the exam, and suddenly an alert appeared. The support person asked me to show th...
- 1998 Views
- 2 replies
- 1 kudos
- 1 kudos
@Cert-TeamOPS @Cert-Team ,Thanks for your quick response. Please find the latest open request no. - #00610129
- 1 kudos
- 9319 Views
- 4 replies
- 2 kudos
Resolved! Databricks Spark Vs Spark on Yarn
I am moving my Spark workloads from EMR/on-premise Spark cluster to Databricks. I understand Databricks Spark is different from Yarn. How is the Databricks architecture different from yarn?
- 9319 Views
- 4 replies
- 2 kudos
- 2 kudos
What about the disadvantages?How can I separate multiple jobs running on the same cluster cleanly in the logs and same in the spark-ui?
- 2 kudos
- 19837 Views
- 5 replies
- 3 kudos
Resolved! Interoperability Spark ↔ Pandas: can't convert Spark dataframe to Pandas dataframe via df.toPandas() when it contains datetime value in distant future
Hi,I have multiple datasets in my data lake that feature valid_from and valid_to columns indicating validity of rows.If a row is valid currently, this is indicated by valid_to=9999-12-31 00:00:00.Example:Loading this into a Spark dataframe works fine...
- 19837 Views
- 5 replies
- 3 kudos
- 3 kudos
Be aware, that in Databricks 15.2 LTS this behavior is broken.I cannot find the code, but most likely related to the following option:https://github.com/apache/spark/commit/c1c710e7da75b989f4d14e84e85f336bc10920e0#diff-f9ddcc6cba651c6ebfd34e29ef049c3...
- 3 kudos
- 3754 Views
- 1 replies
- 3 kudos
How to Export lineage data directly from unity catalog without using system tables
I have been trying to check if there is any direct way to export lineage hierarchy data in data bricks.I have tried to build a workaround solution by accessing system tables as per this link:Monitor usage with system tables - Azure Databricks | Micro...
- 3754 Views
- 1 replies
- 3 kudos
- 3 kudos
I have a similar use case. The Databricks Lineage Graph is nice but only zooms out enough for the most basic lineages. We have lineages/data flows with hundreds of tables. I'd like more flexibility on showing the entire flow in one screen and expo...
- 3 kudos
- 1038 Views
- 1 replies
- 1 kudos
Data ingestion issue with THAI data
I have a use case where my file has data in Thai characters. The source location is azure blob storage, here files are stored in text format. I am using the following code to read the file, but when I am downloading the data from catalog it encloses ...
- 1038 Views
- 1 replies
- 1 kudos
- 1 kudos
Do the quotes exist in original data?
- 1 kudos
- 4174 Views
- 6 replies
- 1 kudos
Too many small files from updates
Hi ,I am updating some data into a delta table , each time I only need to update one row due to which after every update statement it is creating new file, How do I tackle this issue , it doesn't make sense to run optimize command after every upda...
- 4174 Views
- 6 replies
- 1 kudos
- 1 kudos
If you performing 100s of update operations on the delta table, you can opt to run an optimize operation after a batch of 100 updates. There should be no significant performance issue up to 100 such updates
- 1 kudos
- 4742 Views
- 1 replies
- 1 kudos
Resolved! How to deploy an asset bundle job that triggers another one
Hello everyone.Using DAB, is there a dynamic value reference or something equivalent to get a job_id to be used inside the YAML definition of another Databricks job? I'd like to trigger that job from another one, but if I'm using a CI/CD pipeline to ...
- 4742 Views
- 1 replies
- 1 kudos
- 1 kudos
resources: jobs: my-first-job: name: my-first-job tasks: - task_key: my-first-job-task new_cluster: spark_version: "13.3.x-scala2.12" node_type_id: "i3.xlarge" num_workers: 2 ...
- 1 kudos
- 6188 Views
- 2 replies
- 1 kudos
Resolved! How to grant all tables in schema except 1
Hi Guys, I am trying to grant all tables in a schema to a user group in databricks. The only catch is that there is one table I do not want granted. I currently am granting schema access to the group so the benefit is that as tables are add in the fu...
- 6188 Views
- 2 replies
- 1 kudos
- 1 kudos
What you are facing is because of inheritance. https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/upgrade-privilege-model.html I would say this is by design, but please feel free to suggest it as an idea here - https://do...
- 1 kudos
- 1441 Views
- 2 replies
- 0 kudos
JDBC Error Trying to Connect erwin Data Modeler to Databricks
I am trying to connect erWin Data Modeler to Databricks to reverse engineer a physical data model. I am trying to connect manually per erWin and Databricks instructions, but I am getting the following error[Databricks][DatabricksJDBCDriver][500593] C...
- 1441 Views
- 2 replies
- 0 kudos
- 0 kudos
I hope you referred https://docs.databricks.com/en/partners/data-governance/erwin.html It is also possible, it can be a library issue, hope you are using the Databricks JDBC driver.
- 0 kudos
- 886 Views
- 1 replies
- 1 kudos
- 886 Views
- 1 replies
- 1 kudos
- 1 kudos
Congratulations, @AlexCancioBedon! This is a great milestone that showcases your expertise in Data engineering with Databricks. We’d love to have you share your insights with the community, whether by sharing best practices or helping others. Keep up...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
