- 11521 Views
- 4 replies
- 3 kudos
function does not exist in JVM ERROR
Hello guys, I'm building a python package that return 1 row from DF at a time inside data bricks environment.To improve the performance of this package i used multiprocessing library in python, I have background process that his whole purpose is to p...
- 11521 Views
- 4 replies
- 3 kudos
- 3680 Views
- 2 replies
- 1 kudos
Delta Tables copying
Hello, I’m trying to copy a table with all it’s versions to unity catalog, I know I can use deep cloning but I want the table with the full history, is that possible?
- 3680 Views
- 2 replies
- 1 kudos
- 1 kudos
To copy history, you would have to copy files along with the delta log folder and then create a delta table on that location
- 1 kudos
- 1807 Views
- 1 replies
- 0 kudos
Resolved! Welcome
Welcome!
- 1807 Views
- 1 replies
- 0 kudos
- 0 kudos
Welcome!
- 0 kudos
- 5850 Views
- 2 replies
- 0 kudos
databricks view question!
I found this phrase in the document "A view stores the text for a query type again one or more data sources or tables in the metastore."Does "view" in databricks store data in a physical location?
- 5850 Views
- 2 replies
- 0 kudos
- 0 kudos
CREATE VIEW | Databricks on AWS - Constructs a virtual table that has no physical data based on the result-set of a SQL query.
- 0 kudos
- 1116 Views
- 0 replies
- 0 kudos
Set up RBACs with hive catalog
Hello, we are not on unity catalog yet due to limitations on multi cloud implementation of UC. We still want to implement Role Based Acess Control with hive metastore. We are using DBR 11.3. Any pointers will be helpful
- 1116 Views
- 0 replies
- 0 kudos
- 959 Views
- 0 replies
- 0 kudos
Python syntax check
CI/CD
- 959 Views
- 0 replies
- 0 kudos
- 3492 Views
- 3 replies
- 1 kudos
Could not launch jobs due to node_type_id (instance) unavailability
I am running hourly job on a cluster using p3.2xlarge GPU instance, but sometimes cluster couldn't start due to instance unavailability. I wander is there is any fallback mechanism to, for example, try a different instance type if one is not availabl...
- 3492 Views
- 3 replies
- 1 kudos
- 1 kudos
(AWS only) For anyone experiencing capacity related cluster launch failures on non-GPU instance types, AWS Fleet instance types are now GA and available for clusters and instance pools. They help improve chance of successful cluster launch by allowi...
- 1 kudos
- 2239 Views
- 1 replies
- 0 kudos
Instance type in Photon
Can Photon run on all instance/VM types?
- 2239 Views
- 1 replies
- 0 kudos
- 0 kudos
No, Photon is only supported on a limited set of instance types where it's been benchmarked and tested by Databricks to have optimal performance.
- 0 kudos
- 3517 Views
- 3 replies
- 1 kudos
Support for multiple EC2 instance types in a worker pool
As per this thread Databricks now integrates with EC2 CreateFleet API that allows customers to create Databricks pools and get EC2 instances from multiple AZs and multiple instance families & sizes. However, in the Databricks UI you can not select mo...
- 3517 Views
- 3 replies
- 1 kudos
- 1 kudos
Fleet instances on Databricks is now GA and available in all AWS workspaces - you can find more details here: https://docs.databricks.com/compute/aws-fleet-instances.html
- 1 kudos
- 4108 Views
- 2 replies
- 1 kudos
Ingesting more than 7 million rows into a SQL Server Table
Hi All, I hope you're super well. I need your recommendations and solution for my problem.I am using a Databricks instance DS12_v2 which has 28GB RAM and 4 cores. I am ingesting 7.2 million rows into a SQL Server table and it is taking 57 min - 1 hou...
- 4108 Views
- 2 replies
- 1 kudos
- 1 kudos
You can try to use BULK INSERT.https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver16Also using Data Factory instead of Databricks for the copy can be helpful.
- 1 kudos
- 5568 Views
- 4 replies
- 7 kudos
Databricks Academy content for Azure Databricks Customers
Hi! We've recently provisioned an Azure Databricks workspace and started building our pipelines. Do we qualify as Databricks 'customers' who have free access to all self-paced content on Databricks Academy? If so, how do we access it? We don't have a...
- 5568 Views
- 4 replies
- 7 kudos
- 7 kudos
They changed the registration process and added 'Additional Fields' section, where you can provide your company email address, that you use in Azure Databricks. This worked automatically for me and I can access the self-paced trainings for free now.
- 7 kudos
- 12657 Views
- 2 replies
- 5 kudos
How to restart the Spark session within the notebook without reattaching the notebook?
Hi All,I want to run an ETL pipeline in a sequential way in my DB notebook. If I run it without resetting the Spark session or restarting the cluster I am getting a data frame key error. I think this might be because of the Spark cache because If I r...
- 12657 Views
- 2 replies
- 5 kudos
- 5 kudos
Is there a solution to the above problem? I also would like to restart SparkSession to free my cluster's resources, but when callingspark.stop()the notebook automatically detach and the following error occurs:The spark context has stopped and the dri...
- 5 kudos
- 2028 Views
- 0 replies
- 0 kudos
Optimal Azure VM type for EventHub streaming
Hello,our spark jobs stream messages from Event Hub then transform it and finally the messages are peristed in storage. We plan to exercise cluster configurations for these jobs in order to find the optimal and procure Azure reservations. Furtemore, ...
- 2028 Views
- 0 replies
- 0 kudos
- 1509 Views
- 0 replies
- 1 kudos
Missed Associate Developer for Apache Spark 3.0 - Python Due to Power outage
Dear Databricks Certification Team,Unfortunately, I was unable to take the exam as scheduled due to an unforeseen power breakdown in my area. The power outage occurred just before the exam, rendering me unable to access the necessary resources to com...
- 1509 Views
- 0 replies
- 1 kudos
- 6816 Views
- 4 replies
- 0 kudos
Resolved! If 2 users run the same notebook on the same cluster, will they share the same Spark session?
Databricks docs here:https://docs.databricks.com/notebooks/notebook-isolation.htmlstate that "Every notebook attached to a cluster has a pre-defined variable named spark that represents a SparkSession." What if 2 users run the same notebook on the sa...
- 6816 Views
- 4 replies
- 0 kudos
- 0 kudos
The spark session is isolated at the notebook level and is not isolated at the user level. So, two users accessing the same notebook will be using the same spark session
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
