- 4271 Views
- 1 replies
- 0 kudos
How can I create from scratch a brand new Dataframe with Null values using spark.createDataFrame()?
from pyspark.sql.types import * schema = StructType([ StructField("c1", IntegerType(), True), StructField("c2", StringType(), True), StructField("c3", StringType(), True)]) df = spark.createDataFrame([(1, "2", None), (3, "4", None)], schema)
- 4271 Views
- 1 replies
- 0 kudos
- 0 kudos
df = spark.createDataFrame(sc.emptyRDD(), schema)Can you try this?
- 0 kudos
- 4415 Views
- 1 replies
- 1 kudos
Resolved! cluster start Issues
Some of the Jobs are failing in prod with below error message:Can you please check and let us know the reason for this? These are running under pool cluster.Run result unavailable: job failed with error messageUnexpected failure while waiting for the...
- 4415 Views
- 1 replies
- 1 kudos
- 1 kudos
@Kunal Gaurav​ , This status code only occurs in one of two conditions:We’re able to request the instances for the cluster but can’t bootstrap them in time We setup the containers on each instance, but can’t start the containers in timethis is an edg...
- 1 kudos
- 2042 Views
- 1 replies
- 0 kudos
- 2042 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you elaborate mode what you mean by "Encoder" (is it a serializing mechanism), what are the custom data objects? pyspark does support complex and binary formats as long as you can write your own serializer/deserializer.
- 0 kudos
- 2115 Views
- 2 replies
- 1 kudos
Prevent file downloads from /files/ URL
I would like to prevent file download via /files/ URL. For example: https://customer.databricks.com/files/some-file-in-the-filestore.txtIs there a way to do this?
- 2115 Views
- 2 replies
- 1 kudos
- 1 kudos
Unfortunately this is not possible from the platform.You can however use an external Web Application Firewall (e.g. Akmai) to filter all web traffic to your workspaces. This can block both Web access to download root bucket data.
- 1 kudos
- 3738 Views
- 1 replies
- 1 kudos
Resolved! Are there any limitations on my broadcast joins?
I would like to know if there are any broadcast joins limitations.
- 3738 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, there are a couple limitation. Please find below the details:> It will not perform broadcast join if the table has 512 million or more rows > It will not perform broadcast join if the table is larger than 8GB
- 1 kudos
- 3820 Views
- 1 replies
- 1 kudos
Resolved! Getting broadcast join errors
I would like to know how do disable broadcast join in my job to avoid this error message. Is there a Spark configuration?
- 3820 Views
- 1 replies
- 1 kudos
- 1 kudos
You can disable broadcast join by adding the following Spark configuration to you notebook:spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)In addition, you can also add this configuration to your cluster:spark.sql.autoBroadcastJoinThreshold...
- 1 kudos
- 2819 Views
- 1 replies
- 1 kudos
- 2819 Views
- 1 replies
- 1 kudos
- 1 kudos
Databricks on AWS uses a custom cluster manager and not Kubernetes. EKS is not supported yet
- 1 kudos
- 3542 Views
- 1 replies
- 0 kudos
Resolved! how to troubleshot Python version mismatch error in DBconnect?
Im getting some weird messages when trying to run my Dbconnect. I would like to know if there is a troubleshooting guide to solve Python version mismatch errors.
- 3542 Views
- 1 replies
- 0 kudos
- 0 kudos
We have a troubleshooting section in our docs that could help you to solve this issue. Please check the docs here https://docs.databricks.com/dev-tools/databricks-connect.html#python-version-mismatch
- 0 kudos
- 2821 Views
- 1 replies
- 0 kudos
Resolved! can I use Dbconnect for my structured streaming jobs?
I would like to know if I can use Dbconnect to run all my structured streaming jobs.
- 2821 Views
- 1 replies
- 0 kudos
- 0 kudos
Unfortunately, no. You cannot use Dbconnect for your streaming jobs. This is one of Dbconnect's limitations. For more details please check the docs: https://docs.databricks.com/dev-tools/databricks-connect.html#limitations
- 0 kudos
- 3709 Views
- 1 replies
- 0 kudos
Resolved! How often should I run OPTIMIZE on my Delta Tables?
I know it's important to periodically run Optimize on my Delta tables, but how often should I be doing this? Am I supposed to do this after every time I load data?
- 3709 Views
- 1 replies
- 0 kudos
- 0 kudos
It would depend on how frequently you update the table and how often you read it. If you have a daily ETL job updating a delta table, it might make sense to run OPTIMIZE at the end of it so that subsequent reads would benefit from the performance imp...
- 0 kudos
- 4880 Views
- 1 replies
- 0 kudos
Resolved! How do I know which worker type to choose when creating my cluster?
I am new to using Databricks and want to create a cluster, but there are many different worker types to choose from. How do I know which worker type is the right type for my use case?
- 4880 Views
- 1 replies
- 0 kudos
- 0 kudos
For delta workloads, where you could benefit from caching it is recommended to use storage optimized instances that come with NVMe SSDs. For other workloads, it would be a good idea to check Ganglia metrics to see whether your workload is Cpu/Memory ...
- 0 kudos
- 3160 Views
- 2 replies
- 1 kudos
Can you run non-spark jobs on Databricks?
Is spark the only type of code that can run on a Databricks cluster?
- 3160 Views
- 2 replies
- 1 kudos
- 1 kudos
Databricks has a Runtime for Machine Learning that comes with a lot of libraries/frameworks pre-installed. This allows you to run for example PyTorch / TensorFlow code without worrying about infrastructure setup, configuration and dependency manage...
- 1 kudos
- 2020 Views
- 1 replies
- 0 kudos
Resolved! What options do I have for controlling end user access to data?
For security and privacy reasons I need to limit what datasets are available for access by end users. How can I accomplish this in a Databricks workspace?
- 2020 Views
- 1 replies
- 0 kudos
- 0 kudos
Unity Catalog is the recommended approach as it lets you manage fine-grained data permissions using standard ANSI SQL / UI . More details could be found here
- 0 kudos
- 3598 Views
- 2 replies
- 0 kudos
Resolved! When should I set the cluster mode to High Concurrency vs Standard?
How do I know which mode I should be using when creating a cluster?
- 3598 Views
- 2 replies
- 0 kudos
- 0 kudos
High Concurrency clusters are ideal for groups of users who need to share resources or run ad-hoc jobs - for example data scientists sharing a cluster. They come with Query Watchdog, a process which keeps disruptive queries in check by automatically ...
- 0 kudos
- 3160 Views
- 1 replies
- 1 kudos
Resolved! How long does the automatic notebook Revision History store the changes?
I am wondering how far back I can restore old versions of my notebook.
- 3160 Views
- 1 replies
- 1 kudos
- 1 kudos
I believe it stores since the beginning of the creation of the notebook assuming the revision history doesn't get cleared.
- 1 kudos
| User | Count |
|---|---|
| 1628 | |
| 790 | |
| 510 | |
| 349 | |
| 287 |
