- 2289 Views
- 1 replies
- 0 kudos
- 2289 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Yes we have the best practices,I can point on the data bricks Docshttps://docs.databricks.com/administration-guide/access-control/tokens.html
- 0 kudos
- 2479 Views
- 1 replies
- 0 kudos
- 2479 Views
- 1 replies
- 0 kudos
- 0 kudos
we have a limit of 100 Secret scopes and each secret can have 1000 key-value pair of secrets.https://docs.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes
- 0 kudos
- 1852 Views
- 1 replies
- 0 kudos
- 1852 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi we have best practices around streaming with Kinesis and you can find them here .https://docs.databricks.com/spark/latest/structured-streaming/kinesis-best-practices.html
- 0 kudos
- 2062 Views
- 1 replies
- 0 kudos
- 2062 Views
- 1 replies
- 0 kudos
- 0 kudos
nope not possible you can set the name that shows up in azure portal though (but only when you’re first creating the WS i believe)
- 0 kudos
- 1847 Views
- 1 replies
- 0 kudos
- 1847 Views
- 1 replies
- 0 kudos
- 0 kudos
Running Jobs configured to use an Existing All-Purpose Cluster is considered to be a Job client. If you set workload_type.clients.jobs = false on an all-purpose cluster, it can not be used to run jobs.
- 0 kudos
- 12059 Views
- 1 replies
- 1 kudos
- 12059 Views
- 1 replies
- 1 kudos
- 1 kudos
In Databricks, each cluster creates an initial spark session. And each notebook creates a spark subsession within the same. A temporary View created in one notebook isn't accessible to others. If you need to share view across notebooks, you use Globa...
- 1 kudos
- 5903 Views
- 1 replies
- 0 kudos
Resolved! What is the difference between coalesce and repartition when it comes to shuffle partitions in spark
- 5903 Views
- 1 replies
- 0 kudos
- 0 kudos
Coalesce essentially groups multiple partitions into a larger partitions. So use coalesce when you want to reduce the number of partitions (and also tasks) without impacting sort order. Ex:- when you want to write-out a single CSV file output instea...
- 0 kudos
- 3267 Views
- 1 replies
- 0 kudos
- 3267 Views
- 1 replies
- 0 kudos
- 0 kudos
You can disable magic commands for a workspace but not on a per-user basis. Additional details about Workspace ACLs can be found here - https://docs.databricks.com/security/access-control/workspace-acl.html
- 0 kudos
- 10325 Views
- 1 replies
- 0 kudos
- 10325 Views
- 1 replies
- 0 kudos
- 0 kudos
Narrow Transformation: In Narrow transformation, all the elements that are required to compute the records in single partition live in the single partition of parent RDD. Ex:- Select, Filter, Union, Wide Transformation: Wide transformation, all the e...
- 0 kudos
- 5463 Views
- 1 replies
- 0 kudos
- 5463 Views
- 1 replies
- 0 kudos
- 0 kudos
Spark's execution engine is designed to be Lazy. In effect, you're first up build up your analytics/data processing request through a series of Transformations which are then executed by an ActionTransformations are kind of operations which will tran...
- 0 kudos
- 23596 Views
- 2 replies
- 0 kudos
- 23596 Views
- 2 replies
- 0 kudos
- 0 kudos
%run is copying code from another notebook and executing it within the one its called from. All variables defined in the notebook being called are therefore visible to the caller notebook dbutils.notebook.run() is more around executing different note...
- 0 kudos
- 92607 Views
- 2 replies
- 2 kudos
- 92607 Views
- 2 replies
- 2 kudos
- 2 kudos
Z-Ordering is a technique to colocate related information in the same set of files. This co-locality is automatically used by Delta Lake on Databricks data-skipping algorithms to dramatically reduce the amount of data that needs to be read. Syntax fo...
- 2 kudos
- 6167 Views
- 2 replies
- 0 kudos
- 6167 Views
- 2 replies
- 0 kudos
- 0 kudos
Download & Install the Databricks ODBC DriverGet the hostname, port, HTTP Path as described here – there’s slightly different steps for cluster (DDE) or SQL endpoint (DSQL)Get a PAT tokenUse the curl command to validate the network settings using the...
- 0 kudos
- 5090 Views
- 1 replies
- 0 kudos
How can I create from scratch a brand new Dataframe with Null values using spark.createDataFrame()?
from pyspark.sql.types import * schema = StructType([ StructField("c1", IntegerType(), True), StructField("c2", StringType(), True), StructField("c3", StringType(), True)]) df = spark.createDataFrame([(1, "2", None), (3, "4", None)], schema)
- 5090 Views
- 1 replies
- 0 kudos
- 0 kudos
df = spark.createDataFrame(sc.emptyRDD(), schema)Can you try this?
- 0 kudos
- 5088 Views
- 1 replies
- 1 kudos
Resolved! cluster start Issues
Some of the Jobs are failing in prod with below error message:Can you please check and let us know the reason for this? These are running under pool cluster.Run result unavailable: job failed with error messageUnexpected failure while waiting for the...
- 5088 Views
- 1 replies
- 1 kudos
- 1 kudos
@Kunal Gaurav , This status code only occurs in one of two conditions:We’re able to request the instances for the cluster but can’t bootstrap them in time We setup the containers on each instance, but can’t start the containers in timethis is an edg...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
