Databricks Community

For folks that are looking to leverage Databricks as a teaching asset, please sign contact us for Databricks University Alliance. https://databricks.com/p/teach

We have a ton of great resources available for people who are wanting to learn Databricks, specifically for university students. Checkout our our university page, to learn more about Databricks Community Edition, Free workshops, and self-paced course...

Check out our auto loader capabilities that can automatically track and process files that need to be processed. AutoloaderThere are two options: directory listing, which is essentially completing the same steps that you have listed above but in a sl...

With Koalas, which is a Pandas'API on top of Spark Dataframes, there should be minimal code changes required.Please refer to this blog for more info

If you are attempting to read all the files in a directory you should be able to use a wild card and filter using the extension. For example: df = (spark .read .format("com.crealytics.spark.excel") .option("header", "True") .option("inferSchema", "tr...

There is a limit an execution context limit of 145. This means you can have at most 145 notebooks attached to a cluster. https://kb.databricks.com/execution/maximum-execution-context.htmlIf you are primarily using SQL then Databricks SQL Endpoints wo...

Hi Yes we have the best practices,I can point on the data bricks Docshttps://docs.databricks.com/administration-guide/access-control/tokens.html

we have a limit of 100 Secret scopes and each secret can have 1000 key-value pair of secrets.https://docs.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes

Hi we have best practices around streaming with Kinesis and you can find them here .https://docs.databricks.com/spark/latest/structured-streaming/kinesis-best-practices.html

nope not possible you can set the name that shows up in azure portal though (but only when you’re first creating the WS i believe)

Running Jobs configured to use an Existing All-Purpose Cluster is considered to be a Job client. If you set workload_type.clients.jobs = false on an all-purpose cluster, it can not be used to run jobs.