Databricks Community

The fmin function should be of the form:def evaluate_hyperparams(params): """ This method will be passed to `hyperopt.fmin()`. It fits and evaluates the model using the given hyperparameters to get the validation loss. :param params: This d...

If you're looking to partition your `target_path` table, then it's recommended to define the partition keys prior to the COPY INTO command (at the DDL level)E.g.// Drop table if it already exists without the partition key defined (OPTIONNAL) DROP TAB...

Streaming metrics are available/exposed mainly through 3 ways:Streaming UI, which is available from Spark 3/DBR 7Streaming listener/Observable metrics APISpark driver logs. Search for the string "Streaming query made progress". The metrics are logged...

yes. it is actually quite fun to build Looker dashboards on top of delta lake. here is a Databricks on Looker tutorial that I created when Databricks on GCP was released.

You need to create a ticket to data bricks support to change that

If you are already familiar with pandas and want to leverage Spark for big data, we recommend using Koalas. If you are learning Spark from ground up, we recommend you start with PySpark’s API.

If you are using Photon on Databricks SQLClick the Query History icon on the sidebar.Click the line containing the query you’d like to analyze.On the Query Details pop-up, click Execution Details.Look at the Task Time in Photon metric at the bottom.

Check the S3 bucket policy and region. It looks like storage config validation is failing

The maximum number of xxx has been reached indicates that you have hit the soft limits for some of the AWS resources in that region. These are mostly soft limits and you could file a request to AWS support team to increase this

Try deleting the RUNNING workspace, wait for 5-10 minutes, and recreate the same workspace. If that doesn't solve the problem, file a support ticket

It denotes that the input payload is not what is expected the api-endpoint for the e2 accounts api. Possible causes include typo in variable values or json formatting issues ( not providing quotes etc )

To access these driver log files from the UI, you could go to the Driver Logs tab on the cluster details page. You can also configure a log delivery location for the cluster. Both worker and cluster logs are delivered to the location you specify.

When you remove a user from Databricks, a special backup folder is created in the workspace. More details at https://kb.databricks.com/notebooks/get-notebooks-deleted-user.html