- 1372 Views
- 1 replies
- 0 kudos
- 1372 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks jobs api can invoke code from cloud storage. But please note that it will not be stored as a Databricks notebook, rather it would be a source file or jar.
- 0 kudos
- 2766 Views
- 1 replies
- 0 kudos
How do you capture change logs from RDMS source and ingest the changes in Databricks AWS?
A common option is to use AWS DMS, https://databricks.com/blog/2019/07/15/migrating-transactional-data-to-a-delta-lake-using-aws-dms.html
- 2766 Views
- 1 replies
- 0 kudos
- 0 kudos
https://databricks.com/blog/2019/07/15/migrating-transactional-data-to-a-delta-lake-using-aws-dms.html
- 0 kudos
- 6133 Views
- 2 replies
- 0 kudos
How I can test my Python code that I wrote using notebooks?
I've written the code in the notebooks using the Python, and I want to add tests to it to make sure that it won't break when I do more changes.What tools can I use for that tasks?
- 6133 Views
- 2 replies
- 0 kudos
- 0 kudos
@Alex Ott has an awesome answer!Here is a great blog from our engineering team that may help as well. https://databricks.com/blog/2020/01/16/automate-deployment-and-testing-with-databricks-notebook-mlflow.html
- 0 kudos
- 1648 Views
- 0 replies
- 0 kudos
Spark vs. Ray?
Ray has been getting a lot of traction lately for shining at distributed compute.What are the primary differences between Spark and Ray?In which areas/applications would each be best ? (i.e. Reinforcement Learning)In which cases would it make sense f...
- 1648 Views
- 0 replies
- 0 kudos
- 1016 Views
- 0 replies
- 0 kudos
How do you capture delta changes while building a delta pipeline and going from bronze to silver to gold?
You can use change data feed feature of delta tables as described here: https://docs.databricks.com/delta/delta-change-data-feed.html
- 1016 Views
- 0 replies
- 0 kudos
- 2780 Views
- 2 replies
- 0 kudos
Requirement to Run Koalas
Hi I am planning to run Koalas on Databricks environment, What are the requirements for running Koalas there
- 2780 Views
- 2 replies
- 0 kudos
- 0 kudos
Koalas is great! This really helps ease the transition from Pandas to Spark, because you can just use the same Pandas functions/classes through the Koalas API but everything runs in the background in Spark.
- 0 kudos
- 4547 Views
- 1 replies
- 0 kudos
what does this error in hyperopt mean, One error that users commonly encounter with Hyperopt is: There are no evaluation tasks, cannot return argmin of task losses.?
This means that no trial completed successfully. This almost always means that there is a bug in the objective function, and every invocation is resulting in an error. See the error output in the logs for details. In Databricks, the underlying error ...
- 4547 Views
- 1 replies
- 0 kudos
- 0 kudos
The fmin function should be of the form:def evaluate_hyperparams(params): """ This method will be passed to `hyperopt.fmin()`. It fits and evaluates the model using the given hyperparameters to get the validation loss. :param params: This d...
- 0 kudos
- 1964 Views
- 0 replies
- 0 kudos
Hyperopt, how to setup hyper-parameter for categorical vs numerical hyperparameter?
hp.quniform (“quantized uniform”) or hp.qloguniform to generate integers. hp.choice is the right choice when, for example, choosing among categorical choices (which might in some situations even be integers, but not usually).https://databricks.com/b...
- 1964 Views
- 0 replies
- 0 kudos
- 4339 Views
- 3 replies
- 0 kudos
COPY INTO: How to add a partitioning?
The command COPY INTO from Databricks provides an idempotent file ingestion into a delta table, see here. From the docs, an example command looks like this:COPY INTO delta.`target_path` FROM (SELECT key, index, textData, 'constant_value' FROM 'sour...
- 4339 Views
- 3 replies
- 0 kudos
- 0 kudos
If you're looking to partition your `target_path` table, then it's recommended to define the partition keys prior to the COPY INTO command (at the DDL level)E.g.// Drop table if it already exists without the partition key defined (OPTIONNAL) DROP TAB...
- 0 kudos
- 10380 Views
- 1 replies
- 0 kudos
Resolved! Why do I always see "Executor heartbeat timed out" messages in the Spark Driver logs
Often, I see "Executor heartbeat timed out" messages in the Spark driver logs. Sometimes job fails with this error. Will increasing "spark.executor.heartbeatInterval" help to mitigate the issue ?
- 10380 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a common misconception that increasing "spark.executor.heartbeatInterval" will help to mitigate or resolve the heartbeat issues. In fact, increasing the spark.executor.heartbeatInterval will increase the chance of the error and worse the situ...
- 0 kudos
- 3568 Views
- 3 replies
- 0 kudos
How to check my streaming job's metrics?
I would like to know if there is a way to keep track of my running streaming job.
- 3568 Views
- 3 replies
- 0 kudos
- 0 kudos
Streaming metrics are available/exposed mainly through 3 ways:Streaming UI, which is available from Spark 3/DBR 7Streaming listener/Observable metrics APISpark driver logs. Search for the string "Streaming query made progress". The metrics are logged...
- 0 kudos
- 2237 Views
- 1 replies
- 0 kudos
- 2237 Views
- 1 replies
- 0 kudos
- 0 kudos
yes. it is actually quite fun to build Looker dashboards on top of delta lake. here is a Databricks on Looker tutorial that I created when Databricks on GCP was released.
- 0 kudos
- 2150 Views
- 1 replies
- 0 kudos
how do I change my workspace account owner?
If I need to change my account owner for my workspace, how would my team go about doing this?
- 2150 Views
- 1 replies
- 0 kudos
- 0 kudos
You need to create a ticket to data bricks support to change that
- 0 kudos
- 1031 Views
- 1 replies
- 0 kudos
Koalas or Pyspark
Should I use PySpark’s DataFrame API or Koalas, Which one is recommended , is there any performance impact if i use koalas or little slower than pyspark API
- 1031 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are already familiar with pandas and want to leverage Spark for big data, we recommend using Koalas. If you are learning Spark from ground up, we recommend you start with PySpark’s API.
- 0 kudos
- 1253 Views
- 1 replies
- 0 kudos
Photon usage
How do I know how much of a query/job used Photon?
- 1253 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are using Photon on Databricks SQLClick the Query History icon on the sidebar.Click the line containing the query you’d like to analyze.On the Query Details pop-up, click Execution Details.Look at the Task Time in Photon metric at the bottom.
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
