- 5681 Views
- 3 replies
- 0 kudos
Why does the display name of widgets not match the specified name in SQL?
However, I have no problem accessing the widget with the specified name.
- 5681 Views
- 3 replies
- 0 kudos
- 0 kudos
Yep, I figured out the issue now. Both of you gave the right information to solve the problem. My first mistake was as Jacob mentioned, `date` is actually a dataframe object here. To get the string date, I had to do similar to what Amine suggested. S...
- 0 kudos
- 5454 Views
- 2 replies
- 0 kudos
Resolved! Is there a way to validate the values of spark configs?
We can set for example:spark.conf.set('aaa.test.junk.config', 99999) , and then run spark.conf.get("aaa.test.junk.config”) which will return a value.The problem occurs when incorrectly setting to a similar matching property.spark.conf.set('spark.sql....
- 5454 Views
- 2 replies
- 0 kudos
- 0 kudos
You would solve this just like we solve this problem for all lose string references. Namely, that is to create a constant that represents the key-value you want to ensure doesn't get mistyped.Naturally, if you type it wrong the first time, it will be...
- 0 kudos
- 19398 Views
- 2 replies
- 2 kudos
How can I programmatically get my notebook path?
I'm writing some code that trains a ML model using MLflow and a given set of hyperparameters. This code is going to be run by several folks on my team and I want to make sure that the experiment that get's created is created in the same directory as ...
- 19398 Views
- 2 replies
- 2 kudos
- 2 kudos
In Scala the call is dbutils.notebook.getContext.notebookPath.getIn Python the call isdbutils.entry_point.getDbutils().notebook().getContext().notebookPath().getOrElse(None)If you need it in another language, a common practice would be to pass it thr...
- 2 kudos
- 1633 Views
- 1 replies
- 0 kudos
- 1633 Views
- 1 replies
- 0 kudos
- 0 kudos
You have a couple options to write data into a Data Warehouse. Some DWs have special connectors that allow for high performance between Databricks and the DW (for example there is a Spark connector for Snowflake and for Azure Synapse DW). Some data w...
- 0 kudos
- 3633 Views
- 1 replies
- 1 kudos
Auto-deletion of unused jobs
Is there a setting that will auto-cleanup/delete jobs that are of a certain age (say 90 days old for example)?
- 3633 Views
- 1 replies
- 1 kudos
- 1 kudos
It is not available natively in Databricks. But you can write an administration script that analyzes your jobs data and automatically cleans up the older jobs as needed. It would be easiest to do this with the jobs API. List your jobs to get all the ...
- 1 kudos
- 921 Views
- 0 replies
- 0 kudos
Databricks-connect: is it safe to store my PAT in plaintext?
i'm getting started out with databricks-join to connect to my azure databricks cluster the usage of a personal access token. it lo routerloginoks as if this token is being saved in a raw textual content report. if it really is actual, is there a ext...
- 921 Views
- 0 replies
- 0 kudos
- 1249 Views
- 0 replies
- 0 kudos
java.io.IOException: While processing file s3://test/abc/request_dt=2021-07-28/someParquetFile. [XYZ] BINARY is not in the store
Hi Team, I am facing an issue "java.io.IOException: While processing file s3://test/abc/request_dt=2021-07-28/someParquetFile. [XYZ] BINARY is not in the store" The things i did before getting the above exception: 1. Alter table tableName1 add colum...
- 1249 Views
- 0 replies
- 0 kudos
- 1377 Views
- 0 replies
- 0 kudos
Databricks secrets visible in cleartext
I have a requirement that I need a secret when starting a cluster in databricks. And I found following way of providing the secret to my init script. https://docs.databricks.com/security/secrets/secrets.html#store-the-path-to-a-secret-in-an-environm...
- 1377 Views
- 0 replies
- 0 kudos
- 1664 Views
- 1 replies
- 0 kudos
How to configure the spark environment variables to run pyspark in windows 10? where can i find these enviornment variables in windows 10 ?
Hi, I have installed spark,python and pyspark. However, I am unable to run pyspark as it is unable to find python executable path. Please help me in this regard. This is in windows 10 Regards Ramesh
- 1664 Views
- 1 replies
- 0 kudos
- 0 kudos
HiI am attaching the screenshot of the message which may help you to provide me the responseRegardspyspark-env-variables.png
- 0 kudos
- 1357 Views
- 0 replies
- 1 kudos
Databricks spark certification
Hi guys , Is there any way to get 100 showbox% off voucher or a good discount voucher for Databricks spark certification? Currently the certifica speed testtion is very costly (200$). Any help is appreciated.
- 1357 Views
- 0 replies
- 1 kudos
- 3244 Views
- 1 replies
- 1 kudos
Resolved! Tensorboard Profiler did not work on DBR 8.4 ML
The tensorboard profile board did not work. It shows loading data forever
- 3244 Views
- 1 replies
- 1 kudos
- 1 kudos
This is due to a issue reported here : https://github.com/tensorflow/profiler/issues/344The DBR 8.4ML comes with Tensor flow 2.5 and the latest version of tensorboard-plugin-profile is 2.4.To workaround the issue, you can add option --load_fast=false...
- 1 kudos
- 2230 Views
- 0 replies
- 0 kudos
Hi All, I need to extract rows of dates from a dataframe based on list of values(e.g. dates) located in a CSV file. Can anyone please help me? I have tried groupby function but am not able to get the expected result. Thanks in advance.
my dataframe looks like this.df = Datecolumn2column3Machine1-jan-2020A2-jan-2020--- A 18-jan-2020 A 11-jan-2020 B 12-jan-2020 B 6-feb-2020C7-feb-2020---C14-feb-2020C Date details csv file looks like this D = MachineSelected DateA15-jan-2020C12-f...
- 2230 Views
- 0 replies
- 0 kudos
- 2185 Views
- 0 replies
- 1 kudos
Target database.table1 must be delta table
I created a table1 with some data. Then I truncated it to load new dataset. When I do select * from table . I get row count 0. But when I am trying to copy into using following command. I get error saying target table must be delta table: COPY INTO...
- 2185 Views
- 0 replies
- 1 kudos
- 1445 Views
- 0 replies
- 0 kudos
Data bricks Write Performance
I have a requirement to replay ingestion from landing data and build silver table. I am trying to write delta file from raw Avro files based in landing zone. The raw files are located in folder based on date. I am currently using streaming to read d...
- 1445 Views
- 0 replies
- 0 kudos
- 2397 Views
- 0 replies
- 2 kudos
Integrating with Prometheus
How can I integrate Databricks clusters with Prometheus? I tried adding the following Spark property to my cluster but cannot find the Prometheus metrics endpoints. Any thoughts? spark.ui.prometheus.enabled = true
- 2397 Views
- 0 replies
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
