I have a requirement that I need a secret when starting a cluster in databricks. And I found following way of providing the secret to my init script. https://docs.databricks.com/security/secrets/secrets.html#store-the-path-to-a-secret-in-an-environm...
Hi, I have installed spark,python and pyspark. However, I am unable to run pyspark as it is unable to find python executable path. Please help me in this regard. This is in windows 10 Regards Ramesh
HiI am attaching the screenshot of the message which may help you to provide me the responseRegardspyspark-env-variables.png
Hi guys , Is there any way to get 100 showbox% off voucher or a good discount voucher for Databricks spark certification? Currently the certifica speed testtion is very costly (200$). Any help is appreciated.
The tensorboard profile board did not work. It shows loading data forever
This is due to a issue reported here : https://github.com/tensorflow/profiler/issues/344The DBR 8.4ML comes with Tensor flow 2.5 and the latest version of tensorboard-plugin-profile is 2.4.To workaround the issue, you can add option --load_fast=false...
my dataframe looks like this.df = Datecolumn2column3Machine1-jan-2020A2-jan-2020--- A 18-jan-2020 A 11-jan-2020 B 12-jan-2020 B 6-feb-2020C7-feb-2020---C14-feb-2020C Date details csv file looks like this D = MachineSelected DateA15-jan-2020C12-f...
I created a table1 with some data. Then I truncated it to load new dataset. When I do select * from table . I get row count 0. But when I am trying to copy into using following command. I get error saying target table must be delta table: COPY INTO...
I have a requirement to replay ingestion from landing data and build silver table. I am trying to write delta file from raw Avro files based in landing zone. The raw files are located in folder based on date. I am currently using streaming to read d...
How can I integrate Databricks clusters with Prometheus? I tried adding the following Spark property to my cluster but cannot find the Prometheus metrics endpoints. Any thoughts? spark.ui.prometheus.enabled = true
I need to import python notebook present in s3
Can I duplicate a delta table juste by copying/pasting the whole table folder ?
Hello, Is it a good idea to Host a Schema Data Warehouse on Azure Databricks database itself. Usually we use Azure Databricks to Prep the data and then Host it on Azure Sql Database. However question is can we not Host the data on Azure Databricks i...
Hello! I am trying to forward fill a column in a Pandas dataframe based on a keyword. I have come up with: pdf_df['EEName_TEST'] = pdf_df['EEName_TEST'].str.contains('Name:').ffill() This gives me a boolean result but I still can't figure out what ...
I have Azure notebook which take widget input parameter and performs necessary action. But this note book should be called within another notebook using dbutils.notebook.run , How do I pass parameter to the widget?
I was mounting the Datalake Gen1 to Databricks for accessing and processing files, The below code was working great for the past 1 year and all of a sudden I'm getting an errorconfigs = {"df.adl.oauth2.access.token.provider.type": "ClientCredential"...
When using display, more than 1 spaces in strings are ignored. Can we change that behaviour? Are there any options for display functions? code example: display( spark.createDataFrame( [ ( 'a a' , 'a a' ) ], [ 'string_column', 'string_column_2' ] )...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group