cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
rameshkg
by • New Contributor II
  • 1894 Views
  • 1 replies
  • 0 kudos

How to configure the spark environment variables to run pyspark in windows 10? where can i find these enviornment variables in windows 10 ?

Hi, I have installed spark,python and pyspark. However, I am unable to run pyspark as it is unable to find python executable path. Please help me in this regard. This is in windows 10 Regards Ramesh

Data Engineering
Reply
  • 1894 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
rameshkg
New Contributor II
  • 0 kudos

HiI am attaching the screenshot of the message which may help you to provide me the responseRegardspyspark-env-variables.png

  • 0 kudos
fabiwilys84
by • New Contributor II
  • 1483 Views
  • 0 replies
  • 1 kudos

Databricks spark certification

Hi guys , Is there any way to get 100 showbox% off voucher or a good discount voucher for Databricks spark certification? Currently the certifica speed testtion is very costly (200$). Any help is appreciated.

Data Engineering
Reply
  • 1483 Views
  • 0 replies
  • 1 kudos
User16752239289
by • Databricks Employee
  • 3638 Views
  • 1 replies
  • 1 kudos

Resolved! Tensorboard Profiler did not work on DBR 8.4 ML

The tensorboard profile board did not work. It shows loading data forever

image (17)
Data Engineering
Reply
  • 3638 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
User16752239289
Databricks Employee
  • 1 kudos

This is due to a issue reported here : https://github.com/tensorflow/profiler/issues/344The DBR 8.4ML comes with Tensor flow 2.5 and the latest version of tensorboard-plugin-profile is 2.4.To workaround the issue, you can add option --load_fast=false...

  • 1 kudos
SindhuG
by • New Contributor
  • 2402 Views
  • 0 replies
  • 0 kudos

Hi All, I need to extract rows of dates from a dataframe based on list of values(e.g. dates) located in a CSV file. Can anyone please help me? I have tried groupby function but am not able to get the expected result. Thanks in advance.

my dataframe looks like this.df = Datecolumn2column3Machine1-jan-2020A2-jan-2020--- A 18-jan-2020 A 11-jan-2020 B 12-jan-2020 B 6-feb-2020C7-feb-2020---C14-feb-2020C Date details csv file looks like this D = MachineSelected DateA15-jan-2020C12-f...

Data Engineering
Reply
  • 2402 Views
  • 0 replies
  • 0 kudos
AkankshaGupta
by • New Contributor II
  • 2306 Views
  • 0 replies
  • 1 kudos

Target database.table1 must be delta table

I created a table1 with some data. Then I truncated it to load new dataset. When I do select * from table . I get row count 0. But when I am trying to copy into using following command. I get error saying target table must be delta table: COPY INTO...

Data Engineering
Reply
  • 2306 Views
  • 0 replies
  • 1 kudos
rami1
by • New Contributor III
  • 1565 Views
  • 0 replies
  • 0 kudos

Data bricks Write Performance

I have a requirement to replay ingestion from landing data and build silver table. I am trying to write delta file from raw Avro files based in landing zone. The raw files are located in folder based on date. I am currently using streaming to read d...

Data Engineering
Reply
  • 1565 Views
  • 0 replies
  • 0 kudos
TyronZerafa
by • New Contributor II
  • 2536 Views
  • 0 replies
  • 2 kudos

Integrating with Prometheus

How can I integrate Databricks clusters with Prometheus? I tried adding the following Spark property to my cluster but cannot find the Prometheus metrics endpoints. Any thoughts? spark.ui.prometheus.enabled = true

Data Engineering
Reply
  • 2536 Views
  • 0 replies
  • 2 kudos
AbhishekBreeks
by • New Contributor II
  • 1800 Views
  • 0 replies
  • 0 kudos

Host a Star Schema Data Warehouse on Azure Databricks

Hello, Is it a good idea to Host a Schema Data Warehouse on Azure Databricks database itself. Usually we use Azure Databricks to Prep the data and then Host it on Azure Sql Database. However question is can we not Host the data on Azure Databricks i...

Data Engineering
Reply
  • 1800 Views
  • 0 replies
  • 0 kudos
WhatIsHappening
by • New Contributor
  • 1544 Views
  • 0 replies
  • 0 kudos

Pandas Forward Fill Based on Keyword

Hello! I am trying to forward fill a column in a Pandas dataframe based on a keyword. I have come up with: pdf_df['EEName_TEST'] = pdf_df['EEName_TEST'].str.contains('Name:').ffill() This gives me a boolean result but I still can't figure out what ...

Data Engineering
Reply
  • 1544 Views
  • 0 replies
  • 0 kudos
stramzik
by • New Contributor II
  • 2554 Views
  • 1 replies
  • 1 kudos

Unable to mount datalake gen1 to databricks

I was mounting the Datalake Gen1 to Databricks for accessing and processing files, The below code was working great for the past 1 year and all of a sudden I'm getting an errorconfigs = {"df.adl.oauth2.access.token.provider.type": "ClientCredential"...

Data Engineering
Reply
  • 2554 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
stramzik
New Contributor II
  • 1 kudos

bumping up the thread

  • 1 kudos
lycenok
by • New Contributor II
  • 1584 Views
  • 0 replies
  • 0 kudos

display function eats consecutive spaces

When using display, more than 1 spaces in strings are ignored. Can we change that behaviour? Are there any options for display functions? code example: display( spark.createDataFrame( [ ( 'a a' , 'a a' ) ], [ 'string_column', 'string_column_2' ] )...

Data Engineering
Reply
  • 1584 Views
  • 0 replies
  • 0 kudos
Charbel
by • New Contributor II
  • 2559 Views
  • 0 replies
  • 1 kudos

Delta table is not writing data read from kafka

Guys, could you help me? I'm reading 5 kafka threads through a list and saving the data in a Delta table The execution will be 1x a day, it seems that everything is working but I noticed that when I read the topic and it has no message, it still gen...

0693f000007OoRrAAK
Data Engineering
Reply
  • 2559 Views
  • 0 replies
  • 1 kudos
Labels