cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
faycal_merouane
by • New Contributor III
  • 4203 Views
  • 5 replies
  • 6 kudos

Resolved! Databricks Community Edition Not able to Login Account

hello, I'm not able to log in to my Databricks community account, I'm getting the following error when trying to log in:You entered an invalid email or password, or your workspace access is locked because of unusual activity. For help, see Community ...

Data Engineering
Reply
  • 4203 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
faycal_merouane
New Contributor III
  • 6 kudos

hi @Vidula Khanna​, I appreciate your help, I will try to contact them using help@databricks.com.thank you for your assistance.best regards.

  • 6 kudos
4 More Replies
m2chrisp
by • New Contributor II
  • 7185 Views
  • 3 replies
  • 0 kudos

Resolved! Deploying Databricks via ARM, then configuring via databricks-cli: creating a repo results in INVALID_STATE error.

Hi,I'm using databricks-cli to configure a newly-deployed Azure Databricks instance. The ARM deployment works fine, and then the databricks cli commands to create a secrets scope and add users also works just fine.Then I add a GitCredential to Datab...

Data Engineering
Reply
  • 7185 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Chris Platts​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Thanks and Regards

  • 0 kudos
2 More Replies
KJ_Saravana
by • New Contributor III
  • 7472 Views
  • 6 replies
  • 9 kudos

Resolved! Databricks cluster Init scripts on ABFSS location

HI,I have an init script which works on DBFS location during the cluster start up, but when the same shell script file is placed on ABFSS location (ADLS Gen 2 storage) I get the following init script failure error and the cluster is unable to start.E...

Data Engineering
Reply
  • 7472 Views
  • 6 replies
  • 9 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 9 kudos

Hi @Saravana KJ​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest pr...

  • 9 kudos
5 More Replies
KVNARK
by • Honored Contributor II
  • 2734 Views
  • 1 replies
  • 5 kudos

Resolved! sudden decrease in rewards point in Rewards portal

@Vidula Khanna​ May I know why the reward points decreased in Databricks community rewards portal. Could anyone help on this. I haven't ordered for anything. But today when I checked there was sudden decrease in points and the no of points came from ...

Data Engineering
Reply
  • 2734 Views
  • 1 replies
  • 5 kudos
View Replies
jogender
by • New Contributor
  • 2114 Views
  • 1 replies
  • 0 kudos

The hire react js developer from the best react js development company. We provide the most experienced hire reactjs developer as per the requirements...

The hire react js developer from the best react js development company. We provide the most experienced hire reactjs developer as per the requirements cost.

Data Engineering
Reply
  • 2114 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Sovchenko
New Contributor II
  • 0 kudos

Thanks for sharing! By the way, if you are interested in the costs of hire react js developers, I advise you to read this: https://www.cleveroad.com/blog/how-much-does-it-cost-to-hire-a-react-developer/

  • 0 kudos
Harun
by • Honored Contributor
  • 9621 Views
  • 2 replies
  • 0 kudos

Issue with Pyspark GroupBy GroupedData

Hi Guys,I am working on streaming data movement from bronze to silver. My bronze table is having a entity_name column, based on the entity_name column i need to create multiple silver tables.I tried the below approach, But it is failing with error "'...

Data Engineering
Reply
  • 9621 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Harun Raseed Basheer​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ...

  • 0 kudos
1 More Replies
Rsa
by • New Contributor II
  • 5309 Views
  • 2 replies
  • 2 kudos

Resolved! Error while using Array_contains function in left join condition

'Item_id' is column in array format like ["ba1b-5fbe1547ddd5", "88f9-ac3b93334f69", "8bba-4075a47eb814"] in table1 and table2 has column Id with single value like ba1b-5fbe1547ddd5.While join two table select table1.*,table2.*from table1left join tab...

Data Engineering
Reply
  • 5309 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rishabh Shanker​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
1 More Replies
Direo
by • Contributor II
  • 2828 Views
  • 2 replies
  • 1 kudos

Resolved! How does pyspark work in these two scenarios?

I have two scenarios with different outcomes:Scenario 1:from pyspark.sql.functions import *# create sample dataframesdf1 = spark.createDataFrame([(1, 2, 3), (2, 3, 4)], ["a", "b", "c"])df2 = spark.createDataFrame([(1, 5, 6, 7), (2, 8, 9, 10)], ["a", ...

Data Engineering
Reply
  • 2828 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Direo Direo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 1 kudos
1 More Replies
MerelyPerfect
by • New Contributor II
  • 3681 Views
  • 3 replies
  • 1 kudos

read base64 json column with Autoloader and inferschema.

I have json files falling in our blob with two fields, 1. offset(integer), 2. value(base64).This value column is json with unicode. so they sent it as base64. Challenge is this json is very large with 100+ fields. so we cannot define the schema. We c...

Data Engineering
Reply
  • 3681 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @MerelyPerfect Per​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 1 kudos
2 More Replies
Mado
by • Valued Contributor II
  • 9531 Views
  • 1 replies
  • 1 kudos

Resolved! How to get today's date in the local time zone?

I am trying to get today's date in the local time zone:from pyspark.sql.functions import * date = to_date(from_utc_timestamp(current_timestamp(), 'Australia/Melbourne'))What I get using the above code is a column object. How can I get its value in a...

image
Data Engineering
Reply
  • 9531 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Hemant
Valued Contributor II
  • 1 kudos

Hi @Mohammad Saber​ , you can use pytz and datetime python package for your usecase,, attaching code snippet in below screen shot. 

  • 1 kudos
maymay1993
by • New Contributor II
  • 1695 Views
  • 2 replies
  • 2 kudos

where can I can the user manual fo databricks?

Data Engineering
Reply
  • 1695 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @may may​ Your input matters! Help our community thrive by coming back and marking the most helpful and accurate answers. Together, we can make a difference!Regards

  • 2 kudos
1 More Replies
juned
by • New Contributor III
  • 2704 Views
  • 2 replies
  • 1 kudos

How install a library that is under the /Workspace/Shared/ directory using the init.sh script in a cluster?

I would like to install a library that is under the /Workspace/Shared/ directory using the init.sh script in a cluster. How to access the /Workspace/Shared/ folder in shell? This page only shows how to access manually but doesn't show how to access i...

Data Engineering
Reply
  • 2704 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Juned Mala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
Mr__D
by • New Contributor II
  • 7083 Views
  • 2 replies
  • 3 kudos

Do we really need Autoloader for batch processing.?

Hi All,It seem AutoLoader is good option for even driven data ingestion but if my job runs only once , do I still need autoloader ? I dont want to spend money to spin a cluster whole day.I know we have RunOnce option available while running a job but...

Data Engineering
Reply
  • 7083 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Deepak Bhatt​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Thanks and regards

  • 3 kudos
1 More Replies
EDDatabricks
by • Contributor
  • 2475 Views
  • 2 replies
  • 3 kudos

DLT pipeline slow streaming (root cause needs to be identified)

Dear support,we have the following situation where a set of DLT pipelines are streaming with very low rate incoming data and we need to find the root cause of this delay.In order to provide more insight about the setup of the DLT pipelines and some m...

Data Engineering
Reply
  • 2475 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @EDDatabricks EDDatabricks​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that ...

  • 3 kudos
1 More Replies
shiva12494
by • New Contributor II
  • 4555 Views
  • 2 replies
  • 2 kudos

Issue with reading exported tables stored in parquet

Hi All, I am exported all tables from postgres snapshot into S3 in parquet format. I am trying to read the table using databricks and i am unable to do so. I get the following error: "Unable to infer schema for Parquet. It must be specified manually....

Data Engineering
Reply
  • 4555 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @shiva charan velichala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that bes...

  • 2 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels