cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

santhoshi
by New Contributor
  • 1111 Views
  • 0 replies
  • 0 kudos

Learn AEM Training Online in BangaloreAEM Training offered by Techsolidity is closely aligned with industry requirements and gives you strong knowledg...

Learn AEM Training Online in BangaloreAEM Training offered by Techsolidity is closely aligned with industry requirements and gives you strong knowledge to effectively manage marketing content and assets. This AEM Developer training is a combination o...

  • 1111 Views
  • 0 replies
  • 0 kudos
dslin
by Databricks Partner
  • 4038 Views
  • 4 replies
  • 0 kudos

FileNotFoundError when running dbx execute

Hi,I'm very new to databricks, this might be a basic question.I can't find a way to run my local python file with databricks successfully. When I run the following `execute` command, I got a FileNotFoundError.`dbx execute --cluster-id=*** --job=Sampl...

  • 4038 Views
  • 4 replies
  • 0 kudos
Latest Reply
Vidula
Databricks Partner
  • 0 kudos

Hi @Di Lin​ Thanks for the quick response.Regards

  • 0 kudos
3 More Replies
thewfhengineer
by New Contributor III
  • 13158 Views
  • 7 replies
  • 5 kudos

How to implement Source to Target ETL Mapping sheet in PySpark using Delta tables

Schema Design :Source : Miltiple CSV Files like (SourceFile1 ,SourceFile2)Target : Delta Table like (Target_Table)Excel File : ETL_Mapping_SheetFile Columns : SourceTable ,SourceColumn, TargetTable, TargetColum , MappingLogicMappingLogic columns cont...

image
  • 13158 Views
  • 7 replies
  • 5 kudos
Latest Reply
AmanSehgal
Honored Contributor III
  • 5 kudos

Following on @Werner Stinckens​ response, if you can give an example then it will be good.Ideally you can read each row from excel file in python and pass each column as a parameter to a function.Eg; def apply_mapping_logic(SourceTable ,SourceColumn,...

  • 5 kudos
6 More Replies
109005
by Databricks Partner
  • 11297 Views
  • 5 replies
  • 6 kudos
  • 11297 Views
  • 5 replies
  • 6 kudos
Latest Reply
Prabakar
Databricks Employee
  • 6 kudos

Hi @Ayushi Pandey​ are you looking for Geomesa library support? If so please refer this notebook.https://www.databricks.com/notebooks/geomesa-h3-notebook.html

  • 6 kudos
4 More Replies
Sha_1890
by New Contributor III
  • 8874 Views
  • 5 replies
  • 3 kudos

java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class Error on writing a dataframe to a SQL DB in azure

I have to write the extracted data from XML to DB , i am using Dataframe for transformation and trying to load that to DB.I have installed these libraries,com.databricks:spark-xml_2.12:0.15.0com.microsoft.azure:spark-mssql-connector_2.11_2.4:1.0.2and...

  • 8874 Views
  • 5 replies
  • 3 kudos
Latest Reply
Vidula
Databricks Partner
  • 3 kudos

Hi @shafana Roohi Jahubar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...

  • 3 kudos
4 More Replies
IdanYaffe
by New Contributor II
  • 2146 Views
  • 1 replies
  • 2 kudos

Cannot find an AWS Cloudwatch init script that supports runtime 11.x

Hi all,I'm using the AWS CW init global script in order to monitor my clusters' instances.I'm also using data live tables with some autoloader jobs.Unfortunately, the data live tables are now running runtime version 11.As a result, newly created pipe...

  • 2146 Views
  • 1 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Databricks MVP
  • 2 kudos

Unfortunately, in delta live tables, you can not specify runtime (except current and preview, which you mentioned). It could be helpful that DLT runtimes releases are mentioned on the databricks side the same way as SQL, ML, and standard ones @Kaniz ...

  • 2 kudos
antoniodavideca
by New Contributor III
  • 23758 Views
  • 4 replies
  • 3 kudos

Resolved! install dbutils locally

So. Since I would run a git_source as a notebook_task inside a databricks Job, I read that it's possible to forward to the notebook_task (and of course now to git_source) a bunch of parameters via the `base_parameters` field on Rest API.But, on my gi...

  • 23758 Views
  • 4 replies
  • 3 kudos
Latest Reply
antoniodavideca
New Contributor III
  • 3 kudos

The way I was able to fix, was installing on my local dev environment `databricks-connect` as a pip library. This would emulate the whole databricks `dbutils` package, even if wouldn't work locally. But since I just needed to develop to have the func...

  • 3 kudos
3 More Replies
FireStrings
by New Contributor
  • 2422 Views
  • 2 replies
  • 1 kudos

Getting error after try to start cluster

When we try to start any cluster, we get this message, in event log:{"reason": {"code": "UNEXPECTED_LAUNCH_FAILURE","parameters": {"databricks_error_message": "Unexpected failure during launch."}}}All the clusters in this workspace have the same beha...

  • 2422 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vidula
Databricks Partner
  • 1 kudos

Hi @Leo Lira​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
Infosys_18
by Databricks Partner
  • 1894 Views
  • 2 replies
  • 0 kudos

Cannot sign in at databricks partner-academy portal

Hi thereI have used my company email to register an account for customer-academy.databricks.com a while back.Now what I need to do is create an account with partner-academy.databricks.com using my company email too.However when I register at partner-...

  • 1894 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Databricks Partner
  • 0 kudos

Hi @Mandira Mukhopadhyay​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. 

  • 0 kudos
1 More Replies
abhi2
by New Contributor
  • 1724 Views
  • 1 replies
  • 0 kudos

Cannot sign in at databricks customer-academy portal

Hi thereI have used my company email to register an account for partner-academy.databricks.com a while back.Now what I need to do is create an account with customer-academy.databricks.com using my company email too.However when I register at customer...

  • 1724 Views
  • 1 replies
  • 0 kudos
Latest Reply
Vidula
Databricks Partner
  • 0 kudos

Hi @Abhiram Ambadipudi​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. 

  • 0 kudos
Barb
by New Contributor III
  • 6098 Views
  • 6 replies
  • 0 kudos

How do I get my notebooks displayed in the order that they were last modified

I use databricks for analytic work, and I have a lot of notebooks looking at different things. Right now they are listed in alphabetic order by the name of the notebook. How do I make the order display the most recently created/modified notebooks at...

  • 6098 Views
  • 6 replies
  • 0 kudos
Latest Reply
Vidula
Databricks Partner
  • 0 kudos

Hi @Barb Krienke​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
5 More Replies
159312
by New Contributor III
  • 3635 Views
  • 4 replies
  • 2 kudos

Access workflow settings from within a notebook.

I have a notebook used for a dlt pipeline. The pipeline should perform an extra task if the pipeline is run as a full refresh. Right now, I have to set an extra configuration parameter when I run a full refresh. Is there a way to programmatically...

  • 3635 Views
  • 4 replies
  • 2 kudos
Latest Reply
Vidula
Databricks Partner
  • 2 kudos

Hi @Ben Bogart​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 2 kudos
3 More Replies
Newyn
by Databricks Partner
  • 6514 Views
  • 5 replies
  • 6 kudos

Resolved! Where does Python "import" statement in a Notebook search (on Azure) for libraries?

I know that import can find installed cluster libraries, workspace libraries etc. as well as how to create wheel libraries externally and upload them.However, in a project I came across the feature that import was able to find the folder in, if I rem...

  • 6514 Views
  • 5 replies
  • 6 kudos
Latest Reply
Vidula
Databricks Partner
  • 6 kudos

Hi @Jonas Mellin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 6 kudos
4 More Replies
Vashista
by New Contributor II
  • 5764 Views
  • 4 replies
  • 0 kudos

Blank page after logging into community cloud

After logging in on community.cloud.databricks.com. A blank white page appears as if still loading. Does not open even when I refresh. Not an internet connectivity issue. How do I access the page ? I'm running on safari

  • 5764 Views
  • 4 replies
  • 0 kudos
Latest Reply
Vidula
Databricks Partner
  • 0 kudos

Hi @Vashista Thakuri​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 0 kudos
3 More Replies
949991
by New Contributor
  • 1991 Views
  • 1 replies
  • 0 kudos

"assertion failed: Invalid batch"

My autoloader based streaming use microbatch process. I can not pass the following error. "java.lang.AssertionError: assertion failed: Invalid batch: payrollOutputs#217439260,meta#217439261,_metadata#217439267 != payrollOutputs#217484957,meta#2174849...

  • 1991 Views
  • 1 replies
  • 0 kudos
Latest Reply
Vidula
Databricks Partner
  • 0 kudos

Hi @Julian.He​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
Labels