cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
ossinova
by • Contributor II
  • 7305 Views
  • 4 replies
  • 1 kudos

ADF - Failed To Send Request to Databricks Cluster

I have noticed that my orchestrated pipelines (in ADF) sometimes fail due to this error:ErrorCode=FailedToReadFromAzureDatabricksDeltaLake,Failed to read from azure databricks delta lake. Error message : Failed to send request to Azure Databricks Clu...

Data Engineering
Reply
  • 7305 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
3 More Replies
fritzwijaya
by • New Contributor II
  • 2755 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks Premium for Personal

Hi Databricks Community,I want to setup premium cluster with multiple nodes to run some Spark workload for my personal project. Do Databricks offer​s premium upgrade for personal? How to do upgrade?​Thanks.​

Data Engineering
Reply
  • 2755 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @fritzwijaya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 2 kudos
4 More Replies
tanjil
by • New Contributor III
  • 8033 Views
  • 6 replies
  • 6 kudos

Resolved! Read and transform CSVs in parallel.

I need to read and transform several CSV files and then append them to a single data frame. I am able to do this in databricks using simple for loops, but I would like to speed this up.Below is the rough structure of my code: for filepath in all_file...

Data Engineering
Reply
  • 8033 Views
  • 6 replies
  • 6 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 6 kudos

Hi @tanjil​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 6 kudos
5 More Replies
nemesis
by • New Contributor
  • 1466 Views
  • 2 replies
  • 0 kudos

Same Public ip of two machines in workspace.

Is there any way to change public ip of driver node? when I try to launch two different instance (single node), I see those two machine have the same public ip.I'm writing this command into notebook of each instance. I got the same ip.from requests i...

Data Engineering
Reply
  • 1466 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Hakan Tekbas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
1 More Replies
deisou
by • New Contributor
  • 2687 Views
  • 1 replies
  • 0 kudos

How to change Databricks Academy username(Email)?

Is there a way to change my Databricks Academy username(email)? It is greyed out in my profile and I cannot update it. How do I go about getting it updated?

Data Engineering
Reply
  • 2687 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Please go through the Databricks Academy FAQs here: https://files.training.databricks.com/lms/docebo/databricks-academy-faq.pdfAlso, please go through the post here: https://community.databricks.com/s/feed/0D53f00001dq6W6CAI.

  • 0 kudos
Gim
by • Contributor
  • 1709 Views
  • 1 replies
  • 1 kudos

Incremental loading from Delta tables to MySQL database

I was thinking of using Azure Data Factory to orchestrate this. But is it possible to do incremental loads from tables in Delta format going to MySQL database? My Delta table sources will be coming from ADLS storage. Would I need to convert them firs...

Data Engineering
Reply
  • 1709 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, You can refer: https://community.databricks.com/s/question/0D53f00001UCQJ7CAP/delta-tables-incremental-backup-method. Please let us know if you need any further clarification on the same.

  • 1 kudos
RantoB
by • Valued Contributor
  • 8764 Views
  • 10 replies
  • 8 kudos

Resolved! Import notebook with python script using API

Hi,I would like to import a python notebook to my databricks workspace from my local machine using a python script.I manages to create the folder but then I have a status code 400 when I try to import a file :create_folder = requests.post( '{}/api/...

Data Engineering
Reply
  • 8764 Views
  • 10 replies
  • 8 kudos
View Replies
Latest Reply
RantoB
Valued Contributor
  • 8 kudos

Hi, Thanks for your answer.Actually both your code and mine are working. However, I cannot write in the directory Repos which is reserved (but I can create subdirectories...)Thanks to your code I got an error message which helped me to understand. Wi...

  • 8 kudos
9 More Replies
BeginnerBob
by • New Contributor III
  • 7473 Views
  • 3 replies
  • 4 kudos

How to compare notebooks for deployment

I have 2 Azure Databricks workspaces (test and prod)I want to compare the notebooks to see the changes. Are there any tools which can help do this?Thanks,

Data Engineering
Reply
  • 7473 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
BeginnerBob
New Contributor III
  • 4 kudos

Thanks Hubert.What's the best approach here? Do I have 2 trunks in my repos (1 for dev and the other for production)?Can I do a compare between the 2 trunks to generate all the notebooks which need to be deployed. How do I do this compare (in DevOps,...

  • 4 kudos
2 More Replies
santhoshi
by • New Contributor
  • 709 Views
  • 0 replies
  • 0 kudos

Learn AEM Training Online in BangaloreAEM Training offered by Techsolidity is closely aligned with industry requirements and gives you strong knowledg...

Learn AEM Training Online in BangaloreAEM Training offered by Techsolidity is closely aligned with industry requirements and gives you strong knowledge to effectively manage marketing content and assets. This AEM Developer training is a combination o...

Data Engineering
Reply
  • 709 Views
  • 0 replies
  • 0 kudos
dslin
by • New Contributor III
  • 3013 Views
  • 4 replies
  • 0 kudos

FileNotFoundError when running dbx execute

Hi,I'm very new to databricks, this might be a basic question.I can't find a way to run my local python file with databricks successfully. When I run the following `execute` command, I got a FileNotFoundError.`dbx execute --cluster-id=*** --job=Sampl...

Data Engineering
Reply
  • 3013 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Di Lin​ Thanks for the quick response.Regards

  • 0 kudos
3 More Replies
anonymous1
by • New Contributor III
  • 8339 Views
  • 7 replies
  • 5 kudos

How to implement Source to Target ETL Mapping sheet in PySpark using Delta tables

Schema Design :Source : Miltiple CSV Files like (SourceFile1 ,SourceFile2)Target : Delta Table like (Target_Table)Excel File : ETL_Mapping_SheetFile Columns : SourceTable ,SourceColumn, TargetTable, TargetColum , MappingLogicMappingLogic columns cont...

image
Data Engineering
Reply
  • 8339 Views
  • 7 replies
  • 5 kudos
View Replies
Latest Reply
AmanSehgal
Honored Contributor III
  • 5 kudos

Following on @Werner Stinckens​ response, if you can give an example then it will be good.Ideally you can read each row from excel file in python and pass each column as a parameter to a function.Eg; def apply_mapping_logic(SourceTable ,SourceColumn,...

  • 5 kudos
6 More Replies
109005
by • New Contributor III
  • 6064 Views
  • 5 replies
  • 6 kudos

Resolved! Does Databricks support writing Geography/Geometry data into BigQuery?

Data Engineering
Reply
  • 6064 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 6 kudos

Hi @Ayushi Pandey​ are you looking for Geomesa library support? If so please refer this notebook.https://www.databricks.com/notebooks/geomesa-h3-notebook.html

  • 6 kudos
4 More Replies
Sha_1890
by • New Contributor III
  • 7007 Views
  • 5 replies
  • 3 kudos

java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class Error on writing a dataframe to a SQL DB in azure

I have to write the extracted data from XML to DB , i am using Dataframe for transformation and trying to load that to DB.I have installed these libraries,com.databricks:spark-xml_2.12:0.15.0com.microsoft.azure:spark-mssql-connector_2.11_2.4:1.0.2and...

Data Engineering
Reply
  • 7007 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 3 kudos

Hi @shafana Roohi Jahubar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...

  • 3 kudos
4 More Replies
IdanYaffe
by • New Contributor II
  • 1439 Views
  • 1 replies
  • 2 kudos

Cannot find an AWS Cloudwatch init script that supports runtime 11.x

Hi all,I'm using the AWS CW init global script in order to monitor my clusters' instances.I'm also using data live tables with some autoloader jobs.Unfortunately, the data live tables are now running runtime version 11.As a result, newly created pipe...

Data Engineering
Reply
  • 1439 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Unfortunately, in delta live tables, you can not specify runtime (except current and preview, which you mentioned). It could be helpful that DLT runtimes releases are mentioned on the databricks side the same way as SQL, ML, and standard ones @Kaniz ...

  • 2 kudos
antoniodavideca
by • New Contributor III
  • 16085 Views
  • 4 replies
  • 1 kudos

Resolved! install dbutils locally

So. Since I would run a git_source as a notebook_task inside a databricks Job, I read that it's possible to forward to the notebook_task (and of course now to git_source) a bunch of parameters via the `base_parameters` field on Rest API.But, on my gi...

Data Engineering
Reply
  • 16085 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
antoniodavideca
New Contributor III
  • 1 kudos

The way I was able to fix, was installing on my local dev environment `databricks-connect` as a pip library. This would emulate the whole databricks `dbutils` package, even if wouldn't work locally. But since I just needed to develop to have the func...

  • 1 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
li.common.scroll-to.top