cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Yagao
by • New Contributor
  • 4466 Views
  • 5 replies
  • 2 kudos

How to do python within sql query in Databricks ?

Can anyone show me one use case how to do python within sql query ?

Data Engineering
Reply
  • 4466 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
tomasz
Databricks Employee
  • 2 kudos

To run Python within a SQL query you have to first define a Python function and then register it as a UDF. Once that is done you are able to call that UDF within a SQL query. Please take a look at this documentation here:https://docs.databricks.com/s...

  • 2 kudos
4 More Replies
Komal7
by • New Contributor
  • 1097 Views
  • 2 replies
  • 0 kudos

When is AQE becoming GA? Is it coming in older spark versions?

Data Engineering
Reply
  • 1097 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Komal Gyanani​,AQE was a major improvement added to Spark 3.0. It was added since Databricks runtime 7.3 LT (Spark 3.0) https://docs.databricks.com/release-notes/runtime/releases.html and here is docs on AQE https://docs.databricks.com/spark/late...

  • 0 kudos
1 More Replies
Marcosan
by • New Contributor II
  • 1515 Views
  • 2 replies
  • 3 kudos

What’s the best way to pass dependency versions dynamically to a cluster

I am using init scripts and would like to be able to control the version of a component that we release internally and frequently. We are now manually updating a dbfs requirement.txt file but I think that this problem may have been encountered befor...

Data Engineering
Reply
  • 1515 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

You can programmatically create cluster templates in JSON files and include config JSON files with libraries needed. Cluster deployment in that scenario needs to be controlled via API https://docs.databricks.com/dev-tools/api/latest/clusters.html

  • 3 kudos
1 More Replies
avinash_goje
by • New Contributor II
  • 2802 Views
  • 2 replies
  • 2 kudos

How to send metrics from GCP Databricks to Grafana Cloud through Prometheus?

While connecting the Databricks and Grafana, I have gone through the following approach.Install Grafna Agent in Databrics Clusters from Databricks console --> Not working since the system is not booted with systemd as init systemSince Spark 3 has Pro...

Data Engineering
Reply
  • 2802 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

There is a repo with Prometheus gateway https://gist.github.com/Lowess/3a71792d2d09e38bf8f524644bbf8349. In the community, we usually use DataDog as both plays nicely https://docs.datadoghq.com/integrations/databricks/?tabs=driveronly

  • 2 kudos
1 More Replies
MMMM
by • New Contributor III
  • 1256 Views
  • 1 replies
  • 0 kudos

missing notebook from workshop

Hi,I was going through this sessionhttps://tinyurl.com/databrickshcarebut on slides there is link to notebook which is broken. can you guys fix and share the link so I could try these notebooks ?this is mentioned in the slides for notebook linkhttps:...

Data Engineering
Reply
  • 1256 Views
  • 1 replies
  • 0 kudos
View Replies
thushar
by • Contributor
  • 5101 Views
  • 3 replies
  • 2 kudos

Can we use a variable to mention the path in the %run command

To compile the Python scripts in Azure notebooks, we are using the magic command %run.The first parameter for this command is the notebook path, is it possible to mention that path in a variable (we have to construct this path dynamically during the ...

Data Engineering
Reply
  • 5101 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
User16752242622
Valued Contributor
  • 2 kudos

@Thushar R​ I don't think it is possible to pass the notebook path in a variable and run it with a %run.I believe you can make use of notebook workflows. Notebook workflows are a complement to %runhttps://docs.databricks.com/notebooks/notebook-workfl...

  • 2 kudos
2 More Replies
Saurav
by • New Contributor III
  • 5432 Views
  • 4 replies
  • 7 kudos

spark cluster monitoring and visibility

Hey. I'm working on a project where I'd like to be able to view and play around with the spark cluster metrics. I'd like to know what the utilization % and max values are for metrics like CPU, memory and network. I've tried using some open source sol...

Data Engineering
Reply
  • 5432 Views
  • 4 replies
  • 7 kudos
View Replies
Latest Reply
Saurav
New Contributor III
  • 7 kudos

Hey @Kaniz Fatma​, I Appreciate the suggestions and will be looking into them. Haven't gotten to it yet so I didn't want to mention whether they worked for me or not. Since I'm looking to avoid solutions like DataDog, I'll be checking out the Prometh...

  • 7 kudos
3 More Replies
irfanaziz
by • Contributor II
  • 2077 Views
  • 2 replies
  • 3 kudos

How to make a string column with numeric and alphabet values use as partition?

So i have two partitions defined for this delta table, One is year('GJHAR') contains year values, and the other is a string column('BUKS') with around 124 unique values. However, there is one problem with the 2nd partition column('BUKS'), The values ...

Data Engineering
Reply
  • 2077 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

@nafri A​ , So to make sure I understand correctly: if you partition the table with only numeric data in BUKS, new incoming data cannot be added if it contains a string; but the other way around it does work?Could it be that spark has inferred the co...

  • 3 kudos
1 More Replies
marta_cdc
by • New Contributor
  • 3085 Views
  • 2 replies
  • 0 kudos

Automate in code the launching of a sql script

Do you know how to automate in code the launching of a sql script? Currently I do it by selection. 

image
Data Engineering
Reply
  • 3085 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

@Marta Vicente Sánchez​, what tool are you using here? And are we talking about Databricks SQL?

  • 0 kudos
1 More Replies
Ramya
by • New Contributor III
  • 18659 Views
  • 4 replies
  • 3 kudos

Resolved! Databricks Rest API

Hi, I am having an issue accessing data bricks API 2.0/workspace/mkdirs through python. I am using the below azure method to generate the access token. I am not sure why I am getting 404 any suggestions?token_credential = DefaultAzureCredential()sc...

Data Engineering
Reply
  • 18659 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Ramya
New Contributor III
  • 3 kudos

Yes that is correct!. It worked. Thanks

  • 3 kudos
3 More Replies
Dineshkumar_Raj
by • New Contributor
  • 2912 Views
  • 2 replies
  • 1 kudos

why the job running time and command execution time not matching in databricks

I have a azure databricks job and it's triggered via ADF using a API call. I want see why the job has been taking n minutes to complete the tasks. When the job execution results, The job execution time says 15 mins and the individual cells/commands d...

Data Engineering
Reply
  • 2912 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey there @DineshKumar​ Does @Prabakar Ammeappin​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help. Cheers!

  • 1 kudos
1 More Replies
abaschkim
by • New Contributor II
  • 2655 Views
  • 4 replies
  • 0 kudos

Delta Lake table: large volume due to versioning

I have set up a Spark standalone cluster and use Spark Structured Streaming to write data from Kafka to multiple Delta Lake tables - simply stored in the file system. So there are multiple writes per second. After running the pipeline for a while, I ...

Data Engineering
Reply
  • 2655 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hey there @Kim Abasch​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
3 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels