cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
robbe
by • New Contributor III
  • 304 Views
  • 2 replies
  • 1 kudos

Resolved! Get job ID from Asset Bundles

When using Asset Bundles to deploy jobs, how does one get the job ID of the resources that are created?I would like to deploy some jobs through asset bundles, get the job IDs, and then trigger these jobs programmatically outside the CI/CD pipeline us...

Data Engineering
Reply
  • 304 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
robbe
New Contributor III
  • 1 kudos

Thanks @mhiltner. I don't need to run jobs, just to get the ID. So I think that solution 2) is the way to go here. I'll accept the solution.

  • 1 kudos
1 More Replies
Eiki
by • New Contributor
  • 48 Views
  • 1 replies
  • 0 kudos

How to use the same job cluster in diferents job runs inside the one workflow

I created a Workflow with notebooks and some job runs, but I would to use only one job cluster to run every job runs, without creating a new job cluster for each job run. Because I didn't want to increase the execution time with each new job cluster ...

Data Engineering
Reply
  • 48 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brockb
Contributor III
  • 0 kudos

Hi,If I understand correctly, you are hoping to reduce overall job execution time by reducing the Cloud Service Provider instance provisioning time. Is that correct?If so, you may want to consider: Using a Pool of instances: https://docs.databricks.c...

  • 0 kudos
diego_poggioli
by • Contributor
  • 139 Views
  • 1 replies
  • 0 kudos

Streaming foreachBatch _jdf jvm attribute not supported

I'm trying to perform a merge inside a streaming foreachbatch using the command: microBatchDF._jdf.sparkSession().sql(self.merge_query)Streaming runs fine if I use a Personal cluster while if I use a Shared cluster streaming fails with the following ...

Data Engineering
Reply
  • 139 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
holly
New Contributor III
  • 0 kudos

Can you share what runtime your cluster is using?  This error doesn't surprise me, Unity Catalog Shared clusters have many security limitations, but the list is reducing over time. https://docs.databricks.com/en/compute/access-mode-limitations.html#s...

  • 0 kudos
Kayla
by • Contributor II
  • 116 Views
  • 2 replies
  • 0 kudos

Datadog Installation

Is anyone familiar with installing the Datadog agent on clusters? We're not having much luck.  We honestly might not be having the init script run since we're not seeing it in the log, but we can get just a generic "hellow world" init script to run a...

Data Engineering
Reply
  • 116 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Valued Contributor III
  • 0 kudos

Is there any specific error you are receiving when running the init script? Does the run complete start up or fail due to the init script?

  • 0 kudos
1 More Replies
nehaa
by • New Contributor II
  • 106 Views
  • 2 replies
  • 0 kudos

Databricks dashboard publish

How to publish the dashboard created from a notebook? I don't see publish option within the File anymoreWhen referred to the old video they seem to have an option to publish the dashboard  

nehaa_0-1718822255560.png nehaa_2-1718822335195.png
Data Engineering
Reply
  • 106 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Valued Contributor III
  • 0 kudos

Can you share the link to the video you are referring?As per docs, no publish option is currently available, you can do a Present Dashboard to see it.https://docs.databricks.com/en/notebooks/dashboards.html

  • 0 kudos
1 More Replies
jenitjain
by • New Contributor
  • 129 Views
  • 2 replies
  • 0 kudos

Certifications questions

What are the timings and days between which we can get certified? Can we purchase a certification at the location or are we supposed to purchase it beforehand?

Data Engineering
Reply
  • 129 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Cert-Team
Honored Contributor III
  • 0 kudos

Online exams can be purchased and taken anytime. Is this question related to DAIS?

  • 0 kudos
1 More Replies
manish1987c
by • New Contributor III
  • 80 Views
  • 1 replies
  • 0 kudos

Python project || write write_micro_batch in structure streaming

Hi Team,We are in progress of developing a framework in python using datatbricks-connect in VSO,However when we are trying to run micro batches in the foreachbatch function it is giving us error message saying that "few objects are not serializable: ...

Data Engineering
Reply
  • 80 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @manish1987c,  Ensure that the Python version you’re using locally matches the one on your Databricks cluster. Minor version differences are usually acceptable (e.g., 3.10.11 versus 3.10.10).

  • 0 kudos
swathiG
by • New Contributor
  • 369 Views
  • 7 replies
  • 1 kudos

Databrick IP address

I'm trying to call api in databricks notebook .But while calling  api inside databricks notebook  it is giving error saying "403" forbidden. I think it seems issue with IP address can any one help me to know which IP of databricks need to be whitelis...

swathiG_0-1718859645301.png
Data Engineering
Reply
  • 369 Views
  • 7 replies
  • 1 kudos
View Replies
Latest Reply
swathiG
New Contributor
  • 1 kudos

@jacovangelder can you please let me know where can I get compute IP address

  • 1 kudos
6 More Replies
ckarrasexo
by • New Contributor II
  • 561 Views
  • 3 replies
  • 0 kudos

pyspark.sql.connect.dataframe.DataFrame vs pyspark.sql.DataFrame

I noticed that on some Databricks 14.3 clusters, I get DataFrames with type pyspark.sql.connect.dataframe.DataFrame, while on other clusters also with Databricks 14.3, the exact same code gets DataFrames of type pyspark.sql.DataFramepyspark.sql.conne...

Data Engineering
Reply
  • 561 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
satniks
New Contributor
  • 0 kudos

I am also facing the issue due that the spark session using databricks connect returns me pyspark.sql.connect.dataframe.DataFrame whereas the data processing library I am using expects pyspark.sql.DataFrame.Is it possible to convert pyspark.sql.conne...

  • 0 kudos
2 More Replies
mysteryuser000
by • New Contributor
  • 76 Views
  • 1 replies
  • 0 kudos

dlt pipeline will not create live tables

I have created a dlt pipeline based four sql notebooks, each containing between 1 and 3 queries.  Each query begins with "create or refresh live table ..." yet each one outputs a materialized view.  I have tried deleting the materialized views and ru...

Data Engineering
Reply
  • 76 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @mysteryuser000,  In Databricks SQL, materialized views are Unity Catalog managed tables that allow you to precompute results based on the latest data in source tables. Unlike other implementations, the results returned reflect the state of data w...

  • 0 kudos
laudhon
by • New Contributor II
  • 323 Views
  • 6 replies
  • 3 kudos

Why is My MIN MAX Query Still Slow on a 29TB Delta Table After Liquid Clustering and Optimization?

Hello,I have a large Delta table with a size of 29TB. I implemented Liquid Clustering on this table, but running a simple MIN MAX query on the set cluster column is still extremely slow. I have already optimized the table. Am I missing something in m...

Data Engineering
Reply
  • 323 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
LuisRSanchez
New Contributor III
  • 3 kudos

Hithis operation should take seconds because it use the precomputed statistics for the table. Then few elements to verify:if the data type is datetime or integer should work, if it is string data type then it needs to read all data.verify the column ...

  • 3 kudos
5 More Replies
dat_77
by • New Contributor
  • 102 Views
  • 1 replies
  • 0 kudos

Change Default Parallelism ?

HII attempted to parallelize my Spark read process by setting the default parallelism using spark.conf.set("spark.default.parallelism", "X"). However, despite setting this configuration, when I checked sc.defaultParallelism in my notebook, it display...

Data Engineering
Reply
  • 102 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
irfan_elahi
New Contributor II
  • 0 kudos

sc.defaultParallelism is based on the number of worker cores in the cluster. It can't be overridden. The reason you are seeing 200 tasks is because of spark.sql.shuffle.partitions (whose default value is 200). This determines the number of shuffle pa...

  • 0 kudos
Awoke101
by • New Contributor II
  • 261 Views
  • 8 replies
  • 0 kudos

Resolved! Pandas_UDF not working on shared access mode but works on personal cluster

 The "dense_vector" column does not output on show(). Instead I get the error below. Any idea why it doesn't work on the shared access mode? Any alternatives? from fastembed import TextEmbedding, SparseTextEmbedding from pyspark.sql.pandas.functions ...

Data Engineering
pandas_udf
shared_access
udf
Reply
  • 261 Views
  • 8 replies
  • 0 kudos
View Replies
Latest Reply
jacovangelder
Contributor III
  • 0 kudos

For some reason a moderator is removing my pip freeze? no idea why. Maybe too long/spammy for a comment.Anyway, I am using DBR 14.3 LTS with Shared Access Mode. I haven't installed any other version apart from fastembed==0.3.1. Included a screenshot ...

  • 0 kudos
7 More Replies
DavidS1
by • New Contributor
  • 92 Views
  • 1 replies
  • 0 kudos

Cost comparison of DLT to custom pipeline

Hello, our company currently has a number of custom pipelines written in python for ETL, and I want to do an evaluation of DLT to see if that will make things more efficient.  A problem is that there is a restriction on using DLT "because it is too e...

Data Engineering
Reply
  • 92 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Zume
New Contributor II
  • 0 kudos

DLT is expensive in my opinion. I tried to run a simple notebook that just reads a parquet file into a dataframe and write it out to a cloud storage and I got an error that i hit my CPU instance limit for my azure subscription. I just gave up after t...

  • 0 kudos
dbx_687_3__1b3Q
by • New Contributor III
  • 4096 Views
  • 5 replies
  • 4 kudos

Resolved! Databricks Asset Bundle (DAB) from a Git repo?

My earlier question was about creating a Databricks Asset Bundle (DAB) from an existing workspace. I was able to get that working but after further consideration and some experimenting, I need to alter my question. My question is now "how do I create...

Data Engineering
Reply
  • 4096 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
nicole_lu_PM
New Contributor III
  • 4 kudos

We are very close to having an end-to-end solution for deploying DABs from a Git folder (Repo) in the Workspace! Check out my talk on DAIS24 here https://github.com/databricks/dais-cow-bff (video link on README). We are waiting for the feature that a...

  • 4 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all