cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

aothman
by New Contributor
  • 102 Views
  • 1 replies
  • 0 kudos

Databricks on AWS Outposts rack

We need to know if Databricks can be implemented on a private cloud infrastructure or on AWS Outposts rack  

  • 102 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @aothman, Databricks currently supports public cloud providers such as Azure, GCP, and AWS, but it does not offer direct support for private cloud implementations beyond these.While Databricks itself does not directly support Outposts, you ca...

  • 0 kudos
hfyhn
by New Contributor
  • 125 Views
  • 1 replies
  • 0 kudos

DLT, combine LIVE table with data masking and row filter

I need to apply data masking and row filters to my table. At the same time I would like to use DLT Live tables. However, as far as I can see, DLT Live tables are not compatble with Live tables. What are my options? Move the tables from out of the mat...

  • 125 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @hfyhn, You’re correct that DLT Live tables don’t inherently support writing data from multiple live tables into a single target Delta table. This means you cannot directly configure two or more DLT pipelines to update the same table simultaneousl...

  • 0 kudos
FrankTa
by New Contributor II
  • 456 Views
  • 2 replies
  • 2 kudos

Resolved! Unstable workflow runs lately

Hi!We are using Databricks on Azure on production since about 3 months. A big part of what we use Databricks for is processing data using a workflow with various Python notebooks. We run the workflow on a 'Pools' cluster and on a 'All-purpose compute...

  • 456 Views
  • 2 replies
  • 2 kudos
Latest Reply
FrankTa
New Contributor II
  • 2 kudos

Hi holly,Thanks for your reply, good to hear that the 403 errors are on the radar and due to be fixed. I will reach out to support in case of further issues.

  • 2 kudos
1 More Replies
PaulMarsh
by New Contributor II
  • 180 Views
  • 2 replies
  • 0 kudos

Python databricks sql.connect takes 10minutes exact to connect to Serverless SQL Datawarehouse

Have tried this with both the SQL Warehouse up and running and turned off. The connection eventually is made and works without issue, but the initial connect() takes 10minutes to establish. No other connectivity issues between the box and the Warehou...

  • 180 Views
  • 2 replies
  • 0 kudos
Latest Reply
PaulMarsh
New Contributor II
  • 0 kudos

@Kaniz_Fatma thank you for the reply, it seems the issue came down to the script was running with Python 3.12 and I should have been using 3.11.9 

  • 0 kudos
1 More Replies
lprevost
by New Contributor III
  • 558 Views
  • 7 replies
  • 2 kudos

Incremental Loads from a Catalog/DLT

The databricks guide outlines several use cases for loading data via delta live tables:https://docs.databricks.com/en/delta-live-tables/load.html This includes autoloader from cloudfiles, kafka messages, small static datasets, etc.But, one use case i...

  • 558 Views
  • 7 replies
  • 2 kudos
Latest Reply
lucasrocha
New Contributor III
  • 2 kudos

Hello @lprevost , I hope this message finds you well. In order to install this library, you can follow any of the steps below:- First step - search for libraries and versions: Go the cluster;Libraries;Install new;Maven;Search Packages;Change from Spa...

  • 2 kudos
6 More Replies
ameya
by New Contributor
  • 156 Views
  • 1 replies
  • 0 kudos

Adding new columns to a Delta Live table in a CDC process

Hi I am new to databricks and still learning.I am trying to do a CDC on a table.  APPLY CHANGES INTO LIVE.table1 FROM schema2.table2 KEYS (Id) SEQUENCE BY orderByColumn COLUMNS * EXCEPT (col1, col2) STORED AS SCD TYPE 1 ;  table1 is in schema1 and ...

  • 156 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Honored Contributor
  • 0 kudos

Hi @ameya , Scenario 1: Enabling Delta schema evolution in your table or at DLT pipeline level should suffice for the scenario of new fields being added to the schema.  Scenario 2: The INSERT statement doesn't support schema evolution as described in...

  • 0 kudos
NarenderKumar
by New Contributor III
  • 153 Views
  • 2 replies
  • 2 kudos

Resolved! Can we parameterize the compute in job cluster

I have created a workflow job in databricks with job parameters.I want to run the job same with different workloads and data volume.So I want the compute cluster to be parametrized so that I can pass the compute requirements(driver, executor size and...

  • 153 Views
  • 2 replies
  • 2 kudos
Latest Reply
brockb
Valued Contributor
  • 2 kudos

Hi @NarenderKumar , Have you considered leveraging autoscaling for the existing cluster?If this does not meet your needs, are the differing volume/workloads known in advance? If so, could different compute be provisioned using Infrastructure as Code ...

  • 2 kudos
1 More Replies
yusufd
by New Contributor III
  • 1501 Views
  • 9 replies
  • 8 kudos

Resolved! Pyspark serialization

Hi,I was looking for comprehensive documentation on implementing serialization in pyspark, most of the places I have seen is all about serialization with scala. Could you point out where I can get a detailed explanation on it?

  • 1501 Views
  • 9 replies
  • 8 kudos
Latest Reply
yusufd
New Contributor III
  • 8 kudos

Thank you @Kaniz_Fatma  for the prompt reply. This clears the things and also distinguishes between spark-scala and pyspark. Appreciate your explanation. Will apply this and also share any findings based on this which will help the community!

  • 8 kudos
8 More Replies
tseader
by New Contributor II
  • 362 Views
  • 3 replies
  • 1 kudos

Resolved! Python SDK clusters.create_and_wait - Sourcing from cluster-create JSON

I am attempting to create a compute cluster using the Python SDK while sourcing a cluster-create configuration JSON file, which is how it's done for the databricks-cli and what databricks provides through the GUI.  Reading in the JSON as a Dict fails...

  • 362 Views
  • 3 replies
  • 1 kudos
Latest Reply
tseader
New Contributor II
  • 1 kudos

@Kaniz_Fatma The structure of the `cluster-create.json` is perfectly fine.  The issue is as stated above related to the structure is that the SDK does not allow nested structures from the JSON file to be used, and instead they need to be cast to spec...

  • 1 kudos
2 More Replies
jacovangelder
by Contributor III
  • 161 Views
  • 2 replies
  • 2 kudos

How do you define PyPi libraries on job level in Asset Bundles?

Hello,Reading the documentation, it does not state it is possible to define libraries on job level instead of on task level. It feels really counter-intuitive putting libraries on task level in Databricks workflows provisioned by Asset Bundles. Is th...

  • 161 Views
  • 2 replies
  • 2 kudos
Latest Reply
jacovangelder
Contributor III
  • 2 kudos

Thanks @Witold ! Thought so. I decided to go with an init script where I install my dependencies rather than installing libraries. For future reference, this is what it looks like:job_clusters: - job_cluster_key: job_cluster new_cluster: ...

  • 2 kudos
1 More Replies
William_Scardua
by Valued Contributor
  • 502 Views
  • 1 replies
  • 0 kudos

Get Block Size

Hi guys,How can I get the block size ? have any idea ?Thank you

  • 502 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @William_Scardua, To get the total directory size in Databricks, you can use the dbutils.fs.ls command. Here are a few options: To get the size of a specific directory (e.g., /mnt/abc/xyz), you can run the following Scala code: val path = "/mn...

  • 0 kudos
KendraVant
by New Contributor II
  • 10781 Views
  • 7 replies
  • 2 kudos

Resolved! How do I clear all output results in a notebook?

I'm building notebooks for tutorial sessions and I want to clear all the output results from the notebook before distributing it to the participants. This functionality exists in Juypter but I can't find it in Databricks. Any pointers?

  • 10781 Views
  • 7 replies
  • 2 kudos
Latest Reply
holly
Contributor II
  • 2 kudos

Yes! Run >  Clear >  Clear all cell outputs Fun fact, this feature was made ~10 years ago when we realised all our customer demos looked very messy and had lots of spoilers in them!

  • 2 kudos
6 More Replies
kmaley
by New Contributor
  • 151 Views
  • 1 replies
  • 1 kudos

Concurrent append exception - Two streaming sources writing to same record on the delta table

Hi All, I have a scenario where there are 2 streaming sources Stream1( id, col1, col2) and Stream 2( id, col3, col4) and my delta table has columns (id, col1, col2, col3, col4). My requirement is to insert the record into the delta table if the corre...

  • 151 Views
  • 1 replies
  • 1 kudos
Latest Reply
Witold
New Contributor III
  • 1 kudos

I would keep both write operations separate, i.e. they should write in own tables/partitions. In later stages (e.g. silver), you can easily merge them.

  • 1 kudos
kiranpeesa
by New Contributor
  • 254 Views
  • 1 replies
  • 0 kudos

error in notebook execution

Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7f5790c07070>> (for post_run_cell)

  • 254 Views
  • 1 replies
  • 0 kudos
Latest Reply
Witold
New Contributor III
  • 0 kudos

Can you show us your code?

  • 0 kudos
Soma
by Valued Contributor
  • 964 Views
  • 2 replies
  • 2 kudos

node_timeline system table is not producing any output in UC

node_timeline system table is not producing any output in UC.In system tables node_timeline table is not having any data .Do we need to enable to any other services for UC

Data Engineering
spark
Unity Catalog
  • 964 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Soma,  The node_timeline system table in Databricks is designed to provide historical observability across your account. However, if you’re not seeing any data in the node_timeline table, there are a few considerations to keep in mind: Unity ...

  • 2 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels