cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

giladba
by New Contributor III
  • 2374 Views
  • 9 replies
  • 4 kudos

access to event_log TVF

Hi, According to the documentation:https://docs.databricks.com/en/delta-live-tables/observability.html"The event_log TVF can be called only by the pipeline owner and a view created over the event_log TVF can be queried only by the pipeline owner. The...

  • 2374 Views
  • 9 replies
  • 4 kudos
Latest Reply
hcjp
New Contributor II
  • 4 kudos

As per this documentation, https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/unity-catalog, the issue here is documented as a current Limitation:LimitationsThe following are limitations when using Unity Catalog with Delta Live Tabl...

  • 4 kudos
8 More Replies
Babu_Krishnan
by Contributor
  • 37 Views
  • 1 replies
  • 0 kudos

Why my DLT is not working with UC?

My IAM profile is not working when accessing the SQS for file notification based ingestion?

  • 37 Views
  • 1 replies
  • 0 kudos
Latest Reply
jacovangelder
Contributor II
  • 0 kudos

I'm not sure if I fully understand the question, but what location are you monitoring? Is it a DBFS path or mount? If so, consider using a UC Volume. 

  • 0 kudos
cpayne_vax
by New Contributor III
  • 5104 Views
  • 7 replies
  • 3 kudos

Resolved! Delta Live Tables: dynamic schema

Does anyone know if there's a way to specify an alternate Unity schema in a DLT workflow using the @Dlt.table syntax? In my case, I’m looping through folders in Azure datalake storage to ingest data. I’d like those folders to get created in different...

  • 5104 Views
  • 7 replies
  • 3 kudos
Latest Reply
manish1987c
New Contributor III
  • 3 kudos

Please let me know if there is way by which we can define schema dynamically in 1 delta live table pipeline in same catalog  as below  For example:Catalog1 : schemaA : tableACatalog1 : schemaB : tableB  

  • 3 kudos
6 More Replies
stevenayers-bge
by New Contributor III
  • 262 Views
  • 2 replies
  • 1 kudos

Bug with enabling UniForm Data Format?

In the documentation for enabling iceberg compatibility on delta tables, it states that the minReaderVersion for IcebergCompatV1 and IcebergCompatV2 is 2 (https://docs.databricks.com/en/delta/uniform.html#requirements).However, when you run the REORG...

  • 262 Views
  • 2 replies
  • 1 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 1 kudos

@stevenayers-bge I've just checked source code of delta and you're right - documentation states that tat minReaderVersion should be >=2, but source code is upgrading it to 3https://github.com/delta-io/delta/blob/78970abd96dfc0278e21c04cda442bb05ccde4...

  • 1 kudos
1 More Replies
jyothib
by New Contributor
  • 58 Views
  • 1 replies
  • 1 kudos

Resolved! System tables latency

How much time is the latency of system tables#unitycatalog

  • 58 Views
  • 1 replies
  • 1 kudos
Latest Reply
raphaelblg
Contributor III
  • 1 kudos

@jyothib at the current moment, system tables are still under Public Preview stage (more details at: https://docs.databricks.com/en/admin/system-tables/index.html)We don’t offer data freshness SLOs for system tables at this point and there are no pla...

  • 1 kudos
MaximeGendre
by New Contributor II
  • 60 Views
  • 1 replies
  • 0 kudos

How to disable DBFS storage

Hello,I administer a self-service oriented Databricks workspace and I notice that more and more users are storing their data in DBFS due to lack of knowledge.They are not specifying a location when creating their schema or they are not specifying a s...

MaximeGendre_1-1718227315313.png
  • 60 Views
  • 1 replies
  • 0 kudos
Latest Reply
MaximeGendre
New Contributor II
  • 0 kudos

Replacing "/mnt/adl2" by "dbfs:/mnt/adl2" fixed the issue.

  • 0 kudos
sagarsiddhabha
by New Contributor
  • 23 Views
  • 0 replies
  • 0 kudos

Attended Data +AI summit at SFO

It was great experience attending this conference.Got great insights about new features.Got to know about new advances in data industry.Attending the conference was an enriching and transformative experience. I gained invaluable insights into the lat...

  • 23 Views
  • 0 replies
  • 0 kudos
camilo_s
by New Contributor II
  • 31 Views
  • 0 replies
  • 0 kudos

Git credentials for service principals running Jobs

I know the documentation for setting up Git credentials for Service Principals: you have to use a PAT from your Git provider, which is inevitably tied to an user and has a lifecycle of its own.Doesn't this kind of defeats the purpose of running a job...

  • 31 Views
  • 0 replies
  • 0 kudos
leungi
by New Contributor III
  • 31 Views
  • 0 replies
  • 0 kudos

Init Script Fails Intermittently on Workflow Job

An init script is used to install system libraries, per below.Adding the script to a Personal Compute consistently works. The same script is added to a Workflows job via cluster config, which intermittently fails, as shown in error message below.Both...

leungi_0-1718291897408.png
  • 31 Views
  • 0 replies
  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels