cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
pokus
by • New Contributor III
  • 7414 Views
  • 2 replies
  • 2 kudos

Resolved! use DeltaLog class in databricks cluster

I need to use DeltaLog class in the code to get the AddFiles dataset. I have to keep the implemented code in a repo and run it in databricks cluster. Some docs say to use org.apache.spark.sql.delta.DeltaLog class, but it seems databricks gets rid of ...

Data Engineering
Reply
  • 7414 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
dbal
New Contributor III
  • 2 kudos

Thanks for providing a solution @pokus .What I dont understand is why Databricks cannot provide the DeltaLog at runtime. How can this be the official solution? We need a better solution for this instead of depending on reflections.

  • 2 kudos
1 More Replies
VGS777
by • New Contributor III
  • 765 Views
  • 0 replies
  • 0 kudos

Regarding Cloning dbc file from git

Hey FolksI have dbc file in a git repo and i cloned in the databricks when tried to open the .dbc file it is saying ```Failed to load file. The file encoding is not supported```can anyone please advice me on this #help #beginner

Data Engineering
Reply
  • 765 Views
  • 0 replies
  • 0 kudos
RobsonNLPT
by • Contributor II
  • 2034 Views
  • 2 replies
  • 0 kudos

Resolved! scala-xml : how to move child to another parent node

Hi allThe mandatory rowTag for writing to XML cause doesn't make any sense as I have the complete nested dataframe schema.In my case I need to implement an extra step to remove that extra node (default: Row) after xml generation.I need some examples ...

Data Engineering
Reply
  • 2034 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi allThe mandatory rowTag for writing to XML cause doesn't make any sense as I have the complete nested dataframe schema.In my case I need to implement an extra step to remove that extra node (default: Row) after xml generation.I need some examples ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
LoiNguyen
by • New Contributor II
  • 15652 Views
  • 5 replies
  • 2 kudos

The authentication type 10 is not supported

I use below code to connect to postgresql. df = spark.read \ .jdbc("jdbc:postgresql://hostname:5432/dbname", "schema.table", properties={"user": "user", "password": "password"})\ .load() df.printSchema() However, I got the ...

Data Engineering
Reply
  • 15652 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
simboss
New Contributor II
  • 2 kudos

But how are we going to do this for those who use Windows?

  • 2 kudos
4 More Replies
PassionateDBD
by • New Contributor II
  • 917 Views
  • 1 replies
  • 1 kudos

Is it possible to create/update non dlt table in init phase of dlt task?

We have a dlt task that is written in python. Is it possible to create or update a delta table programatically from inside a dlt task? The delta table would not be managed from inside the dlt task because we never want to fully refresh that table. Th...

Data Engineering
Reply
  • 917 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
PassionateDBD
New Contributor II
  • 1 kudos

Thanks for you reply @Retired_mod ! I'm aware of the possibility to create or not create a table based on some parameter.What I'm trying to figure out is basically how to achieve following:-DLT pipeline starts and logs some information to a delta tab...

  • 1 kudos
Hertz
by • New Contributor II
  • 1260 Views
  • 0 replies
  • 0 kudos

Structured Streaming Event in Audit Logs

I am trying to monitor when a table is created or updated using the audit logs. I have found that structured streaming writes/appends are not captured in the audit logs? Am I missing something shouldn't this be captured as a unity catalog event. Eith...

Data Engineering
Audit Logs
structured streaming
Reply
  • 1260 Views
  • 0 replies
  • 0 kudos
srinivas_001
by • New Contributor III
  • 955 Views
  • 1 replies
  • 1 kudos

File trigger options -- cloudFiles.allowOverwrites

I have a Job configured to run on the file arrival I have provided the path as File arrival path: s3://test_bucket/test_cat/test_schema/When a new parquet file arrived in this path the job was triggering automatically and processed the fileIn case of...

Data Engineering
Reply
  • 955 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
srinivas_001
New Contributor III
  • 1 kudos

Hi Kaniz,Thank you for the response.I am using the databricks runtime 11.3, also checked the checkpoint and data source location which are properly configured. Still I am unable to trigger the job.NOTE: Incoming files are pushed to AWS s3 location fr...

  • 1 kudos
Nisha2
by • New Contributor II
  • 1464 Views
  • 1 replies
  • 0 kudos

Databricks spark_jar_task failed when submitted via API

Hello,We are submitting jobs to the data bricks cluster using  /api/2.0/jobs/create this API and running a spark java application (jar that is submitted to this API). We are noticing Java application is executing as expected. however, we see that the...

Data Engineering
API
Databricks
spark
Reply
  • 1464 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello,We are submitting jobs to the data bricks cluster using  /api/2.0/jobs/create this API and running a spark java application (jar that is submitted to this API). We are noticing Java application is executing as expected. however, we see that the...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
Pragati_17
by • New Contributor II
  • 2167 Views
  • 0 replies
  • 0 kudos

Parameters Passing to dataset in Databricks Lakeview Dashboard

I have a date range filter in Lakeview Dashboard and i want to distinct count number of months in selected date range filter and divide it with one of the columns and that column is used in counter viualization. But passing parameters is not possible...

Data Engineering
Reply
  • 2167 Views
  • 0 replies
  • 0 kudos
ElaPG
by • New Contributor III
  • 2101 Views
  • 1 replies
  • 1 kudos

Cluster creation / unrestricted policy option

Hi,as an workspace admin I would like to disable cluster creation with "no isolation" access mode. I created a custom policy for that but I still have the option to create cluster with "unrestricted" policy. How can I make sure that nobody will creat...

Data Engineering
Reply
  • 2101 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
ElaPG
New Contributor III
  • 1 kudos

Hi,thank you for a very informative reply.To sum up, in order to enforce these suggestions:- first solution must be executed on an account level- second solution must be executed on a workspace level (workspace level admin settings)

  • 1 kudos
Gilg
by • Contributor II
  • 1489 Views
  • 0 replies
  • 0 kudos

DLT Performance

Hi,Context:I have created a Delta Live Table pipeline in a UC enabled workspace that is set to Continuous.Within this pipeline,I have bronze which uses Autoloader and reads files stored in ADLS Gen2 storage account in a JSON file format. We received ...

Data Engineering
Reply
  • 1489 Views
  • 0 replies
  • 0 kudos
William_Scardua
by • Valued Contributor
  • 20065 Views
  • 3 replies
  • 0 kudos

How to estimate dataframe size in bytes ?

How guys,How do I estimate the size in bytes from my dataframe (pyspark) ?Have any ideia ?Thank you

Data Engineering
Reply
  • 20065 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Enneagram1w2
New Contributor II
  • 0 kudos

Unveil the Enneagram 1w9 mix: merging Type 1&#x27;s perfectionism with Type 9&#x27;s calm. Explore their key traits, hurdles, and development path.  https://www.enneagramzoom.com/EnneagramTypes/EnneagramType1/Enneagram1w2

  • 0 kudos
2 More Replies
Abdul1
by • New Contributor
  • 1076 Views
  • 1 replies
  • 0 kudos

How to output data from Databricks?

Hello,I am just starting with Databricks in Azure and I need to output the data to an Affinity CRM system.Affinity has an API and I am wondering is there any sort of automated / data pipeline sort of way to tell databricks to just pump the data into ...

Data Engineering
Reply
  • 1076 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Edthehead
Contributor II
  • 0 kudos

We need more info on what kind of data, volume and what the called APi can handle. Calling an API for single records in parallel can be achieved using UDF(see THIS). You need to be careful to batch the records so that the target API can handle the pa...

  • 0 kudos
george_ognyanov
by • New Contributor III
  • 897 Views
  • 1 replies
  • 1 kudos

Orchestrate jobs using a parameter set in a notebook

I am trying to orchestrate my Databricks Workflows tasks using a parameter I would set in a notebook.Given the workflow below I am trying to set a parameter in the Cinderella task which is a python notebook. Once set I would like to use this paramete...

george_ognyanov_0-1701258224857.png george_ognyanov_0-1701259276034.png
Data Engineering
Reply
  • 897 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Panda
Valued Contributor
  • 1 kudos

Here's how we can proceed, follow the instructions below:In your previous task, depending on whether you're using Python or Scala, set the task value like this:dbutils.jobs.taskValues.set("check_value", "2")In your if-else task, you must reference th...

  • 1 kudos
tyas
by • New Contributor II
  • 1800 Views
  • 1 replies
  • 1 kudos

Defining Keys

Hello,I have a DataFrame in a Databricks notebook that I've already read and transformed using PySpark-Python. I want to create a table with defined keys (primary and foreign). What is the best method to do this:Create a table and directly define key...

Data Engineering
Reply
  • 1800 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Remember that keys are for information purposes (they don't validate data integrity). They are used for information in a few places (Feature tables, online tables, PowerBi modelling). The best is to define them in CREATE TABLE syntax, for example:CRE...

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels