cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
cpd
by • New Contributor II
  • 2502 Views
  • 1 replies
  • 0 kudos

Ingesting geospatial data into a table

I'm just getting started with Databricks and wondering if it is possible to ingest a GeoJSON or GeoParquet file into a new table without writing code? My goal here is to load vector data into a table and perform H3 polyfill operations on all the vect...

Data Engineering
Reply
  • 2502 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
cpd
New Contributor II
  • 0 kudos

Thank you @Retired_mod - much appreciated!

  • 0 kudos
ashish577
by • New Contributor III
  • 1982 Views
  • 0 replies
  • 0 kudos

How do we pass parameters which have a "," with bundle run ?

So I have a query "select col1, col2 from table" that I need to pass as a parameter to a databricks job that I am triggering through the bundle run command. Issue is, when I pass this via --params="query=select col1, col2 from table" it splits it bas...

Data Engineering
Reply
  • 1982 Views
  • 0 replies
  • 0 kudos
Brichj
by • New Contributor II
  • 1827 Views
  • 2 replies
  • 0 kudos

%run ../Includes/Classroom-Setup-02.1

I ran the code in the cell as it was given in the presentation. But it failed. Can someone please help?The presentation is the second lesson in the second model of Data Engineering Associate exam prep.

Data Engineering
Reply
  • 1827 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Brichj
New Contributor II
  • 0 kudos

Thanks Ajay-Pandey!This is error that I keep getting when I run the following: %run ./Includes/Classroom-Setup-02.3LI have run dbutils.library.restartPython(), but it did not help.Note: you may need to restart the kernel using dbutils.library.restart...

  • 0 kudos
1 More Replies
Brad
by • Contributor II
  • 1987 Views
  • 3 replies
  • 0 kudos

Inconsistent behavior when displaying chart in notebook

Hi, I'm trying to create some 3D charts. With the same code and same cluster, sometimes it can show, sometimes it cannot. Previously it cannot display, but last week I opened a notebook with failed run and found the result can be shown by itself (as ...

Data Engineering
Reply
  • 1987 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Brad
Contributor II
  • 0 kudos

Also, with same code, same browser, different workspaces, one works, other one not. In the notebook with "script error", if I "Export cell" and get its iframe html and use displayHTML to display it, it works, so this means the JS and HTML inside is o...

  • 0 kudos
2 More Replies
pokus
by • New Contributor III
  • 7333 Views
  • 2 replies
  • 2 kudos

Resolved! use DeltaLog class in databricks cluster

I need to use DeltaLog class in the code to get the AddFiles dataset. I have to keep the implemented code in a repo and run it in databricks cluster. Some docs say to use org.apache.spark.sql.delta.DeltaLog class, but it seems databricks gets rid of ...

Data Engineering
Reply
  • 7333 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
dbal
New Contributor III
  • 2 kudos

Thanks for providing a solution @pokus .What I dont understand is why Databricks cannot provide the DeltaLog at runtime. How can this be the official solution? We need a better solution for this instead of depending on reflections.

  • 2 kudos
1 More Replies
VGS777
by • New Contributor III
  • 762 Views
  • 0 replies
  • 0 kudos

Regarding Cloning dbc file from git

Hey FolksI have dbc file in a git repo and i cloned in the databricks when tried to open the .dbc file it is saying ```Failed to load file. The file encoding is not supported```can anyone please advice me on this #help #beginner

Data Engineering
Reply
  • 762 Views
  • 0 replies
  • 0 kudos
RobsonNLPT
by • Contributor II
  • 2031 Views
  • 2 replies
  • 0 kudos

Resolved! scala-xml : how to move child to another parent node

Hi allThe mandatory rowTag for writing to XML cause doesn't make any sense as I have the complete nested dataframe schema.In my case I need to implement an extra step to remove that extra node (default: Row) after xml generation.I need some examples ...

Data Engineering
Reply
  • 2031 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi allThe mandatory rowTag for writing to XML cause doesn't make any sense as I have the complete nested dataframe schema.In my case I need to implement an extra step to remove that extra node (default: Row) after xml generation.I need some examples ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
LoiNguyen
by • New Contributor II
  • 15506 Views
  • 5 replies
  • 2 kudos

The authentication type 10 is not supported

I use below code to connect to postgresql. df = spark.read \ .jdbc("jdbc:postgresql://hostname:5432/dbname", "schema.table", properties={"user": "user", "password": "password"})\ .load() df.printSchema() However, I got the ...

Data Engineering
Reply
  • 15506 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
simboss
New Contributor II
  • 2 kudos

But how are we going to do this for those who use Windows?

  • 2 kudos
4 More Replies
PassionateDBD
by • New Contributor II
  • 909 Views
  • 1 replies
  • 1 kudos

Is it possible to create/update non dlt table in init phase of dlt task?

We have a dlt task that is written in python. Is it possible to create or update a delta table programatically from inside a dlt task? The delta table would not be managed from inside the dlt task because we never want to fully refresh that table. Th...

Data Engineering
Reply
  • 909 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
PassionateDBD
New Contributor II
  • 1 kudos

Thanks for you reply @Retired_mod ! I'm aware of the possibility to create or not create a table based on some parameter.What I'm trying to figure out is basically how to achieve following:-DLT pipeline starts and logs some information to a delta tab...

  • 1 kudos
Hertz
by • New Contributor II
  • 1249 Views
  • 0 replies
  • 0 kudos

Structured Streaming Event in Audit Logs

I am trying to monitor when a table is created or updated using the audit logs. I have found that structured streaming writes/appends are not captured in the audit logs? Am I missing something shouldn't this be captured as a unity catalog event. Eith...

Data Engineering
Audit Logs
structured streaming
Reply
  • 1249 Views
  • 0 replies
  • 0 kudos
Floody
by • New Contributor II
  • 1218 Views
  • 0 replies
  • 0 kudos

Delta Live Tables use case

Hi all,We have the following use case and wondering if DLT is the correct approach.Landing area with daily dumps of parquet files into our Data Lake container.The daily dump does a full overwrite of the parquet each time, keeping the same file name.T...

Data Engineering
Delta Live Tables
Reply
  • 1218 Views
  • 0 replies
  • 0 kudos
srinivas_001
by • New Contributor III
  • 945 Views
  • 1 replies
  • 1 kudos

File trigger options -- cloudFiles.allowOverwrites

I have a Job configured to run on the file arrival I have provided the path as File arrival path: s3://test_bucket/test_cat/test_schema/When a new parquet file arrived in this path the job was triggering automatically and processed the fileIn case of...

Data Engineering
Reply
  • 945 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
srinivas_001
New Contributor III
  • 1 kudos

Hi Kaniz,Thank you for the response.I am using the databricks runtime 11.3, also checked the checkpoint and data source location which are properly configured. Still I am unable to trigger the job.NOTE: Incoming files are pushed to AWS s3 location fr...

  • 1 kudos
Nisha2
by • New Contributor II
  • 1452 Views
  • 1 replies
  • 0 kudos

Databricks spark_jar_task failed when submitted via API

Hello,We are submitting jobs to the data bricks cluster using  /api/2.0/jobs/create this API and running a spark java application (jar that is submitted to this API). We are noticing Java application is executing as expected. however, we see that the...

Data Engineering
API
Databricks
spark
Reply
  • 1452 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello,We are submitting jobs to the data bricks cluster using  /api/2.0/jobs/create this API and running a spark java application (jar that is submitted to this API). We are noticing Java application is executing as expected. however, we see that the...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
Pragati_17
by • New Contributor II
  • 2149 Views
  • 0 replies
  • 0 kudos

Parameters Passing to dataset in Databricks Lakeview Dashboard

I have a date range filter in Lakeview Dashboard and i want to distinct count number of months in selected date range filter and divide it with one of the columns and that column is used in counter viualization. But passing parameters is not possible...

Data Engineering
Reply
  • 2149 Views
  • 0 replies
  • 0 kudos
ElaPG
by • New Contributor III
  • 2071 Views
  • 1 replies
  • 1 kudos

Cluster creation / unrestricted policy option

Hi,as an workspace admin I would like to disable cluster creation with "no isolation" access mode. I created a custom policy for that but I still have the option to create cluster with "unrestricted" policy. How can I make sure that nobody will creat...

Data Engineering
Reply
  • 2071 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
ElaPG
New Contributor III
  • 1 kudos

Hi,thank you for a very informative reply.To sum up, in order to enforce these suggestions:- first solution must be executed on an account level- second solution must be executed on a workspace level (workspace level admin settings)

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels