cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Jaris
by • New Contributor III
  • 2359 Views
  • 3 replies
  • 1 kudos

CDC Delta table select using startingVersion on Shared cluster running DBR 14.3 does not work

Hello everyone,We have switched from DBR 13.3 to 14.3 on our Shared development cluster and I am no longer able to run following read from a delta table with CDC enabled:data = ( spark.read.format("delta") .option("readChangeFeed", "true") .op...

Data Engineering
Reply
  • 2359 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Hello everyone,We have switched from DBR 13.3 to 14.3 on our Shared development cluster and I am no longer able to run following read from a delta table with CDC enabled:data = ( spark.read.format("delta") .option("readChangeFeed", "true") .op...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
2 More Replies
m-lopez
by • New Contributor II
  • 4179 Views
  • 1 replies
  • 1 kudos

Connection closed by foreign host

Hi, I am setting up Unity Catalog in my environment, I have a workspace with 2 clusters, the older one (10.4 runtime and non-isolated shared mode) without Unity Catalog configuration and the new one with Unity Catalog enabled (13.3 runtime and shared...

Data Engineering
Reply
  • 4179 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 1 kudos

Hi @m-lopez , hope you are doing well!  Were you able to connect to Kafka before? Also, do you get the same issue if you run the telnet command against port 9092?  As per the error message "Trying 10.40.12.243...Connected to 10.40.12.243. Escape char...

  • 1 kudos
PrData05
by • New Contributor II
  • 3997 Views
  • 5 replies
  • 1 kudos

dbfs rest api - Access denied on dbfs rest api call but access works in dataricks notebook

Hello, We are trying to interact with dbfs to upload files and list files in dbfs directory ( we are tring to upload files in volumes ). Although we have necessary permissions on databricks, we are still getting permission denied when we are making r...

Data Engineering
dbfs api
dbfs rest api
REST API
Reply
  • 3997 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
sgdnp
New Contributor II
  • 1 kudos

I am experiencing the same issue as well. Any ideas about how we can upload to volumes using the api? Tried both the /api/2.0/dbfs/put and the streaming apis (/api/2.0/dbfs/create,/api/2.0/dbfs/add-block,/api/2.0/dbfs/close) but no luck so getting it...

  • 1 kudos
4 More Replies
YevheniiY
by • New Contributor II
  • 10182 Views
  • 1 replies
  • 0 kudos

Empty xml tag

 <ItemMaintenance> <Batch> <BathInfo>info</BathInfo> <Item attr1="tekst" attr2="Tekst2"> <ItemId type="Type" id="id"/> <Dates> <Start>2023-11-09</Start> <End>2024-01-02</End> </Dates> <MoreData> More data </MoreData> <...

Data Engineering
empty xml tag
public preview
XML
xml auto loader
xml read
Reply
  • 10182 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
 <ItemMaintenance> <Batch> <BathInfo>info</BathInfo> <Item attr1="tekst" attr2="Tekst2"> <ItemId type="Type" id="id"/> <Dates> <Start>2023-11-09</Start> <End>2024-01-02</End> </Dates> <MoreData> More data </MoreData> <...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
Learnit
by • New Contributor II
  • 4419 Views
  • 1 replies
  • 0 kudos

Databricks Vs ADLS for reporting.

Hi everyone, I'm a business analyst currently facing a decision on how best to develop a report. I need to choose between using Databricks or Azure Data Lake Storage (ADLS) as the data source on the transformed data in csv or excel file format from d...

Data Engineering
Reply
  • 4419 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Palash01
Valued Contributor
  • 0 kudos

Hey @Learnit I'd be glad to help, thanks for posting your concern. To offer the most effective advice, I might need some additional context about your specific situation as looks like your use case is to create reports (dahboards) using local CSV/exc...

  • 0 kudos
Nixon
by • New Contributor II
  • 584 Views
  • 1 replies
  • 0 kudos

Resonable long running time

Hi there, I have a block of code which can be executed around a month ago within 20 mins. But I came back recently and try to execute it again. It takes over 50 minutes still cannot complete (finally got kick out). Any advise and hints is appreciated...

Data Engineering
Reply
  • 584 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Nixon
New Contributor II
  • 0 kudos

Haha...    Should be "unreasonable long" 

  • 0 kudos
Reza
by • New Contributor III
  • 9638 Views
  • 10 replies
  • 3 kudos

Resolved! How can search in a specific folder in Databricks?

There is a keyword search option in Databricks that searches for a command or word in the entire workspace. How can search for a command in a specific folder or repository?

Data Engineering
Reply
  • 9638 Views
  • 10 replies
  • 3 kudos
View Replies
Latest Reply
nelsoncardenas
New Contributor II
  • 3 kudos

I would not consider this resolved. When I visit a forum, it's to find a solution, and I'm not completely sure if this feature has still not been created in Databricks since 2022. Regardless of that, I have submitted the feature request in Databricks...

  • 3 kudos
9 More Replies
pferreira
by • New Contributor II
  • 2241 Views
  • 2 replies
  • 2 kudos

MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

Im facing an error when updating DBR from 13.3 LTS to 14.3LTSIm using the spark:mongo-spark-connector:10.2.1 and running the following script   connectionString = ****** database = ***** collection = ***** spark = SparkSession \ .builder \ ...

Data Engineering
Reply
  • 2241 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
feiyun0112
Honored Contributor
  • 2 kudos

The following notebook shows you how to read and write data to MongoDB Atlas, https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html 

  • 2 kudos
1 More Replies
vsharma
by • New Contributor
  • 2722 Views
  • 3 replies
  • 0 kudos

Where to move cluster-init scripts after latest message (Storing initialization scripts on DBFS is being deprecated. We recommend using a different storage location)?

Recently Databricks has started showing "Storing initialization scripts on DBFS is being deprecated. We recommend using a different storage location" . Is there an alternative of still keep using DBFS or do we need to move to ABSFS ? I could not find...

Data Engineering
Reply
  • 2722 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
vkeziv
New Contributor II
  • 0 kudos

It is suggested that we can use Workspace but databricks CLI not supporting importing shell scripts, but we can import shell script using web page? 

  • 0 kudos
2 More Replies
bergmaal
by • New Contributor III
  • 1888 Views
  • 2 replies
  • 1 kudos

Workflows 7 second delay between tasks

When you have a job in Workflows with multiple tasks running after one another, there seems to be a consistent 7 seconds delay between execution of the tasks. Or, more precisely, every task has an approximate 7 second overhead before the code actuall...

Data Engineering
delay
overhead
tasks
Workflows
Reply
  • 1888 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
JensH
New Contributor III
  • 1 kudos

Hi @bergmaal , I am experiencing the same issue.My Databricks consultant suggested opening a support ticket as this should not be normal behavior.Did you solve this issue yet?We observed these delays do not seem to occur in workflows that use noteboo...

  • 1 kudos
1 More Replies
NotARobot
by • New Contributor III
  • 1727 Views
  • 1 replies
  • 1 kudos

Delta Live Tables UDFs and Versions

Trying to do a url_decode on a column, which works great in development, but running via DLT fails when trying multiple ways.1. pyspark.sql.functions.url_decode - This is new as of 3.5.0, but isn't supported using whatever version running a DLT pipel...

Data Engineering
Reply
  • 1727 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
NotARobot
New Contributor III
  • 1 kudos

Thanks @Retired_mod, for reference if anybody finds this, the DLT release docs are here: https://docs.databricks.com/en/release-notes/delta-live-tables/index.htmlThis shows which versions are running for CURRENT and PREVIEW channels. In this case, wa...

  • 1 kudos
KrzysztofPrzyso
by • New Contributor III
  • 10505 Views
  • 2 replies
  • 0 kudos

Shared job clusters on Azure Data Factory ADF

Hi Databricks Community,If only possible I would like to use Shared Jobs Cluster on external orchestrator like Azure Data Factory (ADF) or Synapse Workspace.The main reasons for using Shared Job cluster are:reduction of start-up time (<1min vs 5 min ...

Data Engineering
Reply
  • 10505 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
KrzysztofPrzyso
New Contributor III
  • 0 kudos

Hi Sai Kumar,Many thanks for your response.Unfortunately using analytical clusters is not really an option for for me due to cost differences between job clusters and analytical clusters.Job cluster also offer assurance that the latest deployed versi...

  • 0 kudos
1 More Replies
Ian_P
by • New Contributor II
  • 5677 Views
  • 5 replies
  • 1 kudos

Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

Hi there, I am getting this error when trying to use Databricks Runtime 13.1, Shared Mode (We need unity catalog), multimode cluster (this works in single user mode, but we need shared mode): py4j.security.Py4JSecurityException: Method public java.la...

Ian_P_0-1690531566535.png
Data Engineering
Databricks
spark
Unity Catalog
Reply
  • 5677 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Ian_P
New Contributor II
  • 1 kudos

@Ayushi_Suthar @Yulei After chatting to databricks support, it seems this behaviour is very intentional and there is no work around since the security around Unity Catalog is strict and necessary. We are just using single user cluster. RegardsIan

  • 1 kudos
4 More Replies
prasad95
by • New Contributor III
  • 8596 Views
  • 3 replies
  • 1 kudos

How to Capture Change Data (CDC) from DynamoDB Streams and Write into Delta table of DataBricks

Can anyone give me Steps to achieve this.

Data Engineering
Delta Lake
Reply
  • 8596 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
saikumar246
Databricks Employee
  • 1 kudos

Hi, @prasad95 Thank you for sharing your concern here.  In addition to the @Retired_mod comments you can follow below To capture Change Data (CDC) from DynamoDB Streams and write it into a Delta table in Databricks: 1. Connect to DynamoDB Streams and...

  • 1 kudos
2 More Replies
User16752245312
by • Databricks Employee
  • 3287 Views
  • 3 replies
  • 2 kudos

How to create a custom, easy to remember URL for my Databricks workspace?

Data Engineering
Reply
  • 3287 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Aria
New Contributor III
  • 2 kudos

We are using azure.I dont see an option for deployment name. Secondly, we have already deployed all our workspaces and wants to have user friendly URLs.Like some changes in DNS server or proxy URLs.

  • 2 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels