cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
alvaro_databric
by • New Contributor III
  • 13896 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks Python Import Bug

Hello,I am reaching to the community in order to shed some light into a Bug I have been encountering recently. The two setups are as follows:SETUP-1 (WORKS):Python Notebook in Workspace FileSystem (this is Shared/folder/...)Python custom wheel librar...

Data Engineering
Reply
  • 13896 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
alvaro_databric
New Contributor III
  • 1 kudos

Solution has comed with an update, as stated in Work with Python and R modules | Databricks on AWS, In Databricks Runtime 13.0 and above, directories added to the Python sys.path are automatically distributed to all executors in the cluster. In Datab...

  • 1 kudos
2 More Replies
PraveenSaini
by • New Contributor
  • 139028 Views
  • 34 replies
  • 6 kudos

How to read excel file using databricks

0 I have a excel file as source file and i want to read data from excel file and convert data in data frame using databricks. I have already added maven dependence for Excel file format. when i a tring below code it is giving error .(Error: java.io....

Data Engineering
Reply
  • 139028 Views
  • 34 replies
  • 6 kudos
View Replies
Latest Reply
Jenish_lodha
New Contributor II
  • 6 kudos

To read an Excel file using Databricks, you can use the Databricks runtime, which supports multiple programming languages such as Python, Scala, and R. Here are the general steps to read an Excel file in Databricks using Python:1. **Upload the Excel ...

  • 6 kudos
33 More Replies
Gilg
by • Contributor II
  • 5969 Views
  • 0 replies
  • 1 kudos

Apply Changes Statistics

Hi Team,I just wanted to know why APPLY_CHANGES do not show how many rows it got upserted in the graph? Is this something in the works in the future?Cheers,G

Data Engineering
Reply
  • 5969 Views
  • 0 replies
  • 1 kudos
SubhaJ
by • New Contributor III
  • 3513 Views
  • 5 replies
  • 4 kudos

Request for a retest voucher

Hi Team,I gave the Databricks Certified Data Engineer Associate (Version 2) on 10th June, 2023. I got 66.66% and I intend to retake it. Is it possible to get a voucher for the retest. That would be a great encouragement and help. I would really appre...

image
Data Engineering
Reply
  • 3513 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
Theor
New Contributor III
  • 4 kudos

Hi Databricks Team,I set for Databricks Certified Machine Learning Professional exam for 2nd time, but didn't pass again. Got 66.66% overall.I am seasoned databricks user but this particular exam is quite unorthodox one. Nevertheless I know I can be ...

  • 4 kudos
4 More Replies
meystingray
by • New Contributor II
  • 7828 Views
  • 2 replies
  • 0 kudos

Time-out when using init script to install GDAL

I'm following this post to run an init script to install gdal.https://community.databricks.com/t5/get-started-discussions/gdal-on-databricks-cluster-runtime-12-2-lts/m-p/39091#M662My script is simply:--------------#!/bin/bashapt-get update && apt-get...

Data Engineering
Reply
  • 7828 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
I'm following this post to run an init script to install gdal.https://community.databricks.com/t5/get-started-discussions/gdal-on-databricks-cluster-runtime-12-2-lts/m-p/39091#M662My script is simply:--------------#!/bin/bashapt-get update && apt-get...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
-werners-
by • Esteemed Contributor III
  • 3127 Views
  • 1 replies
  • 0 kudos

git integration with volumes?

Volumes in unity are said to be ideal for "storing library and config files of arbitrary formats such as .whl or .txt centrally and providing secure access across workspaces to it".So basically this is the same we had with dbfs but with decent access...

Data Engineering
Reply
  • 3127 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Volumes in unity are said to be ideal for "storing library and config files of arbitrary formats such as .whl or .txt centrally and providing secure access across workspaces to it".So basically this is the same we had with dbfs but with decent access...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
BananaHotSauce
by • New Contributor III
  • 1297 Views
  • 1 replies
  • 0 kudos

Cross Account External Table

I created an external table and set the location on other AWS Account (External Location), lets call it Account B. Databricks workspace is deployed in other account. Lets call it Account A.Here is my ScriptCREATE TABLE devtest.governance.dbtable1(col...

Data Engineering
External Location
External Table
S3
spark
Reply
  • 1297 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

I think you should start checking from our cross account role access.https://docs.databricks.com/en/administration-guide/account-settings-e2/audit-aws-cross-account.html#step-3-configure-cross-account-support-optional Check if this is well placed.

  • 0 kudos
pavlos_skev
by • New Contributor III
  • 1037 Views
  • 1 replies
  • 0 kudos

Potential Unity Catalog Bug: Invalid configuration value detected for fs.azure.account.keyInvalid

Hello,We are migrating to Unity Catalog (UC), and for very few of our tables, we get the below error when trying to write or even display them. We are using UC enabled clusters, usually with runtime versions 12.2 LTS. The below error, when it happens...

Data Engineering
Reply
  • 1037 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16539034020
Databricks Employee
  • 0 kudos

Hello,  Thanks for contacting Databricks Support.  The error message indicates a problem with the configuration key fs.azure.account.key. This configuration key is used to provide the access key for the Azure Data Lake Storage account. Not sure if th...

  • 0 kudos
GriffLehman
by • New Contributor II
  • 957 Views
  • 1 replies
  • 0 kudos

PROBLEM- Missing data in "Last Run" column in Databricks Workflows UI

Hello,I am having a pretty major problem with the Databricks Workflows UI- when I look at the list of jobs, the "Last Run" column does not have any data in it. This is kind of a big problem because now I don't have a good way of getting visibility in...

Data Engineering
Reply
  • 957 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
GriffLehman
New Contributor II
  • 0 kudos

  

  • 0 kudos
MariuszJ
by • New Contributor II
  • 1358 Views
  • 1 replies
  • 2 kudos

cdc type 2 - not closing records

Hi,i would not mind small advice, i do have dlt cdc typ2, the definition dlt.create_streaming_table('`my_table_dlt_cdc`') dlt.apply_changes( target = 'my_table_dlt_cdc', source = 'source', keys = ['id'], sequence_by = col('snapshot_date'), ...

Data Engineering
Reply
  • 1358 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
MariuszJ
New Contributor II
  • 2 kudos

For anyone who sees this post in the future. I was missing one argumentapply_as_deletes  

  • 2 kudos
ehpogue
by • New Contributor III
  • 14293 Views
  • 9 replies
  • 3 kudos

how do i re-enable tab complete / autocomplete?

yesterday all of my notebooks seemingly changed to have python formatting (which seems to be in this week's release), but the unintended consequence is that shift + tab (which used to show docstrings in python) now just un-indents code, and tab inser...

Data Engineering
Reply
  • 14293 Views
  • 9 replies
  • 3 kudos
View Replies
Latest Reply
Data_33
New Contributor II
  • 3 kudos

i also facing the same in databricks now.

  • 3 kudos
8 More Replies
rahulgulati89
by • New Contributor II
  • 2294 Views
  • 1 replies
  • 0 kudos

Unable to connect to secured schema registry from Azure Databricks

Hi,I am unable to connect to secure schema registry(running on https) as it is breaking with below mentioned error.SCHEMA_REGISTRY_CONFIGURATION_ERROR] Schema from schema registry could not be initialized. Error while fetching schema for subject 'env...

rahulgulati89_0-1694516966345.png
Data Engineering
Reply
  • 2294 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi,I am unable to connect to secure schema registry(running on https) as it is breaking with below mentioned error.SCHEMA_REGISTRY_CONFIGURATION_ERROR] Schema from schema registry could not be initialized. Error while fetching schema for subject 'env...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
Rpabs
by • New Contributor
  • 792 Views
  • 0 replies
  • 0 kudos

Unable to load historical data via DLT workflow

HiI'm trying to load historical data with the DLT workflow and facing a Out of Memory or Executor heartbeat lost error. The historical data loads fine with normal processing, but fails within DLT workflow. Tried repartitioning and scaling cluster, bu...

Data Engineering
Reply
  • 792 Views
  • 0 replies
  • 0 kudos
DeltaTrain
by • New Contributor II
  • 1497 Views
  • 1 replies
  • 0 kudos

Access Control in hive_metastore Based on Cluster Type

Hello Databricks Community, I asked the same question on the Get Started Discussion page but feels like here is the right place for this question. I'm reaching out with a query regarding access control in the hive_metastore. I've encountered behavior...

DeltaTrain_0-1691618617261.png DeltaTrain_1-1691618617263.png
Data Engineering
Reply
  • 1497 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16752239289
Databricks Employee
  • 0 kudos

That is expected. The single user mode is the legacy standard + UC ACL enabled. https://docs.databricks.com/en/archive/compute/cluster-ui-preview.html#how-does-backward-compatibility-work-with-these-changes For your case, you need the hive table acl ...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels