cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Meshynix
by • New Contributor III
  • 5806 Views
  • 6 replies
  • 0 kudos

Resolved! Not able to create external table in a schema under a Catalog.

Problem StatementCluster 1 (Shared Cluster) is not able to read the file location at "dbfs:/mnt/landingzone/landingzonecontainer/Inbound/" and hence we are not able to create an external table in a schema inside Enterprise Catalog.Cluster 2 (No Isola...

Data Engineering
Reply
  • 5806 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Avi_Bricks
New Contributor II
  • 0 kudos

External table creation failing with error :- UnityCatalogServiceException:[RequestId=**** ErrorClass=INVALID_PARAMETER_VALUE] Unsupported path operation PATH_CREATE_TABLE on volume.Able to access and create files on external location.  

  • 0 kudos
5 More Replies
pshuk
by • New Contributor III
  • 1294 Views
  • 1 replies
  • 0 kudos

run md5 using CLI

Hi,I want to run a md5 checksum on the uploaded file to databricks. I can generate md5 on the local file but how do I generate one on uploaded file on databricks using CLI (Command line interface). Any help would be appreciated.I tried running databr...

Data Engineering
Reply
  • 1294 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi,I want to run a md5 checksum on the uploaded file to databricks. I can generate md5 on the local file but how do I generate one on uploaded file on databricks using CLI (Command line interface). Any help would be appreciated.I tried running databr...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
danial
by • New Contributor II
  • 6211 Views
  • 3 replies
  • 1 kudos

Connect Databricks hosted on Azure, with RDS on AWS.

We have Databricks set up and running on Azure. Now we want to connect it with RDS (AWS) to transfer data from RDS to Azure DataLake using the Databricks.I could find the documentation on how to do it within the same cloud (Either AWS or Azure) but n...

Data Engineering
Reply
  • 6211 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Danial Malik​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 1 kudos
2 More Replies
cszczotka
by • New Contributor III
  • 1804 Views
  • 4 replies
  • 0 kudos

Shallow clone and issue with MODIFY permission to source table

Hi,I'm running shallow clone for external delta tables. The shallow clone is failing for source tables where I don't have MODIFY permission. I'm getting below exception. I don't understand why MODIFY permission to source table is required. Is there a...

Data Engineering
Reply
  • 1804 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Amit_Dass_Chmp
New Contributor III
  • 0 kudos

Also check this documentation on access mode :Shallow clone for Unity Catalog tables | Databricks on AWS Working with Unity Catalog shallow clones in Single User access mode, you must have permissions on the resources for the cloned table source as w...

  • 0 kudos
3 More Replies
Maatari
by • New Contributor III
  • 1520 Views
  • 1 replies
  • 0 kudos

Fixed interval micro-batches and AvailableNow Trigger

What is the fundamental difference between Fixed interval micro-batches and AvailableNow Trigger, given that both can consume in micro-batch based on the desire size of the micro batch ? Is the fundamental difference the fact that AvailableNow shut d...

Data Engineering
Reply
  • 1520 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
What is the fundamental difference between Fixed interval micro-batches and AvailableNow Trigger, given that both can consume in micro-batch based on the desire size of the micro batch ? Is the fundamental difference the fact that AvailableNow shut d...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
surband
by • New Contributor III
  • 5618 Views
  • 9 replies
  • 0 kudos

Pulsar Streaming (Read) - Benchmarking Information

We are doing a first time implementation of data streaming reading from a partitioned pulsar topics to a delta table managed by UC. We are unable to scale the job beyond about ~ 40k msgs/sec. Beyond 40k msgs/sec , the job fails.  I'd imagine Databric...

Data Engineering
Reply
  • 5618 Views
  • 9 replies
  • 0 kudos
View Replies
Latest Reply
surband
New Contributor III
  • 0 kudos

Attached Grafana screenshots

  • 0 kudos
8 More Replies
bradleyjamrozik
by • New Contributor III
  • 620 Views
  • 0 replies
  • 0 kudos

Autoloader Failure Creating EventSubscription

Posting this here too in case anyone else has run into this issue... Trying to set up Autoloader File Notifications but keep getting an "Internal Server Error" message.Failure on Write EventSubscription - Internal error - Microsoft Q&A

Data Engineering
Reply
  • 620 Views
  • 0 replies
  • 0 kudos
JacobKesinger
by • New Contributor II
  • 3302 Views
  • 3 replies
  • 0 kudos

Resolved! Iterating over a pyspark.pandas.groupby.DataFrameGroupBy

I have a pyspark.pandas.frame.DataFrame object (that I called from `pandas_api` on a pyspark.sql.dataframe.DataFrame object).  I have a complicated transformation that I would like to apply to this data, and in particular I would like to apply it in ...

Data Engineering
Reply
  • 3302 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
MichTalebzadeh
Valued Contributor
  • 0 kudos

Hi,The error indicates that the Unity Catalog does not support Spark higher-order functions, such as those used in pandas_udf. This limitation likely comes from architectural or compatibility constraints. To resolve the issue, consider alternative ap...

  • 0 kudos
2 More Replies
nileshtiwaari
by • New Contributor
  • 645 Views
  • 1 replies
  • 0 kudos

Unity Catalog External Tables

what if I delete the external tables files manually on storage account for external table without dropping the table itself?

Data Engineering
Reply
  • 645 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
mhiltner
Databricks Employee
  • 0 kudos

This change won't be registered as metadata thus the table will still appear on unity catalog, but you'll get an error when trying to access it, as the table metadata will point to deleted files.   

  • 0 kudos
FilipezAR
by • New Contributor
  • 5896 Views
  • 0 replies
  • 0 kudos

Failed to create new KafkaAdminClient

I want to create connections to kafka with spark.readStream using the following parameters: kafkaParams = { "kafka.sasl.jaas.config": f'org.apache.kafka.common.security.plain.PlainLoginModule required username="{kafkaUsername}" password="{kafkaPa...

Data Engineering
Reply
  • 5896 Views
  • 0 replies
  • 0 kudos
kazinahian
by • New Contributor III
  • 2391 Views
  • 1 replies
  • 0 kudos

Lowcode ETL in Databricks

Hello everyone,I work as a Business Intelligence practitioner, employing tools like Alteryx or various low-code solutions to construct ETL processes and develop data pipelines for my Dashboards and reports. Currently, I'm delving into Azure Databrick...

Data Engineering
Reply
  • 2391 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello everyone,I work as a Business Intelligence practitioner, employing tools like Alteryx or various low-code solutions to construct ETL processes and develop data pipelines for my Dashboards and reports. Currently, I'm delving into Azure Databrick...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
Yashir
by • New Contributor III
  • 3607 Views
  • 5 replies
  • 4 kudos

Is there a way to add Features descriptions for each of the features in a Feature Store table?

 If not, then I believe that it will be beneficial because the feature tables contain engineered features that its a good idea to document their calc logic for the benefit of other data scientists. Also, even non-engineered features are many times no...

Data Engineering
Reply
  • 3607 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
deep_thought
Contributor
  • 4 kudos

I also would like to see support added for feature description get/set methods.

  • 4 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels