cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
User16826994223
by • Honored Contributor III
  • 1122 Views
  • 3 replies
  • 0 kudos

What is Autolader in Databricks?

Want to Know what is Autoloader and what are its advantages

Data Engineering
Reply
  • 1122 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

The biggest advantage is the ease with which you can star ingesting data from your Cloud Storage directly into a Delta Table. You can choose Directory Listing mode or File Notification mode, depending on what fits your use case best.

  • 0 kudos
2 More Replies
MoJaMa
by • Valued Contributor II
  • 844 Views
  • 1 replies
  • 0 kudos

Does Databricks support a Centralized Feature Store?

Data Engineering
Reply
  • 844 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

Most possibly in future as we progress down our Roadmap.Currently it is per-workspace, and only accessible in Databricks notebooks/jobs.Please refer to our docs:https://docs.databricks.com/applications/machine-learning/feature-store.html#known-limita...

  • 0 kudos
MoJaMa
by • Valued Contributor II
  • 913 Views
  • 1 replies
  • 0 kudos

What is this Photon Engine I keep hearing about?

Data Engineering
Reply
  • 913 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

It's our new high-performance runtime, using a native vectorized engine developed in C++.Please see our blog for a great overview. https://databricks.com/blog/2021/06/17/announcing-photon-public-preview-the-next-generation-query-engine-on-the-databri...

  • 0 kudos
MoJaMa
by • Valued Contributor II
  • 789 Views
  • 1 replies
  • 0 kudos

Does Databricks still require CreateKeyPair and DeleteKeyPair permissions in the cross-account IAM roles on AWS?

Data Engineering
Reply
  • 789 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

We used to require this, but starting June 9, 2021 we no longer do, and have improved our E2 security posture.See https://docs.databricks.com/administration-guide/account-api/iam-role.html for the current permissions required.

  • 0 kudos
MoJaMa
by • Valued Contributor II
  • 975 Views
  • 1 replies
  • 1 kudos

How can I understand Photon pricing for Data Engineering and estimate costs for a workload?

Data Engineering
Reply
  • 975 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
MoJaMa
Valued Contributor II
  • 1 kudos

Use the calculator herehttps://databricks.com/product/aws-pricing/instance-typesOpen two windows side by side, pick Photon and Non-Photon instances of the same type and compare.

  • 1 kudos
sajith_appukutt
by • Honored Contributor II
  • 1109 Views
  • 1 replies
  • 1 kudos

Resolved! Are there any ways to automatically cleanup temporary files created in s3 by the Amazon Redshift connector

The Amazon Redshift data source in Databricks seems to be using S3 for storing intermediate results. Are there any ways to automatically cleanup temporary files created in S3

Data Engineering
Reply
  • 1109 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

You could use storage lifecycle policy for the s3 bucket used for storing intermediate results and configure expiration actions. This way temporary/intermediate results would be automatically cleaned up

  • 1 kudos
User16752246553
by • New Contributor
  • 1004 Views
  • 1 replies
  • 1 kudos

How does Vectorized Pandas UDF work?

Do Vectorized Pandas UDFs apply to batches of data sequentially or in parallel? And is there a way to set the batch size?

Data Engineering
Reply
  • 1004 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

>How does Vectorized Pandas UDF work?Here is a video explaining the internals of Pandas UDFs (a.k.a. Vectorized UDFs) - https://youtu.be/UZl0pHG-2HA?t=123 . They use Apache Arrow, to exchange data directly between JVM and Python driver/executors wit...

  • 1 kudos
User16826992666
by • Valued Contributor
  • 2143 Views
  • 1 replies
  • 0 kudos

Resolved! What is the difference between a trigger once stream and a normal one time write?

It seems to me like both of these would accomplish the same thing in the end. Do they use different mechanisms to accomplish it though? Are there any hidden costs to streaming to consider?

Data Engineering
Reply
  • 2143 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

The biggest reason to use the streaming API over the non-stream API would be to enable the checkpoint log to maintain a processing log. It is most common for people to use the trigger once when they want to only process the changes between executions...

  • 0 kudos
User16752240150
by • New Contributor II
  • 1194 Views
  • 1 replies
  • 0 kudos

What's the best way to use hyperopt to train a spark.ml model and track automatically with mlflow?

I've read this article, which covers:Using CrossValidator or TrainValidationSplit to track hyperparameter tuning (no hyperopt). Only random/grid searchparallel "single-machine" model training with hyperopt using hyperopt.SparkTrials (not spark.ml)"Di...

Data Engineering
Reply
  • 1194 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

It's actually pretty simple: use hyperopt, but use "Trials" not "SparkTrials". You get parallelism from Spark, not from the tuning process.

  • 0 kudos
User16826992666
by • Valued Contributor
  • 1077 Views
  • 1 replies
  • 0 kudos

Resolved! When should I create a Bloom Filter Index on my Delta table?

Data Engineering
Reply
  • 1077 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

A bloom filter index is a space-efficient data structure that enables data skipping on chosen columns, particularly for fields containing arbitrary text. The Bloom filter operates by either stating that data is definitively not in the file, or that i...

  • 0 kudos
User16826994223
by • Honored Contributor III
  • 922 Views
  • 1 replies
  • 0 kudos

Delta concurrency write Issue

What is concurrent issue in delta, If at a time if we try to write same delta table , it some times fail , how to mitigate that

Data Engineering
Reply
  • 922 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

Delta Lake uses optimistic concurrency control to provide transactional guarantees between writes. Read: Reads (if needed) the latest available version of the table to identify which files need to be modified (that is, rewritten).Write: Stages all th...

  • 0 kudos
sajith_appukutt
by • Honored Contributor II
  • 964 Views
  • 1 replies
  • 1 kudos

Resolved! What are the list of connections that I need to open up in my hub inspection firewall for Databricks to work in AWS

Data Engineering
Reply
  • 964 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

You'd need to open connections to Databricks web applicationDatabricks secure cluster connectivity (SCC) relayAWS S3 global URLAWS S3 regional URLAWS STS global URLAWS STS regional URLAWS Kinesis regional URLTable metastore RDS regional URL (by data ...

  • 1 kudos
Anonymous
by • Not applicable
  • 1081 Views
  • 2 replies
  • 0 kudos

Resolved! Collaborative features

What do you mean by collaborative data science? What collaboration features do you support?

Data Engineering
Reply
  • 1081 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

This primarily refers to the fact that notebooks can be shared to the whole org, to groups, to users, and can be limited to read/write/execute. You could argue that MLflow is also a form of collaboration, where multiple users can share an experiment ...

  • 0 kudos
1 More Replies
Srikanth_Gupta_
by • Valued Contributor
  • 1884 Views
  • 2 replies
  • 0 kudos

What are best instance types to use Delta Lake on AWS, Azure and GCP?

Best instance types to use Delta in a better way, are there any recommendations?Example: i3.xlarge vs m5.2x large vs D3v2

Data Engineering
Reply
  • 1884 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Mooune_DBU
Valued Contributor
  • 0 kudos

Depending on your queries, if you're looking for Delta Cache Optimized instances, here's the list per provider:AWS: i3.* (i.e. i3.xlarge)Azure: Ls-types (i.e. L4sv2)GCP: n2-highmem-*

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels