cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
MaximeGendre
by • New Contributor III
  • 698 Views
  • 1 replies
  • 0 kudos

How to disable DBFS storage

Hello,I administer a self-service oriented Databricks workspace and I notice that more and more users are storing their data in DBFS due to lack of knowledge.They are not specifying a location when creating their schema or they are not specifying a s...

MaximeGendre_1-1718227315313.png
Data Engineering
Reply
  • 698 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
MaximeGendre
New Contributor III
  • 0 kudos

Replacing "/mnt/adl2" by "dbfs:/mnt/adl2" fixed the issue.

  • 0 kudos
sagarsiddhabha
by • New Contributor
  • 326 Views
  • 0 replies
  • 0 kudos

Attended Data +AI summit at SFO

It was great experience attending this conference.Got great insights about new features.Got to know about new advances in data industry.Attending the conference was an enriching and transformative experience. I gained invaluable insights into the lat...

Data Engineering
Reply
  • 326 Views
  • 0 replies
  • 0 kudos
mscsu
by • New Contributor
  • 586 Views
  • 0 replies
  • 0 kudos

Unity catalog

Great learning on serverless compute, Unity catalog, etc

Data Engineering
Reply
  • 586 Views
  • 0 replies
  • 0 kudos
Manjula_Ganesap
by • Contributor
  • 360 Views
  • 0 replies
  • 0 kudos

Autoloader on ADLS blobs with archival enabled

Hi All, I'm trying to change our Ingestion process to use Autoloader to identify new files landing in a directory on ADLS. The ADLS directory has access tier enabled to archive files older than a certain time period. When I'm trying to set up Autoloa...

Data Engineering
Reply
  • 360 Views
  • 0 replies
  • 0 kudos
Trilleo
by • New Contributor III
  • 2858 Views
  • 4 replies
  • 2 kudos

Resolved! Handle updates from bronze to silver table stream

Hi Databricks Community,  I am trying to stream from a bronze to a silver table, however, I have the problem that there may be updates in the bronze table. Delta table streaming reads and write does not support skipChangeCommits=false, i.e. handle mo...

Data Engineering
Reply
  • 2858 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Himali_K
New Contributor II
  • 2 kudos

Hi, You can use dlt apply changes to deal with changing source.Delta Live Tables Python language reference | Databricks on AWSThank you

  • 2 kudos
3 More Replies
aliehs0510
by • New Contributor II
  • 426 Views
  • 1 replies
  • 1 kudos

DLT Pipeline does not create the view but it shows up on the DLT graph

I wanted a more filtered data set from a materialized view so I figured a view might be the solution but it doesn't get created under the target schema  however it shows upon in the graph as a part of the pipeline. Can't we use MVs as data source for...

Data Engineering
Reply
  • 426 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Rishabh-Pandey
Esteemed Contributor
  • 1 kudos

Issue at Hand:You mentioned that a view is not created under the target schema but appears in the DLT graph. This situation arises due to how DLT manages views and materialized views.Possible Causes and Solutions:DLT Execution and Target Schema:In DL...

  • 1 kudos
sanket-kelkar
by • New Contributor II
  • 5899 Views
  • 4 replies
  • 1 kudos

Databricks costing - Need details of the Azure VM costs

Hi All,We are using the Azure Databricks platform for one of our Data Engg needs. Here's my setup -1. Job compute that uses Cluster of size - 1 driver and 2 workers - all are of 'Standard_DS3_v2' type. (Photon is disabled).2. The job compute takes th...

Data Engineering
Reply
  • 5899 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
GuillermoM
New Contributor II
  • 1 kudos

To calculate the real cost of an Azure Cluster or Job, there are two ways: DIY, which means querying the Microsoft Cost API and Databricks API and then combining the information to get the exact cost, or you can use a tool such as KopiCloud Databrick...

  • 1 kudos
3 More Replies
CAN
by • New Contributor
  • 950 Views
  • 1 replies
  • 0 kudos

Security Threats in Databricks for File Upload

Dear community, we are using the Azure Databricks service and wondering if uploading a file to the DBFS (or to a storage accessed directly from a notebook in Databricks) could be a potential security threat. Imagine you upload some files with 'malici...

Data Engineering
Reply
  • 950 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
KrunalMedapara
New Contributor II
  • 0 kudos

Uploading a file to the Databricks File System (DBFS) or accessing storage directly from a notebook in Azure Databricks could pose potential security risks if not managed properly. Here are some considerations:Sensitive Data Exposure: Uploading sensi...

  • 0 kudos
User16790091296
by • Contributor II
  • 969 Views
  • 1 replies
  • 0 kudos

How to efficiently read the data lake files' metadata?

I want to read the last modified datetime of the files in data lake in a databricks script. If I could read it efficiently as a column when reading data from data lake, it would be perfect.Thank you:)

Data Engineering
Reply
  • 969 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
KrunalMedapara
New Contributor II
  • 0 kudos

Efficiently reading data lake files involves:Choosing the Right Tools: Select tools optimized for data lake file formats (e.g., Parquet, ORC) and distributed computing frameworks (e.g., Apache Spark, Apache Flink).Partitioning and Indexing: Partition...

  • 0 kudos
mattmunz
by • New Contributor III
  • 21933 Views
  • 4 replies
  • 0 kudos

How can I resolve this SSL error which occurrs when calling databricks-sql-connector/databricks.sql.connect() from my python app?

Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)>  python --versionPython 3.10.4This error seems to be coming from the thrift backend. I suspect but have not confirmed that t...

Data Engineering
Reply
  • 21933 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
twole
Databricks Employee
  • 0 kudos

One way to resolve this could be to ensure your connection values in are surrounded by quotes. host: "hostname.databricks.com" # Requiredhttp_path: "/sql/1.0/warehouses/aaaabbbccc"   

  • 0 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels