With the announcement of Variant format, I'm wondering if this is a good way to consolidate small tables with varying columns and formats into a single normalized table? Any ideas?
Great learning on serverless compute, Unity catalog, etc
Hi All, I'm trying to change our Ingestion process to use Autoloader to identify new files landing in a directory on ADLS. The ADLS directory has access tier enabled to archive files older than a certain time period. When I'm trying to set up Autoloa...
Hi Databricks Community, I am trying to stream from a bronze to a silver table, however, I have the problem that there may be updates in the bronze table. Delta table streaming reads and write does not support skipChangeCommits=false, i.e. handle mo...
Hi, You can use dlt apply changes to deal with changing source.Delta Live Tables Python language reference | Databricks on AWSThank you
We use a software called Apptio Datalink where it allows you to call databases, specifically, we are using it to retrieve data from our Prod Databricks Database. Their connector is returning the following message: { "time" : "1717506136286", "errorC...
Databricks Tutorial : Data Warehouse vs Data Lake vs Delta Lake @Sujitha @Kaniz_Fatma
I wanted a more filtered data set from a materialized view so I figured a view might be the solution but it doesn't get created under the target schema however it shows upon in the graph as a part of the pipeline. Can't we use MVs as data source for...
Issue at Hand:You mentioned that a view is not created under the target schema but appears in the DLT graph. This situation arises due to how DLT manages views and materialized views.Possible Causes and Solutions:DLT Execution and Target Schema:In DL...
Hi All,We are using the Azure Databricks platform for one of our Data Engg needs. Here's my setup -1. Job compute that uses Cluster of size - 1 driver and 2 workers - all are of 'Standard_DS3_v2' type. (Photon is disabled).2. The job compute takes th...
To calculate the real cost of an Azure Cluster or Job, there are two ways: DIY, which means querying the Microsoft Cost API and Databricks API and then combining the information to get the exact cost, or you can use a tool such as KopiCloud Databrick...
Dear community, we are using the Azure Databricks service and wondering if uploading a file to the DBFS (or to a storage accessed directly from a notebook in Databricks) could be a potential security threat. Imagine you upload some files with 'malici...
Uploading a file to the Databricks File System (DBFS) or accessing storage directly from a notebook in Azure Databricks could pose potential security risks if not managed properly. Here are some considerations:Sensitive Data Exposure: Uploading sensi...
I want to read the last modified datetime of the files in data lake in a databricks script. If I could read it efficiently as a column when reading data from data lake, it would be perfect.Thank you:)
Efficiently reading data lake files involves:Choosing the Right Tools: Select tools optimized for data lake file formats (e.g., Parquet, ORC) and distributed computing frameworks (e.g., Apache Spark, Apache Flink).Partitioning and Indexing: Partition...
Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)> python --versionPython 3.10.4This error seems to be coming from the thrift backend. I suspect but have not confirmed that t...
One way to resolve this could be to ensure your connection values in are surrounded by quotes. host: "hostname.databricks.com" # Requiredhttp_path: "/sql/1.0/warehouses/aaaabbbccc"
Hoping to meet people at the data Ai summit!
Do you guys think vector search will be cheaper and perform better than Azure AI Search?
I'm looking to meet with others who have successfully connected databricks to SAP ECC.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group
