Community Activity
- 122 Views
- 2 replies
- 0 kudos
Build a streaming table on top of a snowflake table
Is it possible to create a streaming table on top of a snowflake tablel accessible via the Lakehouse Federation?
- 122 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi abelian-grape,How are you doing today?, As per my understanding, Right now, it’s not possible to create a streaming table directly on top of a Snowflake table that’s accessed through Lakehouse Federation in Databricks. Lakehouse Federation allows ...
- 0 kudos
- 226 Views
- 2 replies
- 0 kudos
Triggering Downstream Workflow in Databricks from New Inserts in Snowflake
Hi Databricks experts,I have a table in Snowflake that tracks newly added items, and a downstream data processing workflow that needs to be triggered whenever new items are added. I'm currently using Lakehouse Federation to query the Snowflake tables...
- 226 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi abelian-grape,Great question! Since you're using Lakehouse Federation to access the Snowflake table, and Databricks can't directly stream from or listen to inserts in Snowflake, the best approach is to use an interval-based polling mechanism in Da...
- 0 kudos
- 36574 Views
- 142 replies
- 9 kudos
Suspended Exam? Don't worry! Our support team can help!
Taking exams is so stressful already, and if your exam gets suspended it is extra stressful. Don't worry! Our support team is awesome and will help you get back on track. The most important step is to file a ticket in our Help Center. Be sure to inc...
- 36574 Views
- 142 replies
- 9 kudos
Latest Reply
- 9 kudos
@Cert-Team I had my certification exam for Databricks Data Engineer Professional scheduled today 3:30PM ET. As soon as I started the exam, within 5 minutes I was told to pause and show the room around; and it was clear per the exam policy. Immediatel...
- 9 kudos
- 645 Views
- 1 replies
- 1 kudos
Vnet Gateway issues on Power bi Conn
Team,We are getting frequent vnet gateway failures on power bi Dataset using DAX(simple DAX not complex) and upon the rerun it is working, is any permanent fix for this ?Error :{"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error...
- 645 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Similar issue. Any solution?
- 1 kudos
- 27 Views
- 0 replies
- 0 kudos
Urgent Issue - Databricks Data Engineer Professional Exam Suspended
Hello Databricks Community - @Cert-TeamOPS ,I am writing with significant concern and frustration regarding my Databricks Data Engineer Professional exam that was scheduled for today, April 4, 2025, at 3:30 PM EDT.I started the exam on time, having p...
- 27 Views
- 0 replies
- 0 kudos
- 24 Views
- 0 replies
- 0 kudos
Databricks Access Management
Hi, Can someone suggest me what's the least level of access we can provide to any external business user to able to run a notebook and download the results. I tried with giving read and run level permissions at notebook level and also with a user gro...
- 24 Views
- 0 replies
- 0 kudos
- 186 Views
- 3 replies
- 2 kudos
Who’s heading to the Data & AI Summit in San Francisco this year?
The Data + AI Summit is happening June 9-12, and we are gearing up for an incredible in-person experience! With 22,000+ data enthusiasts, 700+ sessions, and hands-on training, this is where innovation meets collaboration. Whether you are a data leade...
- 186 Views
- 3 replies
- 2 kudos
- 68 Views
- 1 replies
- 0 kudos
Optimizing Spark Read Performance on Delta Tables with Deletion Vectors Enabled
Hi Databricks Experts,I'm currently using Delta Live Table to generate master data managed within Unity Catalog, with the data stored directly in Google Cloud Storage. I then utilize Spark to read these master data from the GCS bucket. However, I’m ...
- 68 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi Hung, The performance issues you're experiencing with Delta tables and deletion vectors are common challenges when working with Delta Live Tables. Let me address your questions: How to Optimize Spark Read Performance with Deletion Vectors Deletion...
- 0 kudos
- 15784 Views
- 18 replies
- 18 kudos
Resolved! How to connect Neo4j aura to a cluster
Please help resolve this error :org.neo4j.driver.exceptions.SecurityException: Failed to establish secured connection with the serverThis occurs when I want to establish a connection to neo4j aura to my cluster .Thank you.
- 15784 Views
- 18 replies
- 18 kudos
Latest Reply
- 18 kudos
I have added the init script on cluster start up in Spark->config->init.scripts. But my clsuter won't start after this.Cluster scoped init script /Volumes/xxx/xxx/neo4j/neo4j-init.sh failed: Script exit status is non-zero. Could you please help me or...
- 18 kudos
- 130 Views
- 1 replies
- 0 kudos
.py file running stuck on waiting
Hello, hope you are doing well.We are facing an issue when running .py files. This is fairly recent and we were not experiencing this issue last week.As shown in the screenshots below, the .py file hangs on "waiting" after we press "run all". No matt...
- 130 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey @BigAlThePal, It looks like a UI bug, especially in Microsoft Edge. The code actually runs, but the output doesn't show until you refresh. A few quick things you can try:Run cells individually instead of using "Run All"Switch to Chrome or Firefox...
- 0 kudos
- 637 Views
- 2 replies
- 1 kudos
Getting error while creating external delta table in Databricks
I am getting below error while creating external delta table in Databricks, even there is a external location created.[NO_PARENT_EXTERNAL_LOCATION_FOR_PATH] No parent external location was found for path 'abfss://destination@datalakeprojectsid.dfs.co...
- 637 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @Siddalinga,Is the issue fixed ? If not please check below options : 1. external location is created in databricks with proper service account or service principle ? 2. Or any storage location is attached with proper service account. ?Above 2 m...
- 1 kudos
- 315 Views
- 2 replies
- 0 kudos
I’m having trouble connecting to my AWS S3 bucket
Hello,I’m having trouble connecting to my AWS S3 bucket from my local Python environment using boto3. When I try to run the following code:import boto3s3 = boto3.client('s3')response = s3.list_objects_v2(Bucket='my-bucket-name')print(response)error m...
- 315 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello Mate,Are you trying from databricks notebook? if so can you share the dbr version ?If not, you can install the below code :import sslprint(ssl.OPENSSL_VERSION)and update the pip package : `pip install --upgrade pip` try post this. Happy learnin...
- 0 kudos
- 557 Views
- 2 replies
- 0 kudos
Data Management and Governance with Unity Catalog Demo Set up error
I am trying to run classroom setup for Demo:Upgrading Tables to Unity Catalog lab in the above course and get multiple errors. I wasn able to complete previous 2 labs successfully.Here are some errors while running the classrom set up script "%run ./...
- 557 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello Mate,Only few courses are allowing to change the table metadata or type itself for the labsessions. In this case you can download and try in your workspace(if any) that will work easily for the learning purpose.Thanks for the ask, happy learnin...
- 0 kudos
- 662 Views
- 1 replies
- 0 kudos
set up my AWS credentials and configure boto3, but I’m facing an issue
Hi everyone,I’m working on a project where I need to upload files from my local machine to an AWS S3 bucket using the boto3 library in Python. I’ve followed the official documentation to set up my AWS credentials and configure boto3, but I’m facing a...
- 662 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello Mate,Thanks for the ask, are you trying to read via notebook ? If so you need to pass the credentials either cluster level or best approach via secret scopes then it will work.Please ignore if not matching with your ask.Saran
- 0 kudos
- 545 Views
- 1 replies
- 0 kudos
How to check databricks-connect types of objects.
While using `databricks-sdk` in my code, I've found that checking PySpark objects types is not reliable anymore. I've used to do the following: from pyspark.sql import Column, DataFrame, SparkSession isinstance(spark, SparkSession) isinstance(a_df...
- 545 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello Mate,Is the issue fixed ? did you got any reference or support from databricks end ?Saran
- 0 kudos
Announcements
Latest from our Blog
The Ultimate Guide to Chunking Strategies for RAG Applications with Databricks
Over the years, I have collaborated closely with ML engineering leaders across various industries, guiding them on how to make the right chunking strategy decisions for their Retrieval-Augmented Gener...
Migrating ML Workload to Databricks
As machine learning (ML) workloads continue to grow in complexity and scale, organizations are looking for efficient and scalable solutions to manage their ML lifecycle. Databricks offers a powerful p...
Building Trust in Data II: A Guide to Effective Data Testing Tactics
Introduction Building a reliable data pipeline goes beyond setting up a functional workflow — it requires meticulous testing to ensure data accuracy, integrity, and quality across every stage of the p...
Building Trust in Data I: A Guide to Effective Data Testing Frameworks
Introduction When should I test my data? How should it be done? Which tools and testing methods are most effective? Who holds responsibility for data quality?” These are common questions that arise in...
