- 1139 Views
- 1 replies
- 0 kudos
Serverless compute for file notification mode
I am creating a table that ingests data from aws s3 using the 'file notification mode'. With a single user cluster, it works. I would like to use Serverless compute, but I get an error about authentication. Is it possible to do this, or are there alt...
- 1139 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Isa1, Using Serverless compute with Auto Loader in file notification mode can indeed present authentication challenges. Based on the context provided, here are some insights and alternatives: Authentication Issues with Serverless Compute:Server...
- 0 kudos
- 1254 Views
- 1 replies
- 0 kudos
Unity Catalog with Structured Streaming
Hi,Our project uses spark structured streaming scala notebooks to process files stored in an S3 bucket, with the jobs running in Single User access mode.For one of the jobs, we need to use a file arrival trigger. To enable this, the S3 location must ...
- 1254 Views
- 1 replies
- 0 kudos
- 0 kudos
@elkaganeva, When you register an S3 bucket as an external location in Unity Catalog, you can directly access Delta tables stored in that bucket using the spark.readStream and spark.writeStream methods. The metadata for the Delta tables is managed th...
- 0 kudos
- 5337 Views
- 3 replies
- 1 kudos
[Databricks Asset Bundles] Changes are not showing when deploying for a second time
Hi allI've followed this guide https://docs.databricks.com/en/dev-tools/bundles/work-tasks.html and managed to deploy a notebook using DABs, but I then changed the cluster settings and ran the deploy line again and it didn't change the cluster.I dele...
- 5337 Views
- 3 replies
- 1 kudos
- 1 kudos
Apologies if I'm running these in the wrong place but it doesn't seem to find databricks bundle clean or databricks bundle build - it shows:Usage: databricks bundle [command] Available Commands: deploy Deploy bundle deployment Deployment re...
- 1 kudos
- 2635 Views
- 3 replies
- 0 kudos
Matillion ETL using serverless
Can we attach serverless compute to Matillion ETL transformation jobs while running on Databricks workspace? I am aware that we can attach a job compute cluster but not sure whether we can attach serverless compute?
- 2635 Views
- 3 replies
- 0 kudos
- 2187 Views
- 2 replies
- 1 kudos
Resolved! Need an advice of someone with practical experience in DLT
Hi, I'm facing this scenario in my DLT pipeline where in my silver layer I'm doing some filtering to prevent my test data to go to silver schema and then in the end i'm using apply_changes to create the tables and I'm using sequence_by clause within ...
- 2187 Views
- 2 replies
- 1 kudos
- 1 kudos
To address the issue of outdated records moving forward to the silver schema in your Delta Live Tables (DLT) pipeline, you can consider the following approach Modify the Filtering Logic: Instead of filtering out the test records before the apply_chan...
- 1 kudos
- 2201 Views
- 2 replies
- 1 kudos
Autoloader file notification mode error using UC
We have a DLT pipeline we've created that is using autoloader file notification mode. The pipeline ran fine before moving it to UC. Now that we're using UC, we are getting an AWS permissions issue when the autoloader file notification mode is set to ...
- 2201 Views
- 2 replies
- 1 kudos
- 1 kudos
Please make sure you are using Dedicated (single-user) clusters when authenticating to the file notifications service when attempting to authenticate to SQS via instance profile authentication. This likely will change in the future, so stay posted.
- 1 kudos
- 5867 Views
- 4 replies
- 5 kudos
Resolved! Databricks -->Workflows-->Job Runs
In Databricks -->Workflows-->Job Runs we have a column "Run As".From where does this value come. We are getting a user id here but need to change it to a generic account. Any help would be appreciated. Thanks
- 5867 Views
- 4 replies
- 5 kudos
- 5 kudos
I'm surprised why there no options to select "Run as" as something like "system user". Why all this complication with Service Principal? Where to report this ?@DataBricks
- 5 kudos
- 952 Views
- 1 replies
- 0 kudos
Version Control For Alerts, Queries
Is there any inbuilt option for version control for Databricks SQL Queries and Alerts? Tried moving the files into a repo and Git did not recognize the file types.
- 952 Views
- 1 replies
- 0 kudos
- 0 kudos
Currently, Databricks does not have an inbuilt option for version control specifically for SQL Queries and Alerts.
- 0 kudos
- 1268 Views
- 1 replies
- 0 kudos
Resolved! Sending customized mail with databricks notebook with images
How can i send customized message from within the databricks notebook. SQL alerts is not helping.
- 1268 Views
- 1 replies
- 0 kudos
- 0 kudos
You can refer to KB https://kb.databricks.com/en_US/notebooks/send-email
- 0 kudos
- 3512 Views
- 6 replies
- 2 kudos
Resolved! SQL run on cluster creates table different to SQL Warehouse endpoint
I have a Personal cluster version 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12) and a SQL Warehouse in a databricks environment. When I use the following code to create a table in a catalog, it gives me different column types when run on the cl...
- 3512 Views
- 6 replies
- 2 kudos
- 2 kudos
It seems that as per docs as of now this setting is only true by default in warehouses in clusters it is set to false: https://docs.databricks.com/en/sql/language-manual/sql-ref-ansi-compliance.html#ansi-compliance-in-databricks-runtime
- 2 kudos
- 1164 Views
- 1 replies
- 0 kudos
- 1164 Views
- 1 replies
- 0 kudos
- 0 kudos
HiYou can achieve that with pandas. See the following example code:%pip install openpyxlimport pandas as pd file_location_xls = "path/to/excel/1.xlsx" # read the sheet with Name Financials1" into a pandas dataframe pdf = pd.read_excel(file_location...
- 0 kudos
- 4952 Views
- 1 replies
- 0 kudos
How to handle timeout exception in Error Handle Task
Dear team, I have a workflow like this, task_a, task_b and handle_error. How I handle any timeout exception from task_a and task_b or any future task in future and log into handle error task at the end.Best regards,Jensen Nguyen
- 4952 Views
- 1 replies
- 0 kudos
- 0 kudos
@NhanNguyen thanks for your question! Have you maybe consider to: Define a Global Error Task: Add a handle_error task in the workflow that runs conditionally on task failure.Set Failure Conditions: In the UI or through JSON, configure the handle_erro...
- 0 kudos
- 5109 Views
- 1 replies
- 0 kudos
Static Parameters in Feature Functions
Hi,I'm implementing a machine learning pipeline using feature stores and I'm running into a limitation with feature functions. I'd like to perform multiple calculations on my columns with some minor adjustments, but I need to pass a static parameter ...
- 5109 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @NaeemS thanks for your question! Yes, you can pass a static parameter to a feature function to control its behavior in Databricks Feature Store. This allows you to perform multiple calculations on your columns with minor adjustments without defin...
- 0 kudos
- 2337 Views
- 1 replies
- 0 kudos
ORA-01830: date format picture ends before converting entire input string
Hi,Recently we are migrating our hive metastore workloads to unity catalog. As part of which, se are running into an following error with 15.4 DBR (UC) version where as it's working with 10.4 DBR (Hive). The requirement is to read the data from a tab...
- 2337 Views
- 1 replies
- 0 kudos
- 0 kudos
@Kaviprakash thanks for your question! Is this also probably specific to a cluster type ? Shared vs Single user? If Shared Mode, can you please try restarting your cluster with the following Spark Configuration:spark.connect.perserveOptionCasing true...
- 0 kudos
- 9007 Views
- 1 replies
- 1 kudos
Data Lineage without Spark, but with Polars (and Delta Lake) instead
Some context: I am completely new to Databricks; have heard good stuff, but also some things that worry me.One thing that worries me is the performance (and eventual costs) of running Spark with smaller (sub 1TB) datasets. However, one requirement fr...
- 9007 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @dh thanks for your question! I believe It’s possible to run Polars with Delta Lake on Databricks, but automatic data lineage tracking is not native outside of Spark jobs. You would likely need to implement custom lineage tracking or integrate ext...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
