- 307 Views
- 5 replies
- 1 kudos
Resolved! Why am I getting NameError name _all_timezones_unchecked' is not defined
I defined the following local time function get datetime: def get_sysdate(): jkt_tz = pytz.timezone('Asia/Jakarta') sysdate = datetime.now(jkt_tz).strftime('%Y-%m-%d %H:%M:%S') return sysdatespark.udf.register("get_sysdate", get_sysdate)But...
- 307 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Khaja_Zaffer I have connected with MS Support and explained: pytz is no thread safe package, I believe it will have some issue when executor init it parallelly. Second, this is a 3rd party lib, and suggest using the built-in library from ZoneInfo...
- 1 kudos
- 2017 Views
- 2 replies
- 0 kudos
Workspace allows dbf path to install in Databricks 16.4 LTS
Feature: Library installation using requirements.txt on DB Runtime 16.4 LTSAffected Areas: Workspace isolation, Library ManagementSteps to Reproduce:Upload a wheel file to dbfPut the requirements.txt file in the Workspace and put dbfs path in require...
- 2017 Views
- 2 replies
- 0 kudos
- 0 kudos
I would like to know if the workspace isolation is a priority, and only Databricks 14.3 and lower allow installation via DBFS.Why should the requirements.txt allow you to install libraries or packages via dbfs path?Could someone please explain why th...
- 0 kudos
- 2367 Views
- 2 replies
- 0 kudos
Batch reading from sql server tables with cdc on ssql server tables
Hi all,I need to do a batch load from sql server into Databricks. I have CC enabled on some tables. The simple appears to be union CDC and regular table to get a single set of records to load, but this appears to be fraught with risk of out of sequen...
- 2367 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes, you can use TVFs on Databricks. Please check the following link: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-select-tvf#gsc.tab=0Can you please elaborate on how you are loading the SQL Server Data into Databricks? H...
- 0 kudos
- 42 Views
- 2 replies
- 0 kudos
Access to Databricks Volumes via Databricks Connect not working anymore
Hi all, I use the extension to debug my python code regularly and since yesterday accessing files in the Databricks Volume isn't working anymore. The situation in the UI of Databricks is as follows:When I execute a glob statement to list all zip-file...
- 42 Views
- 2 replies
- 0 kudos
- 0 kudos
hi @JuliandaCruz Thank you for reaching out! I was able to reproduce your case while using Databricks Connect. The "Upload and Run file" option worked fine and returned results, which is essentially the same as running from the Databricks UI. Thou...
- 0 kudos
- 8838 Views
- 5 replies
- 1 kudos
SSRS Connect to Databricks
Hi ,I need to connect databricks query from microsoft SSRS.is it possible ? How do you make the connection?
- 8838 Views
- 5 replies
- 1 kudos
- 1 kudos
I am able to read databricks from SSRS using a personal token (corresponding to my account), but that's not good practice for an eventual production environment.Is there a way to connect not using a personal access token?Thanks!
- 1 kudos
- 10 Views
- 0 replies
- 0 kudos
Exposing Databricks API in Salesforce
Use Case:I want to expose a data bricks API URL in Salesforce, Salesforce will hit that exposed end point every time a record is created and data will be transferred from Salesforce to DatabricksWhen i try creating a serving end pointI am unable to s...
- 10 Views
- 0 replies
- 0 kudos
- 31 Views
- 2 replies
- 1 kudos
Job runs on serverless eventhough Job config has cluster definitions
Hi,I am defining the job along with job cluster specification using python sdk. But when the job runs it is using the serverless compute, instead of the defined cluster. I can say the job uses serverless from the job_run log and also from the system....
- 31 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks for checking @ManojkMohan. I found the issue in the job task definition. There is a job_clusters list in the job definition, where I provide the cluster config details. But this alone is not sufficient to have the task use the cluster. The job...
- 1 kudos
- 1323 Views
- 5 replies
- 2 kudos
How to select performance mode for Databricks Delta Live Tables
Hi everyone,Based on the official link,For triggered pipelines, you can select the serverless compute performance mode using the Performance optimized setting in the pipeline scheduler. When this setting is disabled, the pipeline uses standard perfor...
- 1323 Views
- 5 replies
- 2 kudos
- 2 kudos
I have learned that this parameter is not governed in the pipeline configuration itself, but in the job task that runs the pipeline. This is confusing to me and I don't like it.
- 2 kudos
- 3188 Views
- 1 replies
- 0 kudos
Not able to connect to Neo4j Aura Db from databricks
I am trying to connect to a Neo4j AuraDb instance-f9374927. Created a free professional instance of Neo4j. I am able to connect to this instance, add nodes and relationships. Created a Databricks shared cluster 14.3 LTS (includes Apache Spark 3.5.0...
- 3188 Views
- 1 replies
- 0 kudos
- 0 kudos
The connection issue between your Databricks cluster and Neo4j AuraDB instance (f9374927) with the ServiceUnavailableException: No routing server available message is tied to network-level SSL configuration and connectivity rather than incorrect code...
- 0 kudos
- 2752 Views
- 3 replies
- 0 kudos
Deploying using Databricks asset bundles (DABs) in a closed network
Hello, I'm trying to deploy DBX workflows using DABs using an Azure DevOps pipeline, in a network that cannot download the required terraform databricks provider package online, due to firewall/network restrictions.I have followed this post: https://...
- 2752 Views
- 3 replies
- 0 kudos
- 0 kudos
Another thing I noticed is, when running the 'databricks bundle debug terraform' command, it mentions these variables:I have tried setting these variables as environment variables in my ADO pipeline, specially the databricks terraform provider variab...
- 0 kudos
- 45 Views
- 1 replies
- 0 kudos
Lakebase / Feature Store error: “Failed to get identity details for username” (service principal)
Hello,I’m running into a Lakebase / Feature Store issue related to service principal authentication when trying to log or read from the Databricks Feature Store. Migrating from the legacy online tables. Here’s the exact error:psycopg2.OperationalErr...
- 45 Views
- 1 replies
- 0 kudos
- 0 kudos
The error you’re encountering —psycopg2.OperationalError: FATAL: Failed to get identity details for username: "user_uuid" — typically arises from an OAuth identity mismatch or invalid token scope when a Databricks service principal is used to authent...
- 0 kudos
- 51 Views
- 2 replies
- 0 kudos
DLT Pipeline with unknown deleted source data
Hello.. I need help. So the context is : - ERP data for company in my group is stored in sql tables - Currently, once per day we copy the last 2 months of data (creation date) from each table into our datalake landing zone (we can however do full cop...
- 51 Views
- 2 replies
- 0 kudos
- 0 kudos
Your solution #1 is very frustrating to me as well, for a number of reasons. Simply put, we have to be able to compare incoming data to target data for normal ETL operations. One way around this is to create a view of your target silver table, outsi...
- 0 kudos
- 31 Views
- 1 replies
- 0 kudos
DLT pipeline failed: streaming table query reading from an unexpected Delta table ID
Hi everyone,I'm running a DLT pipeline that loads data from Bronze to Silver using dlt.apply_changes(SCD type 2)The first run of the pipeline worked fine -- data was written successfully into the target Silver tables.However, when I ingested new data...
- 31 Views
- 1 replies
- 0 kudos
- 0 kudos
This “unexpected Delta table ID” error typically means your Delta Live Tables (DLT) pipeline detected that the underlying Delta table it was reading from has changed since the last checkpoint. When you use dlt.apply_changes() (for SCD Type 2), this i...
- 0 kudos
- 48 Views
- 2 replies
- 0 kudos
Community edition cluster - UI shows incorrect cores
Hi,I am a community edition user which gives me cluster ( as per below image)15GB of memory and 2 cores with one driver node ONLY.However,when I read a csv file of 181MB size,1) it generates 8 partitiones.As per default maxPartitionBytes is set to 12...
- 48 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you Louis for detailed explaination.Including notifying me about CE updates.However, I have noticed this ( below is the screenshot)spark.sql.files.minPartitionNum does not restun any result.Its wierd.Am I missing anything?Thanks
- 0 kudos
- 30 Views
- 1 replies
- 0 kudos
Unable to Add Multiple Columns in Single ALTER TABLE Statement on Iceberg Table via Unity REST Catal
Hello Databricks Team,I have implemented code to integrate the Iceberg Unity REST Catalog with the Teradata OTF engine and successfully performed read and write operations, following the documentation at https://docs.databricks.com/aws/en/external-ac...
- 30 Views
- 1 replies
- 0 kudos
- 0 kudos
This error stems from the Iceberg table metadata update constraints enforced by the Unity Catalog's REST API. Specifically, the Iceberg REST Catalog currently does not support multiple schema changes in a single commit. Each ALTER TABLE operation tha...
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 468 | |
| 349 | |
| 287 |
