- 2203 Views
- 2 replies
- 2 kudos
Autoloader Error Loading and Displaying
Hi there,I'd appreciate some assistance with troubleshooting what is supposed to be a (somewhat) simple use of autoloader. Below are some screenshots highlighting my issue:When I attempt to create the dataframe via spark.readStream.format("cloudFiles...
- 2203 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @ChristianRRL This is a common issue with Spark Structured Streaming and the display() function.The error occurs because you're trying to display a streaming DataFrame, which requires special handling. Here are several solutions:1. Use writeStrea...
- 2 kudos
- 11409 Views
- 2 replies
- 0 kudos
Resolved! DB2 JDBC Connection from Databricks cluster
Has anyone successfully connected to a DB2 database on ZOS from a Databricks cluster using a JDBC connection?I also need to specify an SSL certificate path and not sure if I need to use an init script on the cluster to do so.Any examples would be ver...
- 11409 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Karl , Greetings! I've outlined the steps below to connect from Databricks to IBM DB2 using JDBC:Step 1: Obtain the DB2 JDBC Driver Visit the IBM website to download the appropriate JDBC driver for DB2 on z/OS.Reference Document: IBM DB2 JDBC Dr...
- 0 kudos
- 1812 Views
- 2 replies
- 2 kudos
Resolved! Delta sharing to pandas error
we are on a trial databricks premium workspace (fully managed by databricks)we are trying to test Delta Sharing feature, where we are sharing UC table with a recipient using python client (outside of databricks). We are using `delta-sharing` python l...
- 1812 Views
- 2 replies
- 2 kudos
- 2 kudos
Yes - you are right, works on trial - i was able to setup S3 as an external location and configured the UC schema to use this S3 ext location, and then was able to query data from an external python client.
- 2 kudos
- 6415 Views
- 3 replies
- 0 kudos
create databricks scope by reading AWS secrets manager
Hi, I have datbricks on AWS, I created some secrets in AWS Secrets Manger, I would need to create the scopes based on AWS secrets manager.When I use Azure's Key Vault, when creating the scope, it uses the option -scope-backend-type AZURE_KEYVAULT, bu...
- 6415 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @thiagoawstest Step 1: Create Secret ScopeYou can create a secret scope using the Databricks REST API as shown below: python import requests import json # Define the endpoint and headers url = "https://<databricks-instance>/api/2.0/secrets/scope...
- 0 kudos
- 1565 Views
- 1 replies
- 0 kudos
Liquid clustering not improved performance
Hi There,I have a table of 160 GB with partition applied on country and yearmonth columns, I maintain a previous history of 6 years and replace the partitions (latest 2 months) to add the new data.I use overwrite mode to replace the effected partitio...
- 1565 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @SusmithaBadam, based on your use case, partitioned tables are performing better because they work kind of like labeled folders. When you group by, it can quickly go to the exact folder instead of scanning everything, so it’s much faster.Liquid cl...
- 0 kudos
- 2250 Views
- 2 replies
- 0 kudos
Issue with Resetting Checkpoint Metadata in DLT with Unity Catalog
Hi Community,Hope someonne can help with this DLT question.I am currently working in a Databricks environment using Delta Live Tables (DLT) with Unity Catalog enabled, and I'm encountering a blocker related to schema evolution and checkpoint metadata...
- 2250 Views
- 2 replies
- 0 kudos
- 0 kudos
I feel you.Probably not the way to go, but did to try to destroy and re-deploy your pipeline?
- 0 kudos
- 7616 Views
- 4 replies
- 3 kudos
Resolved! Best Practices for Designing Bronze Layer with SQL Server Source in Medallion Architecture
Hi Databricks Experts,I'm working on a Medallion Architecture implementation in Databricks, where the source data is coming from SQL Server. I would like some advice on how to handle the bronze layer correctly and cost-effectively.Should I create a b...
- 7616 Views
- 4 replies
- 3 kudos
- 3 kudos
Create the bronze table using Auto Loader and store it in Delta format. Although it might seem like you'll only read from bronze once to populate the silver layer, in real-world production scenarios, you'll often need to re-read from bronze—for repro...
- 3 kudos
- 4669 Views
- 3 replies
- 0 kudos
git integration with volumes?
Volumes in unity are said to be ideal for "storing library and config files of arbitrary formats such as .whl or .txt centrally and providing secure access across workspaces to it".So basically this is the same we had with dbfs but with decent access...
- 4669 Views
- 3 replies
- 0 kudos
- 0 kudos
I totally get where you’re coming from! Managing config files with access control is crucial, but the lack of Git integration in Unity volumes can be a bit frustrating. In my experience, finding a smooth workflow can take time. I’ve found using tools...
- 0 kudos
- 2436 Views
- 4 replies
- 0 kudos
When formatting dates using the yyyyMMddHHmmssSSS pattern, an error occurred
An error occurred while converting a timestamp in the yyyyMMddHHmmssSSS formatfrom pyspark.sql.functions import to_timestamp_ntz, col, lit df = spark.createDataFrame( [("20250730090833000")], ["datetime"]) df2 = df.withColumn("dateformat", to_t...
- 2436 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @liu ,I think it could be related to following bug in Java. I suspect that internally to_timestamp_ntz uses DateTimeFormatter.[JDK-8031085] DateTimeFormatter won't parse dates with custom format "yyyyMMddHHmmssSSS" - Java Bug SystemNow what's inte...
- 0 kudos
- 3520 Views
- 1 replies
- 1 kudos
Resolved! HuggingFace bert-large-uncased gives NameError
Hello,I am trying to run the LLM bert-large-uncased from HuggingFace.I have downloaded the transformers from github. I have installed the various packages. I am now getting an error message: NameError: name 'torch' is not defined.Not sure what the pr...
- 3520 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Carl_B The error indicates that PyTorch is not installed in your environment. Just try below this.Install PyTorchOption 1: Using pippip install torch torchvision torchaudioOption 2: Using conda (if you're using Anaconda/Miniconda)conda install py...
- 1 kudos
- 10812 Views
- 5 replies
- 1 kudos
Performance Issue – Writing Large Dataset to ADLS from Oracle via JDBC
Hi there, I am currently working on a notebook where I pull data from an Oracle database using an Oracle SQL script with a JDBC connection. Due to the large dataset size and joins in my query, I’ve implemented the /*+ parallel(n) */ hint, which works...
- 10812 Views
- 5 replies
- 1 kudos
- 1 kudos
I can suggest few tweaks in the compute, the current D series is good enough, but we are handling huge data, please try bumping up minimum workers from 1 to at least 4; change the VM type - to a bigger one - Standard_E64ds_v5, and if not try to use a...
- 1 kudos
- 3254 Views
- 7 replies
- 0 kudos
Cannot Remove Table from Delta Share
Hello! We have some production data pipelines using delta shares that have previously worked but broken today and I can't find a new solution. I create a delta share: CREATE SHARE IF NOT EXISTS `local_new_2022` Then, like normal, I add a table to th...
- 3254 Views
- 7 replies
- 0 kudos
- 0 kudos
We are using the databricks_sql python sdk: ```pythonfrom databricks import sql as databricks_sqlconnection = databricks_sql.connect( server_hostname=server_hostname, http_path=http_path, access_token=access_token)with connection.cursor() as...
- 0 kudos
- 1348 Views
- 2 replies
- 3 kudos
Unity Catalog not detected until I manually restarted the cluster
I just spent a lot of time setting up Unity Catalog with external locations, access connectors, and all the right permissions. My cluster had Dedicated access mode, Unity Catalog has been assigned properly, and everything was configured correctly. Bu...
- 1348 Views
- 2 replies
- 3 kudos
- 3 kudos
Appreciate you posting this, @Dharshan15! It’s something that can easily be overlooked. After enabling Unity Catalog, the cluster needs to be restarted for the changes to take effect. You can also share this feedback directly with the Databricks team...
- 3 kudos
- 752 Views
- 1 replies
- 0 kudos
How to get a programmatically get he non dlt pipeline logs.
We are triggering the jobs which notebooks. Notebook is consist of python and sql queries. We need to read the error messages when it is triggered using an job thanksprasanna
- 752 Views
- 1 replies
- 0 kudos
- 0 kudos
Use some logging to send the error logs to a delta table or file. Can you please share the code. I wanted to see what you are trying to do.
- 0 kudos
- 4931 Views
- 5 replies
- 3 kudos
Resolved! Problems and questions with deploying Lakeflow Declarative Pipeline using Databricks Bundles
Hi all,I met some problems and have some questions about deploying Lakeflow Declarative Pipeline using Databricks Bundles. Could anyone kindly help?Below is my current bundle resource file for the pipeline: resources: pipelines: dbr_d365_crm_p...
- 4931 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @AlbertWang ,I think some of those issues could be related to your databricks assets bundle version. For example the glob thing is in Beta. It could be available in UI, but not in your version of databricks cli.The same applies for root_path: As o...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
