- 7146 Views
- 6 replies
- 4 kudos
Resolved! Write Spark DataFrame into OpenSearch
Hi Databricks Community,I'm trying to read an index from OpenSearch or write a DataFrame into an OpenSearch index using the native Spark OpenSearch connector:host = dbutils.secrets.get(scope="opensearch", key="host") port = dbutils.secrets.get(scope=...
- 7146 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi,I am getting the same error and i also was able to connect using opensearch-py.I also founded in this doc https://github.com/opensearch-project/opensearch-hadoop/blob/main/README.md#requirements that i need to have some jars i already add it witho...
- 4 kudos
- 4536 Views
- 5 replies
- 3 kudos
Resolved! I can't use my own .whl package in Databricks app with databricks asset bundles
I am building a databricks app using databricks asset bundles. I need to use a helpers packages that i built as an artifact and using in other resources outside the app. The only way to use it is to have the built package inside the app source code f...
- 4536 Views
- 5 replies
- 3 kudos
- 3 kudos
Thank you very much. I hope translate your tips to my case which does not use asset bundles.
- 3 kudos
- 1067 Views
- 2 replies
- 1 kudos
DLT Autoloader schemaHints from JSON file instead of inline list?
Hi @Witold, @Hubert-Dudek,I’m using a DLT pipeline to ingest realtime data from Parquet files in S3 into Delta tables using Auto Loader. The pipeline is written in SQL notebooks.Problem:Sometimes decimal columns in the Parquet files get inferred as I...
- 1067 Views
- 2 replies
- 1 kudos
- 1 kudos
- dlt use automatically cloudFiles.schemaLocation So the schema is stored automatically, and in many cases, it will be stable, but it does not - keep using cloudFiles.schemaHints, but just load JSON to a variable and pass that variable (I guess you w...
- 1 kudos
- 1049 Views
- 3 replies
- 2 kudos
Resolved! how to avoid extra column after retry upon UnknownFieldException
With autoloader.option("cloudFiles.schemaEvolutionMode", "addNewColumns") I have done retry after getting org.apache.spark.sql.catalyst.util.UnknownFieldException: [UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during par...
- 1049 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Hubert-Dudek the input is csv. readStream reads csv with .option("cloudFiles.inferColumnTypes", "true"). then, df.toDF( ) is called to rename the column name. The original csv header has space, that's why error message has "test 1_2 Prime". The r...
- 2 kudos
- 4702 Views
- 5 replies
- 2 kudos
How to run a python task that uses click for CLI operations
Hey,in my application I am using click to facilitate CLI operations. It works locally, in notebooks, when scripts are run locally, but it fails in Databricks. I defined a task that, as an entrypoint, accepts the file where the click-decorated functio...
- 4702 Views
- 5 replies
- 2 kudos
- 2 kudos
Hey guys,I think I managed to find a workaround. I will leave it here for everyone that is seeking the same answers, including future me.What I did is basically this piece of code:def main(): try: assign_variants(standalone_mode=False) ...
- 2 kudos
- 1582 Views
- 4 replies
- 6 kudos
Resolved! ONLY PNG format is available for databricks dashboard table download
I couldn't download the data straight from databricks dashboards in csv format starting from last night (night of Dec 1st, 2025). The only format that is available right now is PNG. I've tried downloading the data on multiple browsers but only the PN...
- 1582 Views
- 4 replies
- 6 kudos
- 6 kudos
Hey @Advika you saya quick workaround is to right-click and download the CSV from there.What do you mean? Where? I am right clicking all over my dashboard widget and don't see CSV download option. Can you be more specific?
- 6 kudos
- 1892 Views
- 7 replies
- 3 kudos
Resolved! Declarative Pipelines: set Merge Schema to False
Dear Team!I want to prevent at a certain table that the schema is automatically updated. With plain strucutred streaming I can do the following:silver_df.writeStream \ .format("delta") \ .option("mergeSchema", "false") \ .option("checkpoi...
- 1892 Views
- 7 replies
- 3 kudos
- 3 kudos
It is automatic in DLT. If there are significant schema changes, you need to full refresh. Maybe consider storing everything (the whole JSON) in a single VARIANT column and unpacking only what is necessary later - this way you will have it under cont...
- 3 kudos
- 1419 Views
- 3 replies
- 2 kudos
Resolved! API Call to return more than 100 jobs
Hello I have around 150 jobs and this is likely to increase.I use this call to get all the jobs and write them into a list called json.My logic here is to match a name to a job id and run the job using the job id. response = requests.get(hostHTTPS, j...
- 1419 Views
- 3 replies
- 2 kudos
- 1965 Views
- 3 replies
- 5 kudos
Resolved! user standard serverless with asset bundle on Azure
Anyone running into issues with using standard serverless with Asset bundle we tried all options with below line performance_target: STANDARDbut it ignore above value and uses performance optimized cluster which is not expected any lead with ri...
- 1965 Views
- 3 replies
- 5 kudos
- 5 kudos
resources: jobs: my_dabs: performance_target: STANDARD Please check whether it is on the correct level in the YAML. Also consider updating the CLI. I've just tested it, and it worked properly.
- 5 kudos
- 975 Views
- 3 replies
- 4 kudos
Resolved! Getting below error when trying to create a Data Quality Monitor for the table. ‘Cannot create Monit
Getting below error when trying to create a Data Quality Monitor for the table.‘Cannot create Monitor because it exceeds the number of limit 500'.
- 975 Views
- 3 replies
- 4 kudos
- 4 kudos
Maybe this is not a situation, but trial accounts have lower quotas. Tried to find the quota related to the monitor in "databricks resource-quotas list-quotas" but couldn't find it.Your account contact in databricks can probably adjust it or find wit...
- 4 kudos
- 3735 Views
- 4 replies
- 4 kudos
Resolved! How to create a Unity Catalog Connection to SQL Server using service principal??
I am trying to use the Databricks Python SDK (v 0.63.0) to create a Unity Catalog connection to a Azure-hosted SQL Server database using an Azure service principal to authenticate. I have successfully done this via the Workspace UI, but I am trying t...
- 3735 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @ScottH ,You need to configure it in following way (I've tested it and it works). In a place where a red arrow is pointing you need to provide your own tenant_id:
- 4 kudos
- 4939 Views
- 5 replies
- 0 kudos
Unity Catalog is missing column in Catalog Explorer
I have just altered one of my tables and added a column.ALTER TABLE tpch.customer ADD COLUMN C_CUSTDETAILS struct<key:string,another_key:string,boolean_key:boolean,extra_key:string,int_key:long,nested_object:struct<more:long,arrayOne:array<string>>>A...
- 4939 Views
- 5 replies
- 0 kudos
- 0 kudos
I had this error just recently. I did basically the same table transformation 4 times over the course of 2 days. Added two new fields to two different tables using the same SQL syntax. It worked 3 out of 4 times, but for 1 the column is not available...
- 0 kudos
- 2200 Views
- 3 replies
- 2 kudos
Lakebridge code conversion from oracle to databricks sql
Hi Community,I’m working on migrating several Oracle views to SparkSQL using the Databricks Labs Lakehouse Bridge tool. I’m facing issues while converting the code .For oracle views and materialized views .Problems I’m encountering:The converted SQL...
- 2200 Views
- 3 replies
- 2 kudos
- 2 kudos
@Louis_Frolio - will there be similar guidelines for other 'source' code too such as T-SQL, Teradata. Any such comprhensive docs on Laeebridge to cover per source.
- 2 kudos
- 2358 Views
- 1 replies
- 2 kudos
Resolved! Moving files using DBUtils is so slow
I am using the platform DBUtils.fs.mv() on databricks clusters, and facing issues with move operation slowness.I move files in UC Volumes or ADLS storage abfss links, which work but is so slow.I mean it takes hours to transfer files that used to take...
- 2358 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @murtadha_s , here are some helpfult tips and hints to help you further diagnose the slowness. Totally expected behavior here: object-storage moves with dbutils.fs.mv will be much slower than HDFS. Under the hood, dbutils isn’t doing an atom...
- 2 kudos
- 859 Views
- 1 replies
- 0 kudos
Serverless Compute – ADLS Gen2 Authorization Failure with RBAC
We are facing an authorization issue when using serverless compute with ADLS Gen2 storage. Queries fail with:Code AbfsRestOperationException: Operation failed: "This request is not authorized to perform this operation.", 403 AuthorizationFailureDetai...
- 859 Views
- 1 replies
- 0 kudos
- 0 kudos
private link from serverless, as probably you are not allowing public internet access. Configure private connectivity to Azure resources - Azure Databricks | Microsoft Learn you need to add both dfs and blob
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
