- 1707 Views
- 1 replies
- 0 kudos
AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Server failed to a
Actually My first part of the code works fine: dbutils.widgets.text("AutoLoanFilePath", "")inputPath = dbutils.widgets.get("AutoLoanFilePath")# inputPath = 'SEPT_2024/FAMILY_SECURITY'autoPath = 'dbfs:/mnt/dbs_adls_mnt/Prod_landing/'+inputPathautoLoa...
- 1707 Views
- 1 replies
- 0 kudos
- 0 kudos
This looks like an authentication issue when trying to access Azure Blob Storage from your Databricks environment. Can you please check the storage credentials and the setup? Consider using an Azure AD service principal with the appropriate RBAC rol...
- 0 kudos
- 610 Views
- 1 replies
- 0 kudos
How to detect gap in filenames (Autoloader)
So my files arrive at the cloud storage and I have configured an autoloader to read these files.The files have a monotonically increasing id in their name.How can I detect a gap and stop the DLT as soon as there is a gap?eg.Autoloader finds file1, in...
- 610 Views
- 1 replies
- 0 kudos
- 0 kudos
It doesn't seem like this can be done through the DLT autoloader. Particularly you require an automatic stop without manual intervention. You can write a custom Structured Streaming job and use a sequence-checking logic, and foreachBatch to process i...
- 0 kudos
- 807 Views
- 1 replies
- 0 kudos
Auto Loader with File Notification mode not picking up new files in Delta Live Tables pipeline
Dear,I am developing a Delta Live Table pipeline and use Auto Loader with File Notification mode to pick up files inside an Azure storage account (which is not the storage used by the default catalog). When I full refresh the target streaming table, ...
- 807 Views
- 1 replies
- 0 kudos
- 0 kudos
Based on the error "Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key", the pipeline was still trying to use an account key authentication method instead of service principal au...
- 0 kudos
- 1473 Views
- 2 replies
- 0 kudos
parsing json string value column into dataframe structure
Hi All,I have to read kafka payload which has value column with json string. But the format of the json is as below.{ "data": [ { "p_al4": "N/A", "p_a5": "N/A", "p_ad": "OA003", "p_aName": "Abc", "p_aFlag": true ,....(dynamic)} ] }In data key it can ...
- 1473 Views
- 2 replies
- 0 kudos
- 36564 Views
- 8 replies
- 19 kudos
cannot import name 'sql' from 'databricks'
I am working on Databricks version 10.4 premium cluster and while importing sql from databricks module I am getting below error. cannot import name 'sql' from 'databricks' (/databricks/python/lib/python3.8/site-packages/databricks/__init__.py).Trying...
- 36564 Views
- 8 replies
- 19 kudos
- 19 kudos
if you ever received this kind of error after installing the correct Python package then try running the below command. dbutils.library.restartPython()
- 19 kudos
- 1701 Views
- 3 replies
- 0 kudos
Idempotency token
Hi Team, I have used idempotency token in my dag code to avoid duplicate runs.note: Idempotency token given as static valueIssue: If dag fails once ...because of this idempotency token, airflow is not allowing to connect dbx ...can you please help me...
- 1701 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Vivian,Thanks for the response. Even i feel like it can be airflow issue. why because, even i dont have dbx job running at dbx end.. airflow still pointing to that idempotency token run id and results that error.current version we are using : 2.2....
- 0 kudos
- 12005 Views
- 8 replies
- 3 kudos
Ownership change for table using SQL
It's not possible to use the ALTER TABLE tblname OWNER TO serviceprinc1 command in Azure Databricks, as this isn't supported. I was trying to set a catalog table's ownership, but it resulted in an error. How can I achieve this using a script
- 12005 Views
- 8 replies
- 3 kudos
- 3 kudos
I was getting same error in python notebook and I found typo in my sql:Changing from ALTER TABLE table_name SET OWNER TO 'principal' to below fixed the issue. ALTER TABLE table_name SET OWNER TO `principal`
- 3 kudos
- 2222 Views
- 1 replies
- 0 kudos
Photon is not supported for a query
(1) LocalTableScan Output [11]: [path#23524, partitionValues#23525, size#23526L, modificationTime#23527L, dataChange#23528, stats#23529, tags#23530, deletionVector#23531, baseRowId#23532L, defaultRowCommitVersion#23533L, clusteringProvider#23534] Arg...
- 2222 Views
- 1 replies
- 0 kudos
- 0 kudos
sukanya09 Any solution on this?
- 0 kudos
- 1259 Views
- 2 replies
- 1 kudos
Resolved! Notebook Import Failed Due to Workspace Quota Exceeded Error
I am using Databricks Community Edition. I am writing to seek assistance regarding an issue I encountered while attempting to upload a notebook to my Databricks Community Edition Workspace. I received the following error message:It appears that I hav...
- 1259 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks . What is the maximum storage limit of Databricks Community Edition ?
- 1 kudos
- 1198 Views
- 2 replies
- 0 kudos
Delta Live Table Pipeline Migration from one workspace to another workspace
Hi Team, Can you please help me to migrate Delta Live Table Pipeline Migration from one workspace to another workspace in automation way.Thanks,Padmaja M.
- 1198 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Walter,Thank you for the information! Could you please guide me on where I can find more details about the idea DB-I-5757? Is there a public portal or forum where this idea is being tracked, or can I access updates on its status? Any additional in...
- 0 kudos
- 3431 Views
- 1 replies
- 3 kudos
How to use Pylint to check your pyspark code quality ?
Hi guys,I would like to use the Pylint to check my pyspark scripts, do you do that ?Thank you ?
- 3431 Views
- 1 replies
- 3 kudos
- 3 kudos
Currently I am working on Databricks (Notebooks) and have the same issue as unable to find a linter that is well integrated with Python, Pyspark and databricks notebooks.
- 3 kudos
- 941 Views
- 1 replies
- 0 kudos
Resolved! Creating notebooks which work on both normal databricks jobs as well as dlt pipeline
We are working on automation of our databricks ingestion. We want to make our python scrips or notebooks such that they work on both databricks jobs and dlt pipelines.When i say databricks jobs it means normal run without dlt pipeline.How shall we wo...
- 941 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ashraf1395, To address your goal of creating Python scripts or notebooks that work both in Databricks Jobs and Delta Live Tables (DLT) pipelines, here are some ideas: Unified Script Approach:Table Creation: As you mentioned, DLT supports two t...
- 0 kudos
- 1863 Views
- 1 replies
- 1 kudos
Resolved! Create a Delta Table with PK and FK constraints for a streaming source data
1. How to create a Delta Table with PK and FK constraints for a streaming source data?2. When the streaming data in the silver layer gets updated, will the delta table also be updated?My use case is:We have a streaming data in the silver layer as SCD...
- 1863 Views
- 1 replies
- 1 kudos
- 1 kudos
1. You can use primary key and foreign key relationships on fields in Unity Catalog tables. Primary and foreign keys are informational only and are not enforced. Foreign keys must reference a primary key in another table. You can declare primary keys...
- 1 kudos
- 2253 Views
- 4 replies
- 1 kudos
java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.trees.Origin.<init>
While trying to run spark.sql("CALL iceberg_catalog.system.expire_snapshots(table => 'iceberg_catalog.d11_stitch.rewards_third_party_impact_base_query', older_than => TIMESTAMP '2024-03-06 00:00:00.000')") to this Im getting Py4JJavaError: An erro...
- 2253 Views
- 4 replies
- 1 kudos
- 1 kudos
@ashutosh0710 you can inspect the libraries available in a cluster at /databricks/jars to understand which jars and versions are available, or similarly simply inspect the Spark UI Environment Classpath list.
- 1 kudos
- 1059 Views
- 1 replies
- 0 kudos
Issue with Updating Dashboards During Assessment Using UCX
Hello everyone,I am using UCX for the migration to Unity, and I've noticed that re-running the assessment does not update the dashboards with jobs that are incompatible with Unity. To get the dashboards updated, I had to uninstall and reinstall UCX, ...
- 1059 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,I am having a similar issue as Domenico. I have installed UCX in my workspace and ran the UCX_assesment job 3 days back. It created crawl_table.log which showed that UCX had scanned through all of my legacy hive_metastore schemas and listed the ta...
- 0 kudos
| User | Count |
|---|---|
| 1632 | |
| 790 | |
| 512 | |
| 349 | |
| 287 |
