- 1242 Views
- 4 replies
- 2 kudos
Resolved! Changing git's author field when committing through Databricks
I have a git folder to a Bitbucket repo. Whenever I commit something, the commit uses my Bitbucket username (the unique name) in the field author, making it less readable when I'm reading a list of commits.For example, commits end up like this: commi...
- 1242 Views
- 4 replies
- 2 kudos
- 2 kudos
Just found us suffering from the same issue since we enforced a GitHub ruleset to require commit emails to match our Org email pattern of `*@ourorgdomain.com`.
- 2 kudos
- 749 Views
- 1 replies
- 1 kudos
Streaming IoT Hub Data Using Delta Live Table Pipeline
Hello,I'm attempting to stream IoT Hub data using a delta live table pipeline. The issue I'm running into is that eventhub streaming isn't supported on shared clusters ([UNSUPPORTED_STREAMING_SOURCE_PERMISSION_ENFORCED] Data source eventhubs is not ...
- 749 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @dfish8124 ,Is is possible to share with us code?
- 1 kudos
- 838 Views
- 2 replies
- 0 kudos
Small json files issue . taking 2 hours to read 3000 files
Hello I am trying to read 3000 json files which has only one records. It is taking 2 hours to read all the files . How can I perform this operation faster pls suggest.
- 838 Views
- 2 replies
- 0 kudos
- 0 kudos
This is the code ---df1 = spark.read.format("json").options(inferSchema="true", multiLine="true").load(file1)
- 0 kudos
- 554 Views
- 2 replies
- 0 kudos
DLT Runtime Values
When my pipeline runs, I have a need to query a table in the pipeline before I actually create another table. I need to know the target catalog and target schema for the query. I figured the notebook might run automatically in the context of the cata...
- 554 Views
- 2 replies
- 0 kudos
- 0 kudos
can you set up notebook parameters and pass them in the DLT pipeline? https://docs.databricks.com/en/jobs/job-parameters.html
- 0 kudos
- 763 Views
- 4 replies
- 2 kudos
Obtain refresh mode from within Delta Live Table pipeline run
Is it possible to obtain somehow if a DLT pipeline run is running in Full Refresh or incremental mode from within a notebook running in the pipeline?I looked into the pipeline configuration variables but could not find anything.It would be benefitial...
- 763 Views
- 4 replies
- 2 kudos
- 2 kudos
We found a solution where we do not need to determine the refresh mode anymore. But I still do not know how to get the current refresh mode of the current pipeline run from within a notebook that is running in the pipeline. This may would still be be...
- 2 kudos
- 1075 Views
- 1 replies
- 0 kudos
AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Server failed to a
Actually My first part of the code works fine: dbutils.widgets.text("AutoLoanFilePath", "")inputPath = dbutils.widgets.get("AutoLoanFilePath")# inputPath = 'SEPT_2024/FAMILY_SECURITY'autoPath = 'dbfs:/mnt/dbs_adls_mnt/Prod_landing/'+inputPathautoLoa...
- 1075 Views
- 1 replies
- 0 kudos
- 0 kudos
This looks like an authentication issue when trying to access Azure Blob Storage from your Databricks environment. Can you please check the storage credentials and the setup? Consider using an Azure AD service principal with the appropriate RBAC rol...
- 0 kudos
- 393 Views
- 1 replies
- 0 kudos
How to detect gap in filenames (Autoloader)
So my files arrive at the cloud storage and I have configured an autoloader to read these files.The files have a monotonically increasing id in their name.How can I detect a gap and stop the DLT as soon as there is a gap?eg.Autoloader finds file1, in...
- 393 Views
- 1 replies
- 0 kudos
- 0 kudos
It doesn't seem like this can be done through the DLT autoloader. Particularly you require an automatic stop without manual intervention. You can write a custom Structured Streaming job and use a sequence-checking logic, and foreachBatch to process i...
- 0 kudos
- 505 Views
- 1 replies
- 0 kudos
Auto Loader with File Notification mode not picking up new files in Delta Live Tables pipeline
Dear,I am developing a Delta Live Table pipeline and use Auto Loader with File Notification mode to pick up files inside an Azure storage account (which is not the storage used by the default catalog). When I full refresh the target streaming table, ...
- 505 Views
- 1 replies
- 0 kudos
- 0 kudos
Based on the error "Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key", the pipeline was still trying to use an account key authentication method instead of service principal au...
- 0 kudos
- 812 Views
- 2 replies
- 0 kudos
parsing json string value column into dataframe structure
Hi All,I have to read kafka payload which has value column with json string. But the format of the json is as below.{ "data": [ { "p_al4": "N/A", "p_a5": "N/A", "p_ad": "OA003", "p_aName": "Abc", "p_aFlag": true ,....(dynamic)} ] }In data key it can ...
- 812 Views
- 2 replies
- 0 kudos
- 31522 Views
- 8 replies
- 19 kudos
cannot import name 'sql' from 'databricks'
I am working on Databricks version 10.4 premium cluster and while importing sql from databricks module I am getting below error. cannot import name 'sql' from 'databricks' (/databricks/python/lib/python3.8/site-packages/databricks/__init__.py).Trying...
- 31522 Views
- 8 replies
- 19 kudos
- 19 kudos
if you ever received this kind of error after installing the correct Python package then try running the below command. dbutils.library.restartPython()
- 19 kudos
- 905 Views
- 3 replies
- 0 kudos
Idempotency token
Hi Team, I have used idempotency token in my dag code to avoid duplicate runs.note: Idempotency token given as static valueIssue: If dag fails once ...because of this idempotency token, airflow is not allowing to connect dbx ...can you please help me...
- 905 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Vivian,Thanks for the response. Even i feel like it can be airflow issue. why because, even i dont have dbx job running at dbx end.. airflow still pointing to that idempotency token run id and results that error.current version we are using : 2.2....
- 0 kudos
- 7223 Views
- 8 replies
- 3 kudos
Ownership change for table using SQL
It's not possible to use the ALTER TABLE tblname OWNER TO serviceprinc1 command in Azure Databricks, as this isn't supported. I was trying to set a catalog table's ownership, but it resulted in an error. How can I achieve this using a script
- 7223 Views
- 8 replies
- 3 kudos
- 3 kudos
I was getting same error in python notebook and I found typo in my sql:Changing from ALTER TABLE table_name SET OWNER TO 'principal' to below fixed the issue. ALTER TABLE table_name SET OWNER TO `principal`
- 3 kudos
- 1590 Views
- 1 replies
- 0 kudos
Photon is not supported for a query
(1) LocalTableScan Output [11]: [path#23524, partitionValues#23525, size#23526L, modificationTime#23527L, dataChange#23528, stats#23529, tags#23530, deletionVector#23531, baseRowId#23532L, defaultRowCommitVersion#23533L, clusteringProvider#23534] Arg...
- 1590 Views
- 1 replies
- 0 kudos
- 0 kudos
sukanya09 Any solution on this?
- 0 kudos
- 792 Views
- 2 replies
- 1 kudos
Resolved! Notebook Import Failed Due to Workspace Quota Exceeded Error
I am using Databricks Community Edition. I am writing to seek assistance regarding an issue I encountered while attempting to upload a notebook to my Databricks Community Edition Workspace. I received the following error message:It appears that I hav...
- 792 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks . What is the maximum storage limit of Databricks Community Edition ?
- 1 kudos
- 800 Views
- 2 replies
- 0 kudos
Delta Live Table Pipeline Migration from one workspace to another workspace
Hi Team, Can you please help me to migrate Delta Live Table Pipeline Migration from one workspace to another workspace in automation way.Thanks,Padmaja M.
- 800 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Walter,Thank you for the information! Could you please guide me on where I can find more details about the idea DB-I-5757? Is there a public portal or forum where this idea is being tracked, or can I access updates on its status? Any additional in...
- 0 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
