- 5881 Views
- 3 replies
- 2 kudos
Resolved! Autoloader maxFilesPerTrigger not working correctly
Hi,am trying to apply batch size in autoloader and code is as below. But its picking all the changes in one go even if I have put maxFilesPerTrigger as 10. Appreciate any help.(spark.readStream.format("json").schema(streamSchema).option("cloudFiles.b...
- 5881 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Sanjay Jain​ , Since you have provided the trigger as once, the maxFilesPerTrigger will not take effect here. With trigger once, all the files will be read together. You need to change the trigger for this option to come into effect.Please refer ...
- 2 kudos
- 4178 Views
- 4 replies
- 1 kudos
sample
Help parsing the JSON using Spark SQL or python. Sample json attached.
- 4178 Views
- 4 replies
- 1 kudos
- 3046 Views
- 4 replies
- 3 kudos
Can we share exam voucher to another databricks account
Hi, I received free voucher for lakehouse webinar, My friend also got free voucher, by any chance can i use my friend voucher to shedule another exam for me.
- 3046 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi guysUnfortunately, it is not possible to share an exam voucher with another Databricks account. Exam vouchers are typically tied to specific accounts or individuals and cannot be transferred or shared. Free Fire
- 3 kudos
- 9312 Views
- 2 replies
- 3 kudos
Resolved! Access Excel file in delta live pipeline
I'm having an issue accessing the excel through dlt pipeline. the file is in ADLS I'm using pandas to read the Excel. It seems pandas are not able to understand abfss protocol is there any way to read Excel with pandas in dlt pipeline?I'm getting thi...
- 9312 Views
- 2 replies
- 3 kudos
- 3 kudos
Thanks for the info. It works just need to install an additional library using "%pip install openpyxl".
- 3 kudos
- 2981 Views
- 1 replies
- 1 kudos
Resolved! Is there any maintenance (patches , upgrade for VMs created by DataBricks on Azure) from DataBricks
We are using Databricks on Azure. Infra team noticed we have some VMs created in the past for DataBricks clusters on version Linux (ubuntu 18.04). Is there maintenance previewed for that, upgrade? Are there any patches for created in Azure objects by...
- 2981 Views
- 1 replies
- 1 kudos
- 1 kudos
Finally while I was posting this question, AzureDataBricks upgraded VMs to the supported version 20, not the latest , 22. It was a week after old version was no longer supported by Microsoft
- 1 kudos
- 13013 Views
- 6 replies
- 7 kudos
Resolved! PDF Attachment on an Alert
Currently my Alert is an HTML table using data pointing to an SQL query.I was wondering if it is possible to attach the resulting table from this SQL query as a PDF to the alert email.If anyone has successfully implemented this, please let me know! T...
- 13013 Views
- 6 replies
- 7 kudos
- 7 kudos
Ok understood the concern, so basically the issue is with PDF rendering as much I understood. Let me know if I am wrong. Let me see if there is any improvement by our engineering team on this front.
- 7 kudos
- 2168 Views
- 2 replies
- 0 kudos
Registering a dataframe coming from a CDC data stream removes the CDC columns from the resulting temporary view, even when explicitly adding a copy of the column to the dataframe.
df_source_records.filter(F.col("_change_type").isin("delete", "insert", "update_postimage")) .withColumn("ROW_NUMBER", F.row_number().over(window)) .filter("ROW_NUMBE...
- 2168 Views
- 2 replies
- 0 kudos
- 0 kudos
Seems to work now actually. No idea what changed, as I tried multiple times exactly in this way and it did.not.work.from pyspark.sql.functions import expr from pyspark.sql.utils import AnalysisException import pyspark.sql.functions as f data = [(...
- 0 kudos
- 3462 Views
- 4 replies
- 2 kudos
How to replace the SSO certifcate on our workspace?
We have Azure AD SSO setup on our workspace but the three year certificate is due to expire on Monday. I have logged onto the Admin Console & Single Sign-on tab. All the options are greyed out and there is no update or edit buttons as can be seen in ...
- 3462 Views
- 4 replies
- 2 kudos
- 2 kudos
@Debayan​ our version is branch-3.96-1682169174-f2e2f130 if this helps any?
- 2 kudos
- 9079 Views
- 1 replies
- 0 kudos
Run result unavailable: run failed with error message Notebook not found:
I'm trying to create a workflow job that fetches the notebook from a remote git repository (Bitbucket cloud)I tried everything in the Path field and nothing is working. Note that the bitbucket repo is connected to databricks already and no issues che...
- 9079 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @harraz (Customer)​ , Could you please confirm if files in repos has been enabled? https://docs.databricks.com/files/workspace.html#configure-support-for-files-in-repos.You can use the command %sh pwd in a notebook inside a repo to check if Files ...
- 0 kudos
- 2438 Views
- 2 replies
- 0 kudos
how to setup the path to a remote notebook in bitbucket to run as a jobI tried everything in the path and nothing is workingI keep getting this error:...
how to setup the path to a remote notebook in bitbucket to run as a jobI tried everything in the path and nothing is workingI keep getting this error:Run result unavailable: run failed with error message Notebook not found:Note that I already connec...
- 2438 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @mohamed harraz​ , Could you please confirm if files in repos has been enabled? https://docs.databricks.com/files/workspace.html#configure-support-for-files-in-repos.You can use the command %sh pwd in a notebook inside a repo to check if Files in...
- 0 kudos
- 1365 Views
- 1 replies
- 1 kudos
Return notebook path from job that is run remotely from the repo
I'm wanting to set up some email alerts for issues in the data as a part of a job run. I am wanting to point the user to the notebook that the issue occurred in. I think this would be simple enough but another layer is that the job is going to be run...
- 1365 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, Could you please clarify what do you mean by return the file from the remote repo?Please tag @Debayan​ with your next response which will notify me, Thank you!
- 1 kudos
- 827 Views
- 0 replies
- 0 kudos
Hi, Cat! I’m applying for a position at Databricks and was hoping to get some current Brickster insights. I’ve been wanting to join the company for a ...
Hi, Cat! I’m applying for a position at Databricks and was hoping to get some current Brickster insights. I’ve been wanting to join the company for a while!! Thanks in advance
- 827 Views
- 0 replies
- 0 kudos
- 2535 Views
- 1 replies
- 0 kudos
Using Repos CLI to create a repo but getting Parent Directory /Repos/develop/ does not exist.
I would like to create the directory develop under Repos as part of the script, then link it to github and update it? How can I do this?
- 2535 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Adam:Repos CLI does not have specific functionality to create directories in Databricks Repos. Please check the following doc for more information: https://docs.databricks.com/dev-tools/cli/repos-cli.htmlYou cold use run databricks workspace mkd...
- 0 kudos
- 7977 Views
- 2 replies
- 0 kudos
Could you please guide us on connecting ServiceNow data in databricks
Would like to extract data like ticket info, resolve time, etc., from ServiceNow in databricks.Not finding much information in community and appreciate your guidance on the same.
- 7977 Views
- 2 replies
- 0 kudos
- 0 kudos
ServiceNow offers API capabilities. You can consume the ServiceNow API within a Databricks notebook to extract data from ServiceNow. Following is a suggested prompt to use with ChatGPT for example python code to connect to ServiceNow's api. PROMPT: ...
- 0 kudos
- 3360 Views
- 2 replies
- 3 kudos
Resolved! DLT pipeline that reads data from JDBC source
Could you please guide on how to create the DLT pipeline that directly reads the data from jdbc.When I created the DLT pipeline it give me error at Setting up table, If I ran interactively in notebooks it run successfully, but in non interactive mode...
- 3360 Views
- 2 replies
- 3 kudos
- 3 kudos
What you try do to is not possible.dlt uses autoloader, not jdbcno jars (dlt is sql/python only)I'd skip DLT for this scenario and use an ordinary notebook, nothing wrong with that.
- 3 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
