- 3213 Views
- 2 replies
- 0 kudos
I understand that DLT is a separate job compute but I would like to use an existing all purpose cluster for the DLT pipeline. Is there a way I can achieve this?
- 3213 Views
- 2 replies
- 0 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
by
feed
• New Contributor III
- 3925 Views
- 7 replies
- 3 kudos
TesseractNotFoundError: tesseract is not installed or it's not in your PATH. See README file for more information. in databricks
- 3925 Views
- 7 replies
- 3 kudos
Latest Reply
%sh apt-get install -y tesseract-ocr this command is not working in my new Databricks free trail account, earlier it worked fine in my old Databricks instance. I get below error: E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Per...
6 More Replies
- 1150 Views
- 2 replies
- 1 kudos
Hello All,Following command on running through databricks notebook is not working Command%sh# Bash code to print 'Hello, PowerShell!'echo 'Hello, PowerShell!'# powershell.exe -ExecutionPolicy Restricted -File /dbfs:/FileStore/Read_Vault_Inventory.ps1...
- 1150 Views
- 2 replies
- 1 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
by
hv129
• New Contributor
- 941 Views
- 2 replies
- 1 kudos
I have around 25GBs of data in my Azure storage. I am performing data ingestion using Autoloader in databricks. Below are the steps I am performing:Setting the enableChangeDataFeed as true.Reading the complete raw data using readStream.Writing as del...
- 941 Views
- 2 replies
- 1 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
- 464 Views
- 3 replies
- 0 kudos
I'd like to create a storage credential for an Azure Storage Account in an AWS workspace. I then plan to use this storage credential to create an external volume.Is this possible, and if so what are the steps? Thanks for any help!
- 464 Views
- 3 replies
- 0 kudos
Latest Reply
Thanks for your help.I'm struggling to create the Storage Credential. I have created a managed identity via an Azure Databricks Access Connector and am making an API call based on what I'm reading in the API docs: Create a storage credential | Storag...
2 More Replies
- 463 Views
- 2 replies
- 0 kudos
We are trying to retrieve xml file name using _metadata but not working. we are not able to use input _file_name() also as we are using shared cluster.we are reading the xml files using com.datadricks.spark.xml library
- 463 Views
- 2 replies
- 0 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
by
ElaPG
• New Contributor III
- 1601 Views
- 7 replies
- 1 kudos
Is there any possibility to restrict usage of specified commands (like mount/unmount or SQL grant) based on group assignment? I do not want everybody to be able to execute these commands.
- 1601 Views
- 7 replies
- 1 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
6 More Replies
by
Cas
• New Contributor III
- 876 Views
- 3 replies
- 1 kudos
Hi!As we are migrating from dbx to asset bundles we are running into some problems with the dynamic insertion of job clusters in the job definition as with dbx we did this nicely with jinja and defined all the clusters in one place and a change in th...
- 876 Views
- 3 replies
- 1 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
2 More Replies
- 780 Views
- 5 replies
- 1 kudos
I want to stream data from kinesis using DLT. the Data is in json format. How can I use structured streaming to automatically infer the schema? I know auto-loader has this feature but it doesn't make sense for me to use autoloader since my data is st...
- 780 Views
- 5 replies
- 1 kudos
Latest Reply
I wanted to use Databricks for this. I don't want to depend on AWS Glue. Same way I could do it with AutoLoader...
4 More Replies
- 545 Views
- 2 replies
- 0 kudos
Hello team,I'm trying to upgrade our databricks runtime to 13.3 from 9.1, but i've been having issues installing libraries on the compute from our internal artifactoryHowever, when I tried this on a unity-catalog enabled workspace, it works seamless...
- 545 Views
- 2 replies
- 0 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
- 806 Views
- 2 replies
- 1 kudos
I am using SHOW PARTITIONS <<table_name>> to get all the partitions of a table. I want to use max() on the output of this command to get the latest partition for the table.However, I am not able to use SHOW PARTITIONS <<table_name>> in a CTE/sub-quer...
- 806 Views
- 2 replies
- 1 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
by
Simha
• New Contributor II
- 1131 Views
- 2 replies
- 1 kudos
Hi All,I am trying to write a csv file on to the blob and ADLS from databricks notebook using pyspark and a separate folder is created with the mentioned filename and a partition is created within the folder.I want only file to be written. Can anyone...
- 1131 Views
- 2 replies
- 1 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
by
Mr__D
• New Contributor II
- 2863 Views
- 7 replies
- 1 kudos
Hi All, Could you please suggest to me the best way to write PySpark code in Databricks,I don't want to write my code in Databricks notebook but create python files(modular project) in Vscode and call only the primary function in the notebook(the res...
- 2863 Views
- 7 replies
- 1 kudos
Latest Reply
Certainly! To write PySpark code in Databricks while maintaining a modular project in VSCode, you can organize your PySpark code into Python files in VSCode, with a primary function encapsulating the main logic. Then, upload these files to Databricks...
6 More Replies
- 13504 Views
- 6 replies
- 2 kudos
When I use the following code: df
.coalesce(1)
.write.format("com.databricks.spark.csv")
.option("header", "true")
.save("/path/mydata.csv")it writes several files, and when used with .mode("overwrite"), it will overwrite everything in th...
- 13504 Views
- 6 replies
- 2 kudos
Latest Reply
Hi Daniel,May I know, how did you fix this issue. I am facing similar issue while writing csv/parquet to blob/adls, it creates a separate folder with the filename and creates a partition file within that folder.I need to write just a file on to the b...
5 More Replies
- 520 Views
- 3 replies
- 0 kudos
[Situation]I am using AWS DMS to store mysql cdc in S3 as a parquet file.I have implemented a streaming pipeline using the DLT module.The target destination is Unity Catalog.[Questions and issues].- Where are the tables and materialized views specifi...
- 520 Views
- 3 replies
- 0 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
2 More Replies
