- 4835 Views
- 5 replies
- 0 kudos
Resolved! How to process all Azure storage file from Databricks
Hi,I want to process all files that are in my azure storage using databricks, What is the process?
- 4835 Views
- 5 replies
- 0 kudos
- 1903 Views
- 3 replies
- 2 kudos
Resolved! Databricks Certified Data Engineer Associate Certificate or Badge not received
I have cleared the Databricks certified Data Engineer Associate exam on 14th July 2023 14:30 hrs. I have received the email stating that i have cleared the exam but haven't received the certificate and badge yet. Attaching the screenshot here. Thanks...
- 1903 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Govind_PS Could you please let us know if you've received the credential? if not please share your webassessor email address.
- 2 kudos
- 9803 Views
- 4 replies
- 8 kudos
Resolved! Creating a service principal with admin role on account level in Azure Databricks using Terraform
Dear Community,In the GUI I can grant the admin role to a service principal with a simple switch.How can I achive the same in Terraform? Do you have some code examples?
- 9803 Views
- 4 replies
- 8 kudos
- 8 kudos
Hi @Adrian Wyss​​, It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us mark the questi...
- 8 kudos
- 1726 Views
- 3 replies
- 4 kudos
Resolved! AI assistant and machine Learning
I am looking to create a basic virtual assistant (AI) that implements machine learning mechanisms.I have some basic knowledge of Python and I have seen some courses on the internet (youtube in particular) that look very interesting.But for the moment...
- 1726 Views
- 3 replies
- 4 kudos
- 4 kudos
Hey everyone!I'm clearly excited about this topic since I'm a huge fan of AI assistants and machine learning. MissyT, creating a basic virtual assistant with machine learning capabilities is an excellent idea! With your simple knowledge of Python and...
- 4 kudos
- 2645 Views
- 1 replies
- 5 kudos
Resolved! Load multiple delta tables at once from Sql server
What’s the best way to efficiently move multiple sql tables in parallel into delta tables
- 2645 Views
- 1 replies
- 5 kudos
- 5 kudos
@Data4 To enable parallel read and write operations, the ThreadPool functionality can be leveraged. This process involves specifying a list of tables that need to be read, creating a method for reading these tables from the JDBC source and saving t...
- 5 kudos
- 5398 Views
- 5 replies
- 6 kudos
Resolved! SFTP Autoloader
Hello, Don't know if it is possible, but I am wondering if it is possible to ingest files from a SFTP server using autoloader ? Or do I have to first copy the files to my dbfs and then use autoloader on that location ? Thank you !
- 5398 Views
- 5 replies
- 6 kudos
- 6 kudos
Hi @erigaud We haven't heard from you since the last response from​, @BriceBuso and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to others. Al...
- 6 kudos
- 4333 Views
- 3 replies
- 3 kudos
Run a multiple %command in the same cell
Hello, is there a way to run multiple %command in a same cell ? I heard that's not possible but would like a confirmation and maybe if it could be an idea for future updates.Moreover, is there a way to mask the output of cells (especially markdown) w...
- 4333 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @BriceBuso Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you. Thanks...
- 3 kudos
- 2784 Views
- 4 replies
- 3 kudos
Data Pull from S3
I have some file in S3, I want to process through Databricks, How it possible? Could you please help me regarding the same.
- 2784 Views
- 4 replies
- 3 kudos
- 3 kudos
access_key = dbutils.secrets.get(scope = "aws", key = "aws-access-key") secret_key = dbutils.secrets.get(scope = "aws", key = "aws-secret-key") encoded_secret_key = secret_key.replace("/", "%2F") aws_bucket_name = "<aws-bucket-name>" mount_name = "<m...
- 3 kudos
- 4782 Views
- 3 replies
- 1 kudos
Deduplication, Bronze (raw) or Silver (enriched)
Need some help in choosing between where to do deduplication of data. So I have sensor data in blob storage that I'm picking up with Databricks Autoloader. The data and files can have duplicates in them.Which of the 2 options do I choose?Option 1:Cre...
- 4782 Views
- 3 replies
- 1 kudos
- 1 kudos
@peter_mcnally You can use watermark to pick the late records and send only the latest records to the bronze table. This will ensure that you always have the latest information in your bronze table.This feature is explained in detail here - https://w...
- 1 kudos
- 5109 Views
- 2 replies
- 1 kudos
Error "Distributed package doesn't have nccl built in" with Transformers Library.
I am trying to run a simple training script using HF's transformers library and am running into the error `Distributed package doesn't have nccl built in` error.Runtime: DBR 13.0 ML - SPark 3.4.0 - Scala 2.12Driver: i3.xlarge - 4 coresNote: This is a...
- 5109 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @anastassia_kor1,For CPU-only training, TrainingArguments has a no_cuda flag that should be set.For transformers==4.26.1 (MLR 13.0) and transformers==4.28.1 (MLR 13.1), there's an additional xpu_backend argument that needs to be set as well. Try u...
- 1 kudos
- 1524 Views
- 4 replies
- 1 kudos
- 1524 Views
- 4 replies
- 1 kudos
- 1744 Views
- 2 replies
- 0 kudos
Databricks CLI's workspace export_dir command adds unnecessary edits despite not making any change in the workspace
databricks workspace export_dir / export command with overwrite option enabled adds non-existent changes in the target directory. 1. It introduces new line deletion and 2. add/deletion of MAGIC comments despite not making any meaningful changes in th...
- 1744 Views
- 2 replies
- 0 kudos
- 0 kudos
I am encountering this issue as well and it did not happen previously. Additionally, you see this pattern if you are using repos internally and make a change to a notebook in another section.
- 0 kudos
- 1624 Views
- 1 replies
- 1 kudos
Create job cluster with a docker image in azure data factory
Is there a way to create a job cluster in azure data factory with a docker image either through API or UI
- 1624 Views
- 1 replies
- 1 kudos
- 1 kudos
Does anyone handle how to run from ADF custom image as jobs compute?
- 1 kudos
- 1319 Views
- 1 replies
- 3 kudos
Resolved! stream from specific time
Hello,How should I start stream a delta table from specific point in time?
- 1319 Views
- 1 replies
- 3 kudos
- 3 kudos
If you are streaming from a Delta table, you can specify the starting version or timestamp. You can refer the document for complete detail: https://docs.databricks.com/structured-streaming/delta-lake.html#specify-initial-position
- 3 kudos
- 1963 Views
- 5 replies
- 1 kudos
When developing a Delta Live Table, can I test my code in the notebook?
Just wondering about the dev flow when building Delta Live Tables. I write my code in the notebook so it would be useful to be able to test it out from within that environment.
- 1963 Views
- 5 replies
- 1 kudos