- 6417 Views
- 8 replies
- 6 kudos
Resolved! Compile all the scripts under the workspace folder
In workspace one folder I have around 100+ pyspark scripts, all these scripts need to be compiled before running the main program. In order to compile all these files, we are using the %run magic command like %run ../prod/netSales. Since we have 100+...
- 6417 Views
- 8 replies
- 6 kudos
- 6 kudos
Problem is that you can list all files in workspace only via API call and than you can run every one of them using:dbutils.notebook.run()This is the script to list files from workspace (probably you need to add some filterning):import requests ctx = ...
- 6 kudos
- 6227 Views
- 4 replies
- 3 kudos
Resolved! Deploy tar.gz package from private git hub
We created Python package (.tar.gz) and kept it under private git.We can able to connect to that git (using PAT) from the Azure databricks notebook.Our requirement is to install that package from .tar.gz file for that notebook"pip install https://USE...
- 6227 Views
- 4 replies
- 3 kudos
- 3 kudos
For installing the package using pip you need to package the repo using setup.py. check this link for more details https://packaging.python.org/en/latest/tutorials/packaging-projects/alternatively you can pass the tar.gz using --py-files while submi...
- 3 kudos
- 1049 Views
- 0 replies
- 0 kudos
www.scent-sational-waxmelts.co.uk
Ignite your senses with distinctive and fm world shop delightful fragrances for your home, Discover scents to set the mood and inspire fragrant memories
- 1049 Views
- 0 replies
- 0 kudos
- 877 Views
- 0 replies
- 0 kudos
www.scent-sational-waxmelts.co.uk
Ignite your senses with distinctive and soy wax melts delightful fragrances for your home, Discover scents to set the mood and inspire fragrant memories
- 877 Views
- 0 replies
- 0 kudos
- 6017 Views
- 5 replies
- 1 kudos
Resolved! Notebook level automated pipeline monitoring or failure notif
Hi, is there any way other than adf monitoring where in automated way we can get notebook level execution details without getting to go to each pipeline and checking
- 6017 Views
- 5 replies
- 1 kudos
- 1 kudos
@Vibhor Sethi​ - Would you be happy to mark @Werner Stinckens​' answer as best if it resolved your question?
- 1 kudos
- 2563 Views
- 1 replies
- 0 kudos
Databricks to Databricks connection
How do you connect to Azure Databricks instance from another Databricks instance? I needed to access (database) Views created in a Databricks instance from a Pyspark notebook running in another Databricks instance. Appreciate if anyone has any samp...
- 2563 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi there, @Venkata Ramakrishna Alvakonda​! My name is Piper, and I'm a moderator for the community. Thank you for your great question! Let's give the community a chance to respond first, and then we'll circle back around. If the community's response ...
- 0 kudos
- 8590 Views
- 1 replies
- 0 kudos
- 8590 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi there,My name is Piper, and I'm one of the moderators for Databricks. Thank you for coming to us with this. Let's give our members a chance to respond first, then we'll come back to see how things went.
- 0 kudos
- 8064 Views
- 1 replies
- 1 kudos
How to configure Databricks Connect to 'Assume Role' when accessing file from an AWS S3 bucket?
I have a Databricks cluster configured with an instance profile to assume role when accessing an AWS S3 bucket.Accessing the bucket from the notebook using the cluster works properly (the instance profile can assume role to access the bucket).However...
- 8064 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello, @lsoewito​ - My name is Piper, and I'm a moderator for the Databricks community. Welcome and thank you for coming to us with your question. I'm sorry to hear that you're having trouble. Let's give your peers a chance to answer your question. W...
- 1 kudos
- 9648 Views
- 2 replies
- 3 kudos
Resolved! How to reset an autoloader?
Hii'm using an autoloader with Azure Databricks:df = (spark.readStream.format("cloudFiles") .options(**cloudfile) .load("abfss://dev@std******.dfs.core.windows.net/**/*****)) at my target checkpointLocation folder there are some files and subdirs...
- 9648 Views
- 2 replies
- 3 kudos
- 3 kudos
@Aman Sehgal​ - My name is Piper, and I'm one of the moderators for Databricks. I wanted to jump in real quick to thank you for being so generous with your knowledge.
- 3 kudos
- 2135 Views
- 0 replies
- 2 kudos
Can not log in to https://community.cloud.databricks.com/login.html
Hi , I created the community cloud account , even I got a mail for resetting password . But once I try to log in to https://community.cloud.databricks.com/login.html , it does not give error , but simple hanging for some time and again login screen ...
- 2135 Views
- 0 replies
- 2 kudos
- 12796 Views
- 3 replies
- 2 kudos
Resolved! Issue in reading parquet file in pyspark databricks.
One of the source systems generates from time to time a parquet file which is only 220kb in size.But reading it fails."java.io.IOException: Could not read or convert schema for file: 1-2022-00-51-56.parquetCaused by: org.apache.spark.sql.AnalysisExce...
- 12796 Views
- 3 replies
- 2 kudos
- 2 kudos
@nafri A​ - Howdy! My name is Piper, and I'm a community moderator for Databricks. Would you be happy to mark @Hubert Dudek​'s answer as best if it solved the problem? That will help other members find the answer more quickly. Thanks
- 2 kudos
- 5444 Views
- 3 replies
- 5 kudos
Ways to validate final Dataframe schema against JSON schema config file
Hi Team,We have to validate transformed dataframe output schema with json schema config file.Here is the scenario Our input json schema and target json schema are different. Using Databricks we are doing the required schema changes. Now, we need to v...
- 5444 Views
- 3 replies
- 5 kudos
- 5 kudos
@Sailaja B​ - Hi! My name is Piper, and I'm a moderator for the community. Thanks for your question. Please let us know how things go. If @welder martins​' response answers your question, would you be happy to come back and mark their answer as best?...
- 5 kudos
- 9458 Views
- 1 replies
- 5 kudos
Resolved! Best practices for implementing Unit Test cases in databricks and Azure devops
Hello,Please suggest the best practices/ ways to implement the unit test cases in Databricks python to pass code coverage at Azure devops
- 9458 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi, the process is like traditional software development practices.Docs to refer: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops#unit-tests-in-azure-databricks-notebooksAzure DevOps Best Practices: https://docs.m...
- 5 kudos
- 8780 Views
- 3 replies
- 0 kudos
Resolved! com.databricks.sql.io.FileReadException Caused by: com.microsoft.azure.storage.StorageException: Blob hash mismatch
Hi, I am getting the following error:com.databricks.sql.io.FileReadException: Error while reading file wasbs:REDACTED_LOCAL_PART@blobStorageName.blob.core.windows.net/cook/processYear=2021/processMonth=12/processDay=30/processHour=18/part-00003-tid-4...
- 8780 Views
- 3 replies
- 0 kudos
- 0 kudos
yes, I can read from notebook with DBR 6.4, when I specify this path: wasbs:REDACTED_LOCAL_PART@blobStorageName.blob.core.windows.net/cook/processYear=2021/processMonth=12/processDay=30/processHour=18but the same using DBR 6.4 from spark-submit, it f...
- 0 kudos
- 9775 Views
- 4 replies
- 0 kudos
Resolved! Databricks-Connect and Change Data Feed query error
I have installed Databricks-Connect (9.1 LTS). I am able to send queries to the cluster. However, when the query includes a call to the 'table_changes' function that is a part of Change Data Feed, I get the following error:AnalysisException("could ...
- 9775 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Kaniz Fatma​ , the table_changes function is an internal Databricks function used in Change Data Feed (CDF).Please refer to the article below. It discusses the table_changes function.https://docs.databricks.com/delta/delta-change-data-feed.html
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
