- 6262 Views
- 4 replies
- 0 kudos
Resolved! Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)
I have upgraded the Azure Databricks from 6.4 to 9.1 which enable me to use Spark3. As far as I know, the Hive version has to be upgraded to 2.3.7 as well as discussed in: https://community.databricks.com/s/question/0D53f00001HKHy2CAH/how-to-upgrade-...
- 6262 Views
- 4 replies
- 0 kudos
- 0 kudos
I'm asking about Datatricks version 9.1. I've follow the url given (https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore). Do you mind letting me know where in the table is mentioning the supported hive version fo...
- 0 kudos
- 7214 Views
- 8 replies
- 6 kudos
Resolved! Compile all the scripts under the workspace folder
In workspace one folder I have around 100+ pyspark scripts, all these scripts need to be compiled before running the main program. In order to compile all these files, we are using the %run magic command like %run ../prod/netSales. Since we have 100+...
- 7214 Views
- 8 replies
- 6 kudos
- 6 kudos
Problem is that you can list all files in workspace only via API call and than you can run every one of them using:dbutils.notebook.run()This is the script to list files from workspace (probably you need to add some filterning):import requests ctx = ...
- 6 kudos
- 6861 Views
- 4 replies
- 3 kudos
Resolved! Deploy tar.gz package from private git hub
We created Python package (.tar.gz) and kept it under private git.We can able to connect to that git (using PAT) from the Azure databricks notebook.Our requirement is to install that package from .tar.gz file for that notebook"pip install https://USE...
- 6861 Views
- 4 replies
- 3 kudos
- 3 kudos
For installing the package using pip you need to package the repo using setup.py. check this link for more details https://packaging.python.org/en/latest/tutorials/packaging-projects/alternatively you can pass the tar.gz using --py-files while submi...
- 3 kudos
- 1164 Views
- 0 replies
- 0 kudos
www.scent-sational-waxmelts.co.uk
Ignite your senses with distinctive and fm world shop delightful fragrances for your home, Discover scents to set the mood and inspire fragrant memories
- 1164 Views
- 0 replies
- 0 kudos
- 1043 Views
- 0 replies
- 0 kudos
www.scent-sational-waxmelts.co.uk
Ignite your senses with distinctive and soy wax melts delightful fragrances for your home, Discover scents to set the mood and inspire fragrant memories
- 1043 Views
- 0 replies
- 0 kudos
- 6942 Views
- 5 replies
- 1 kudos
Resolved! Notebook level automated pipeline monitoring or failure notif
Hi, is there any way other than adf monitoring where in automated way we can get notebook level execution details without getting to go to each pipeline and checking
- 6942 Views
- 5 replies
- 1 kudos
- 1 kudos
@Vibhor Sethi​ - Would you be happy to mark @Werner Stinckens​' answer as best if it resolved your question?
- 1 kudos
- 2720 Views
- 1 replies
- 0 kudos
Databricks to Databricks connection
How do you connect to Azure Databricks instance from another Databricks instance? I needed to access (database) Views created in a Databricks instance from a Pyspark notebook running in another Databricks instance. Appreciate if anyone has any samp...
- 2720 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi there, @Venkata Ramakrishna Alvakonda​! My name is Piper, and I'm a moderator for the community. Thank you for your great question! Let's give the community a chance to respond first, and then we'll circle back around. If the community's response ...
- 0 kudos
- 8803 Views
- 1 replies
- 0 kudos
- 8803 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi there,My name is Piper, and I'm one of the moderators for Databricks. Thank you for coming to us with this. Let's give our members a chance to respond first, then we'll come back to see how things went.
- 0 kudos
- 8319 Views
- 1 replies
- 1 kudos
How to configure Databricks Connect to 'Assume Role' when accessing file from an AWS S3 bucket?
I have a Databricks cluster configured with an instance profile to assume role when accessing an AWS S3 bucket.Accessing the bucket from the notebook using the cluster works properly (the instance profile can assume role to access the bucket).However...
- 8319 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello, @lsoewito​ - My name is Piper, and I'm a moderator for the Databricks community. Welcome and thank you for coming to us with your question. I'm sorry to hear that you're having trouble. Let's give your peers a chance to answer your question. W...
- 1 kudos
- 10147 Views
- 2 replies
- 3 kudos
Resolved! How to reset an autoloader?
Hii'm using an autoloader with Azure Databricks:df = (spark.readStream.format("cloudFiles") .options(**cloudfile) .load("abfss://dev@std******.dfs.core.windows.net/**/*****)) at my target checkpointLocation folder there are some files and subdirs...
- 10147 Views
- 2 replies
- 3 kudos
- 3 kudos
@Aman Sehgal​ - My name is Piper, and I'm one of the moderators for Databricks. I wanted to jump in real quick to thank you for being so generous with your knowledge.
- 3 kudos
- 2259 Views
- 0 replies
- 2 kudos
Can not log in to https://community.cloud.databricks.com/login.html
Hi , I created the community cloud account , even I got a mail for resetting password . But once I try to log in to https://community.cloud.databricks.com/login.html , it does not give error , but simple hanging for some time and again login screen ...
- 2259 Views
- 0 replies
- 2 kudos
- 14219 Views
- 3 replies
- 2 kudos
Resolved! Issue in reading parquet file in pyspark databricks.
One of the source systems generates from time to time a parquet file which is only 220kb in size.But reading it fails."java.io.IOException: Could not read or convert schema for file: 1-2022-00-51-56.parquetCaused by: org.apache.spark.sql.AnalysisExce...
- 14219 Views
- 3 replies
- 2 kudos
- 2 kudos
@nafri A​ - Howdy! My name is Piper, and I'm a community moderator for Databricks. Would you be happy to mark @Hubert Dudek​'s answer as best if it solved the problem? That will help other members find the answer more quickly. Thanks
- 2 kudos
- 5739 Views
- 3 replies
- 5 kudos
Ways to validate final Dataframe schema against JSON schema config file
Hi Team,We have to validate transformed dataframe output schema with json schema config file.Here is the scenario Our input json schema and target json schema are different. Using Databricks we are doing the required schema changes. Now, we need to v...
- 5739 Views
- 3 replies
- 5 kudos
- 5 kudos
@Sailaja B​ - Hi! My name is Piper, and I'm a moderator for the community. Thanks for your question. Please let us know how things go. If @welder martins​' response answers your question, would you be happy to come back and mark their answer as best?...
- 5 kudos
- 9853 Views
- 1 replies
- 5 kudos
Resolved! Best practices for implementing Unit Test cases in databricks and Azure devops
Hello,Please suggest the best practices/ ways to implement the unit test cases in Databricks python to pass code coverage at Azure devops
- 9853 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi, the process is like traditional software development practices.Docs to refer: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops#unit-tests-in-azure-databricks-notebooksAzure DevOps Best Practices: https://docs.m...
- 5 kudos
- 9324 Views
- 3 replies
- 0 kudos
Resolved! com.databricks.sql.io.FileReadException Caused by: com.microsoft.azure.storage.StorageException: Blob hash mismatch
Hi, I am getting the following error:com.databricks.sql.io.FileReadException: Error while reading file wasbs:REDACTED_LOCAL_PART@blobStorageName.blob.core.windows.net/cook/processYear=2021/processMonth=12/processDay=30/processHour=18/part-00003-tid-4...
- 9324 Views
- 3 replies
- 0 kudos
- 0 kudos
yes, I can read from notebook with DBR 6.4, when I specify this path: wasbs:REDACTED_LOCAL_PART@blobStorageName.blob.core.windows.net/cook/processYear=2021/processMonth=12/processDay=30/processHour=18but the same using DBR 6.4 from spark-submit, it f...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
