- 4054 Views
- 1 replies
- 0 kudos
Resolved! Installing a private pypi package from Gitlab on a cluster
I have published a pypi package in a private Gitlab repository and I want to install it in my notebook but I don't know how and the documentation doesn't help me much either. I have created a Gitlab token that I use in the index url and I try to inst...
- 4054 Views
- 1 replies
- 0 kudos
- 0 kudos
This problem was solved by removing the `python>=3.11` requirement.
- 0 kudos
- 6186 Views
- 2 replies
- 0 kudos
Resolved! Remove ZOrdering
Hi, I am trying to demonstrate the effectiveness of ZOrdering but to do this would like to remove the existing ZOrdering first. So my plan is:1. Remove existing ZOrdering2. Run a query and show the explain plan3. Add ZOrdering to column used for Joi...
- 6186 Views
- 2 replies
- 0 kudos
- 0 kudos
@DaveLeach - you can try dropping the table and create it again instead of #1.
- 0 kudos
- 4053 Views
- 3 replies
- 2 kudos
Resolved! Columns tab in Data Explorer doesn't reflect schema changes to table
Hey team, we've noticed that schema changes to a table after creation aren't reflected in the "Columns" tab in the Data Explorer. For example, we added a column called signal_description to a table but its addition isn't reflected in the UI. Is this ...
- 4053 Views
- 3 replies
- 2 kudos
- 2 kudos
@Tharun-Kumar Is this behavior the default behavior in case a schema change happens on a table registered in unity catalog?In that case I would have to run the repair command regularly in order to ensure that the schema displayed is actually the one ...
- 2 kudos
- 3431 Views
- 1 replies
- 0 kudos
User: anonymous is not authorized to perform: sqs:receivemessage on resource
from pyspark.sql import functions as F from pyspark.sql import types as T from pyspark.sql import DataFrame, Column from pyspark.sql.types import Row import dlt S3_PATH = 's3://datalake-lab/xxxx/' S3_SCHEMA = 's3://datalake-lab/xxxx/schemas/' @dl...
- 3431 Views
- 1 replies
- 0 kudos
- 63016 Views
- 7 replies
- 0 kudos
How to check file exists in databricks
I Have a while loop there i have to check a file exists or not if exists read the file in a data frame else go to another file
- 63016 Views
- 7 replies
- 0 kudos
- 0 kudos
How to check if a file exists in DBFS?Let's write a Python function to check if the file exists or not-------------------------------------------------------------def file_exists(path): try: dbutils.fs.ls(path) return True except ...
- 0 kudos
- 14766 Views
- 1 replies
- 2 kudos
Universal Azure Credential Passthrough
At the moment, Azure Databricks has the feature to use AzureAD login for the workspace and create single user clusters with Azure Data Lake Storage credential passthrough. But this can only be used for Data Lake Storage.Is there already a way, or are...
- 14766 Views
- 1 replies
- 2 kudos
- 2 kudos
I have exactly the same issue. I have the need to call a protected API within a notebook but have no access to the current user's access token. I've had to resort to nasty workarounds involving installing and running the Azure CLI from within the not...
- 2 kudos
- 4976 Views
- 1 replies
- 0 kudos
pytesseract.pytesseract.TesseractNotFoundError in databricks notebook
I'm trying to extract the text data from image file in Databricks notebook I have installed below libraries using pip command: %pip install pytesseract tesseract pillow --upgradebut it didn't work and threw below error pytesseract.pytesseract.Tessera...
- 4976 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @neha_ayodhya - can you please try the following via an init script to the Databricks cluster sudo apt-get update -y sudo apt-get install -y tesseract-ocr sudo apt-get install -y libtesseract-dev /databricks/python/bin/pip install pytesseract a...
- 0 kudos
- 1661 Views
- 1 replies
- 0 kudos
Delete S3 files after vacuum
Hi,I'm trying to purge a table of stale data. My databricks host is on cloud.databricks.com.I've set delta.deletedFileRetentionDuration=interval 7 days, deleted many (billions) rows, and followed up with VACUUM tablename RETAIN 168 HOURS, however my ...
- 1661 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Bill - Based on the delta delete file retention period, the files will no longer be there after 7 days . Thanks, Shan
- 0 kudos
- 3154 Views
- 3 replies
- 0 kudos
Resolved! Refresh the Spark UI automatically
Hello!I'm using Databricks with Azure. On a daily basis, I check the status of numerous jobs through the Spark UI. At the moment, the Spark UI does not refresh by itself. I have to refresh the webpage to get the latest status. I wonder if there is a ...
- 3154 Views
- 3 replies
- 0 kudos
- 2004 Views
- 3 replies
- 1 kudos
I have to optimise our initial load, for that I want to perform batch inserts while loading data.
No
- 2004 Views
- 3 replies
- 1 kudos
- 6396 Views
- 4 replies
- 1 kudos
Resolved! Databricks Auto Loader cloudFiles.backfillInterval
Hello, I have been reading databricks Auto Loader documentation about cloudFiles.backfillInterval configuration, and have a question about a specific detail on how it works still. I was only able to find examples of it being set to 1 day or 1 week. ...
- 6396 Views
- 4 replies
- 1 kudos
- 1 kudos
Hey @therealchainman The last backfill (lastBackfillFinishTimeMs) will be recorded as part of the checkpoint -> offset files, this helps the autoloader to know when the last backfill is triggered and to trigger the next periodic backfill.Hope this an...
- 1 kudos
- 1700 Views
- 1 replies
- 0 kudos
alter function owner in UC
I have a number of functions in a schema in a catalog in Unity Catalog, is there a coding way to be able to change the owner of the functions created without having to do it manually via the gui?
- 1700 Views
- 1 replies
- 0 kudos
- 0 kudos
Check this notebook out, I assume you can change it a bit to do what you want. https://docs.databricks.com/en/_extras/notebooks/source/set-owners-notebook.htmlI assume you can loop through the rows in the resulting df (that has the ALTER statements),...
- 0 kudos
- 1863 Views
- 1 replies
- 0 kudos
Hostname could not be verified
Hi all,We have a job that combines historical tables with live tables to give us up to date information. It works for almost all of the tables in our source postgres database, but there's one table that keeps giving the following error. Any ideas why...
- 1863 Views
- 1 replies
- 0 kudos
- 0 kudos
Could you share the full error stack trace please? Also try to test your connectivity by doing "%sh nc -zv {hostname} {port}"
- 0 kudos
- 2791 Views
- 1 replies
- 0 kudos
Error when running Spark-DL notebooks
I try several Spark Deep Learning inference notebooks on Windows. I run Spark in standalone mode with 1 worker with 12 cores (both driver-memory and executor-memory are set to 8G). I always get the same error when applying the deep learning model to ...
- 2791 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a connectivity issue. Check your connectivity by doing "%sh nc -zv {hostname} {port}" from your notebook
- 0 kudos
- 2106 Views
- 1 replies
- 0 kudos
Issue importing library deltalake
Hello,I'm currently seeing a rather cryptic error message whenever I try to import the deltalake library into Databricks (without actually doing anything else).import datalake"ImportError: /local_disk0/.ephemeral_nfs/envs/pythonEnv-cbe496f6-d064-40ae...
- 2106 Views
- 1 replies
- 0 kudos
- 0 kudos
Are you trying to import this library to a Databricks notebook? are you using open source spark in your local machine?
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
