- 944 Views
- 1 replies
- 0 kudos
Resolved! My Databrick exam got Suspended.
Hello Databrick Support Team, request ID: #00667456I faced issue while taking my Databrick Certified Associate Developer for Apache Spark exam. This is completely unfair that alerts are coming while taking mid of exam due to blinking of eyes and proc...
- 944 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Siddartha01! It looks like this post duplicates the one you recently posted. A response has already been provided to the Original thread. I recommend continuing the discussion in that thread to keep the conversation focused and organised.
- 0 kudos
- 3109 Views
- 0 replies
- 0 kudos
Seeking GitBook Alternatives for Documenting Databricks Pipelines & ML Workflows
We’re scaling our use of Databricks notebooks and ML pipelines, and keeping everything properly documented is becoming a challenge. I’ve used GitBook in past projects, but it’s starting to feel a bit clunky, especially when trying to keep markdown do...
- 3109 Views
- 0 replies
- 0 kudos
- 4049 Views
- 1 replies
- 0 kudos
Unable to create a folder inside DBFS on Community Edition
Im using the Community Edition.Trying to create a storage folder inside DBFS -> Filestore for my datasets. I click on Create, give a folder name, and poof. Nothing. No new folder.Tried refreshing, logging out and logging in. Tried to create folder mu...
- 4049 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Sachin, For DBFS V1 used in Community Edition, there may be limitations regarding the creation of folders or changes to file structures, particularly under specific paths such as /FileStore, which is commonly used to store datasets, plots, and oth...
- 0 kudos
- 1514 Views
- 3 replies
- 0 kudos
Resolved! Does Unity Catalog support Iceberg?
Hi, I came across this post on querying an AWS Glue table in S3 from Databricks using Unity Catalog: https://medium.com/@sidnahar13/aws-glue-meets-unity-catalog-effortless-data-governance-querying-in-databricks-5138f1e574a4. The post says support for...
- 1514 Views
- 3 replies
- 0 kudos
- 0 kudos
As of now, Databricks does not support non-Spark methods (such as direct REST API calls, JDBC/ODBC connections outside Spark, or other query engines within Databricks) to query S3 Tables directly. Unity Catalog and Databricks’ native cataloging foc...
- 0 kudos
- 10487 Views
- 12 replies
- 13 kudos
Resolved! Uploading local file
 Since, last two day i getting an error called "ERROR OCCURRED WHEN PROCESSING FILE:[OBJECT OBJECT]" While uploading any "csv" or "json" file from my local system but it shows or running my previous file but give error after uploading a new file
- 10487 Views
- 12 replies
- 13 kudos
- 13 kudos
If you are using databricks community edition, the error you are facing is because the file you are trying to upload contains PII or SPII data ( Personally Identifiable Information OR Sensitive Personally Identifiable Information) words like dob, To...
- 13 kudos
- 4201 Views
- 3 replies
- 5 kudos
Getting java.util.concurrent.TimeoutException: Timed out after 15 seconds on community edition
Im using databricks communtiy edition for learning purpose and im whenever im running notebook, im getting:Exception when creating execution context: java.util.concurrent.TimeoutException: Timed out after 15 seconds databricks.I have deleted cluster ...
- 4201 Views
- 3 replies
- 5 kudos
- 5 kudos
@stevieg95 The issue is that, you've run the notebook with old connector connected to your old deleted cluster with same names, when you ran a terminated cluster, you see the error. First delete existing cluster and logout and detach old cluster as b...
- 5 kudos
- 2036 Views
- 6 replies
- 0 kudos
Issue with Multiple Stateful Operations in Databricks Structured Streaming
Hi everyone,I'm working with Databricks structured streaming and have encountered an issue with stateful operations. Below is my pseudo-code: df = df.withWatermark("timestamp", "1 second") df_header = df.withColumn("message_id", F.col("payload.id"))...
- 2036 Views
- 6 replies
- 0 kudos
- 0 kudos
This should according to this blog post basically work, right? However, I'm getting the same errorMultiple Stateful Streaming Operators | Databricks BlogOr am I missing something? rate_df = spark.readStream.format("rate").option("rowsPerSecond", "1")...
- 0 kudos
- 1538 Views
- 3 replies
- 0 kudos
Databricks Apps have become unavailable
We've become unable to access or renable Apps in our workspace. As of yesterday the tab "Apps" has dissapeared from the compute page in our workspaces. Additionally the toggle that we originally used to enable the public preview for apps has dissapea...
- 1538 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Renu_, thanks for the advice. I just did a double check in the account panel but I couldn't find anything there that seemed related to enabling Apps that we have access to. Would you happen to know which specific preview/setting we would need to ...
- 0 kudos
- 1425 Views
- 2 replies
- 2 kudos
Resolved! Unstructured Data (Images) training in Databricks
I'm looking for a solution where1. Need a delta table that saves the pointers(path) of the images in volume2. Train a Pytorch, tensorflow or keras model with the data from delta lake.I tried multiple methods :1. Storing the data as vectors or binary ...
- 1425 Views
- 2 replies
- 2 kudos
- 2 kudos
I managed to find a few solution accelerators that are in the ballpark, albeit not exact, to what you are trying to accomplish. Have a look: 1. https://www.databricks.com/solutions/accelerators/digital-pathology 2. https://www.databricks.com/resour...
- 2 kudos
- 1737 Views
- 4 replies
- 0 kudos
How to get Databricks usage invoices?
Hey guys,I'm wondering if there are people who wanted to see invoices? I've been using Databricks and I registered my credit card. I've been paying for it.Now I just want to see the invoices but I can't find it. Is there anybody who experienced simil...
- 1737 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello @freshmint! To clarify, are you looking for invoices related to courses you've purchased, or are you referring to other Databricks services?
- 0 kudos
- 852 Views
- 1 replies
- 0 kudos
How do I display output from applyinPandas function?
I'm using databricks version 13.3. I have a function which I'm calling by using the applyInPandas function. I need to see the attributes of my df dataset which I'm using inside my function. My sample code looks likedef train_model(df):# Copy input Da...
- 852 Views
- 1 replies
- 0 kudos
- 0 kudos
Here are some idead/approaches to consider: To inspect the attributes of a df dataset within a function used in applyInPandas on a Databricks Runtime 13.3 cluster, you can use debugging techniques that help you explore the structure and content of ...
- 0 kudos
- 673 Views
- 1 replies
- 0 kudos
Assistance Needed with Databricks DQX Framework for Data Quality
Hi Community Experts,I hope this message finds you well. Our team is currently working on enhancing data quality within our Databricks environment and we are utilizing the Databricks DQX framework for this purpose. We are seeking detailed guidance a...
- 673 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @shubham_007! It looks like this post duplicates the one you recently posted. A response has already been provided there. I recommend continuing the discussion in that thread to keep the conversation focused and organised
- 0 kudos
- 1037 Views
- 3 replies
- 0 kudos
The case of the phantom files!
I developed a notebook that uses *.py files as module imports. On my cluster, the logic in the notebook works fine, my custom modules get loaded, code executes.Using an AzDO pipeline, I deploy the notebook and supporting files to a separate workspac...
- 1037 Views
- 3 replies
- 0 kudos
- 0 kudos
Got it.My background is not SWE, I have always been a 'data guy', but I definitely appreciate a proper dev workflow (ci/cd, git integration, tests).When we started using databricks like 7 or 8 years ago, we went for notebooks as this got us up to spe...
- 0 kudos
- 2991 Views
- 3 replies
- 2 kudos
UI menu customisation
If I want to customize the UI menu so some users/groups can't for example create jobs or make experiments etc.As I see it, when unity catalog is enabled, everyone can create jobs (If they have attach to cluster permission). But in my organization, th...
- 2991 Views
- 3 replies
- 2 kudos
- 2 kudos
I would also like a way to customize what users see when they log in. For example we don't want most of them to even have the option of creating a Genie Space so ideally we could simply remove "Genie" from their menu
- 2 kudos
- 2810 Views
- 11 replies
- 14 kudos
Smaller dataset causing OOM on large cluster
I have a pyspark job reading the input data volume of just ~50-55GB Parquet data from a delta table on Databricks. Job is using n2-highmem-4 GCP VM and 1-15 worker with autoscaling on databricks. Each workerVM of type n2-highmem-4 has 32GB memory and...
- 2810 Views
- 11 replies
- 14 kudos
- 14 kudos
Next, use the repartition(n) to increase your dataframe to twice the number you got earlier. For example, if num_partitions was 30, then repartition(60) prior to running your query. With half the data in each Memory Partition, I'm guessing you won't...
- 14 kudos
