- 1149 Views
- 4 replies
- 2 kudos
Resolved! add new column to a table and failing the previous jobs
Hello community! I’m new to Databricks and currently working on a project structured in Bronze / Silver / Gold layers using Delta Lake and Change Data Feed.I recently added 3 new columns to a table and initially applied these changes via PySpark SQ...
- 1149 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @leticialima__ Good dayCan you please share the error observed on the driver log. is it : [Errno 13] Permission denied or No such file or directory? Please let me know the error on the driver log. THank you.
- 2 kudos
- 345 Views
- 1 replies
- 0 kudos
Steamlit in Databricks
HiI have developed a streamlit app locally on my desktop using dummy data, and now I want to be able to use actual data stored in azure blog storage. I have tried to run the same code within a notebook, but keep on getting dependency errors. Is there...
- 345 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @aw1! What exact dependency errors or permission failures are you getting? Can you please share the error message?
- 0 kudos
- 444 Views
- 0 replies
- 0 kudos
Exclusive Masterclass: Oracle Fusion + Databricks Integration using Orbit Analytics
Transform Your Oracle Fusion and Databricks Data into AI-Powered Business Intelligence with Orbit AnalyticsExclusive Masterclass: Oracle Fusion + Databricks Integration for more information click hereWhy This Matters to YouAre you struggling to unloc...
- 444 Views
- 0 replies
- 0 kudos
- 549 Views
- 2 replies
- 0 kudos
databricks dashboard deployment (schema and catalog modification)
I have a databricks dashboard. I have deployed the lvdash.json file through yml (resource.json) from dev to qa env.Now I can see my dashboard published version in resources folder.I want to change the catalog and schema of those underlying queries I ...
- 549 Views
- 2 replies
- 0 kudos
- 0 kudos
You can try using DAB to promote the dashboard and parameterize the query. For more details, check out the DAB dashboard documentation.
- 0 kudos
- 22749 Views
- 34 replies
- 1 kudos
Resolved! My Databrick exam got suspended just for coming closer to laptop screen to read question and options
Hi team,My Databricks Certified Data Engineer Associate exam got suspended within 10 minutes.I had also shown my exam room to the proctor. My exam got suspended due to eye movement. I was not moving my eyes away from laptop screen. It's hard to focus...
- 22749 Views
- 34 replies
- 1 kudos
- 1 kudos
@Cert-TeamOPS I am writing to raise a concern regarding an interruption that occurred during my Databricks Certified Data Engineer Associate exam scheduled for today at 1:15 PM. I began the exam at 1:00 PM, and the experience was smooth until I recei...
- 1 kudos
- 1067 Views
- 7 replies
- 5 kudos
Resolved! How to create classes that can be instantiated from other notebooks?
Hi,I am familiar with object oriented programming and cannot really get my head around the philosophy of coding in Databricks. My approach that naturally consists in creating classes and instantiating objects does not seem to be the right one.Can som...
- 1067 Views
- 7 replies
- 5 kudos
- 2240 Views
- 3 replies
- 1 kudos
Google PubSub for DLT - Error
I'm trying to create a delta live table from a Google PubSub stream.Unfortunately I'm getting the following error:org.apache.spark.sql.streaming.StreamingQueryException: [PS_FETCH_RETRY_EXCEPTION] Task in pubsub fetch stage cannot be retried. Partiti...
- 2240 Views
- 3 replies
- 1 kudos
- 1 kudos
@itamarwe can you please share which permission resulted into the issue and how it got resolved
- 1 kudos
- 1934 Views
- 2 replies
- 2 kudos
Data load issue
I have a job in Databricks which completed successfully but the data is not been written into the target table, I have checked all the possible ways, each n every thing is correct in the code, target table name, source table name, etc etc. It is a Fu...
- 1934 Views
- 2 replies
- 2 kudos
- 2 kudos
This looks like a misconfigured Query Watchdog, specifically the below config: spark.conf.get("spark.databricks.queryWatchdog.outputRatioThreshold") Please check the value of this config - it is 1000 by default. Also, we recommend using Jobs Comput...
- 2 kudos
- 509 Views
- 1 replies
- 1 kudos
Delta UniForm
When we save a delta table using the UniForm option we are seeing a 50% drop in table size. When we add UniForm to a delta table in post we are seeing no change in data size. Is this expected or are others seeing this as well?
- 509 Views
- 1 replies
- 1 kudos
- 1 kudos
Re:When we save a delta table using the UniForm option we are seeing a 50% drop in table size What format are you starting with? e.g. csv -> Delta.
- 1 kudos
- 845 Views
- 1 replies
- 2 kudos
Resolved! AutoLoader Pros/Cons When Extracting Data (Cross-Post)
Cross-posting from: https://community.databricks.com/t5/data-engineering/autoloader-pros-cons-when-extracting-data/td-p/127400Hi there, I am interested in using AutoLoader, but I'd like to get a bit of clarity if it makes sense in my case. Based on e...
- 845 Views
- 1 replies
- 2 kudos
- 2 kudos
You’ve already identified data duplication as a potential con of landing the data first, but there are several benefits to this approach that might not be immediately obvious:Schema Inference and Evolution: AutoLoader can automatically infer the sche...
- 2 kudos
- 912 Views
- 3 replies
- 2 kudos
Resolved! Python module import with Dedicated access mode
I currently have a repo connected in databricks and I was able to correctly import a python module from src folder located in the same root.Since I am using a Machine Learning runtime, I am force to choose a Dedicated (formerly: Single user) access m...
- 912 Views
- 3 replies
- 2 kudos
- 2 kudos
Thanks @szymon_dybczak ! I confirm that's a permission issue and assigning "CAN MANAGE" solves it.I still find it not really intuitive, since the goal is to use a shared cluster (with ML runtime) for development purposes. I mean, it would make sense ...
- 2 kudos
- 713 Views
- 0 replies
- 0 kudos
Automating technical documentation in ETL pipelines using LLMs
Generate pipeline documentation using LLMs and rich metadata extract As enterprise data environments expand, the complexity of maintaining accurate and current documentation across ETL pipelines has intensified. While modern platforms such as Databri...
- 713 Views
- 0 replies
- 0 kudos
- 357 Views
- 1 replies
- 0 kudos
Unity Catalog tool function with custom parameters not being used
I have created a UC tool that takes in a few custom STRING parameters. I gave this tool to an ai agent using the mosaic ai agent framework with hardcoded parameter values for testing. The issue is my ai agent hallucinates and injects its own ai gener...
- 357 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Boban12335,Can we get UC function definition to understand your problem better?Best Regards,Nivethan V
- 0 kudos
- 682 Views
- 3 replies
- 3 kudos
Resolved! AutoLoader - Write To Console (Notebook Cell) Long Running Issue
Hi there,I am likely misunderstanding how to use AutoLoader properly while developing/testing. I am trying to write a simple AutoLoader notebook cell to *read* the contents of a path with json files, and *write* them to console (i.e. notebook cell) i...
- 682 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @ChristianRRL ,It looks like spark.readStream with Auto Loader creates a continuous streaming job by default, which means it keeps running while waiting for new files.To avoid this, you can control the behaviour using trigger(availableNow=True), w...
- 3 kudos
- 3172 Views
- 2 replies
- 3 kudos
Resolved! Documentation for spatial SQL public preview - Where is it?
Hi everybody,since DBR 17.1 spatial sql functions (st_point(), st_distancesphere, ... ) are in public preview.The functionality is presented in this talk Geospatial Insights With Databricks SQL: Techniques and Applications or discussed here in the fo...
- 3172 Views
- 2 replies
- 3 kudos
- 3 kudos
Is this what you were after?https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-st-geospatial-functions
- 3 kudos
