- 35608 Views
- 7 replies
- 2 kudos
How can I embed image to the cell using markdown or code?
I have been trying to embed the image from the dbfs location, when I run the code, the image is unknown or question mark. I have tried following code: The path of the file is dbfs:/FileStore/tables/svm.jpgdisplayHTML("<img src ='dbfs:/FileStore/tabl...
- 35608 Views
- 7 replies
- 2 kudos
- 2 kudos
@WiliamRosa You've stated:"1. Drag and drop images directly into Markdown cellsYou can simply drag an image file from your local system into a markdown cell. Databricks will upload it automatically to your workspace directory and display it inline in...
- 2 kudos
- 4687 Views
- 7 replies
- 2 kudos
Maximum string length to pass for Databricks notebook widget
Is there a limitation on the string length to pass for Databricks notebook widget? ADF lookup outputs about 1000 tables that I am trying to pass to the databricks notebook via widget parameter. ADF spends 30 mins to open the Databricks notebook and e...
- 4687 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi @CzarR ,Yes, there's a limitation. A maximum of 2048 characters can be input to a text widget.https://docs.databricks.com/aws/en/notebooks/notebook-limitations#databricks-widgets
- 2 kudos
- 3066 Views
- 4 replies
- 5 kudos
Resolved! AutoLoader - Cost of Directory Listing Mode
I'm curious to get thoughts and experience on this. Intuitively, the directory listing mode makes sense to me in order to ensure that only the latest unprocessed files are picked up and processed, but I'm curious about what the cost impact of this wo...
- 3066 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @ChristianRRL Autoloader ingests your data incrementally regardless of whether you are on directory listing mode or file notification mode. The key difference lies in how it discovers new files. In directory listing mode, Autoloader queries the cl...
- 5 kudos
- 2196 Views
- 1 replies
- 1 kudos
Resolved! read_files vs. cloud_file
I was wondering what is the difference between read_files and cloud_file.I can't find explicit explanation or comparison in Databriks Documentation.Best Regards Ivaylo
- 2196 Views
- 1 replies
- 1 kudos
- 1 kudos
## Key Differences Between `read_files` and `cloud_files`### **`read_files` Function**`read_files` is a table-valued function that reads files under a provided location and returns the data in tabular form. It supports reading JSON, CSV, XML, TEXT, B...
- 1 kudos
- 4102 Views
- 1 replies
- 4 kudos
Resolved! Recommended approach for handling deletes in a Delta table
What is the recommended approach for handling deletes in a Delta table?I have a table in MySQL (no soft delete flag) that I read and write into Azure as a Delta table. My current flow is:- If an ID exists in both MySQL and the Delta table → update th...
- 4102 Views
- 1 replies
- 4 kudos
- 4 kudos
The recommended way of handling CDC in Databricks is by using the merge command.https://docs.databricks.com/aws/en/sql/language-manual/delta-merge-intoIf you using SQL.-- Delete all target rows that have a match in the source table.> MERGE INTO targe...
- 4 kudos
- 19745 Views
- 5 replies
- 0 kudos
Is there a way to programatically retrieve a workspace name ?
Is there a spark command in databricks that will tell me what databricks workspace I am using? I’d like to parameterise my code so that I can update delta lake file paths automatically depending on the workspace (i.e. it picks up the dev workspace na...
- 19745 Views
- 5 replies
- 0 kudos
- 0 kudos
To programmatically retrieve the Databricks workspace name from within a notebook, you can use Spark configuration or the notebook context. One method is to read the workspace URL using spark.conf.get("spark.databricks.workspaceUrl") and then extract...
- 0 kudos
- 6421 Views
- 6 replies
- 1 kudos
Monitoring structure streaming in externar sink
Hi! Today working trying to collect some metrics to create a splot in my spark structure streaming. It is configured with a trigger(processingTime="30 seconds") and I am trying to collect data with the following Listener Class (just an example). # D...
- 6421 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi everyone,I recently worked on a similar requirement and would like to share a structured approach to monitoring Structured Streaming when writing to external sinks.1. Use a Unique Query NameAlways assign a clear and meaningful name to each streami...
- 1 kudos
- 1798 Views
- 3 replies
- 1 kudos
Help required for executing geospetial query
we have requirement to find a specific distance based on longitude and latitude. Can you please help me with the details step how we can achieve this using pyspark? Thank you.
- 1798 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @dholea ,Databricks Runtime 17.1 Beta added native support for Spatial SQL. So for example it let's you calculate distance between coordinates.I think you can try with ST_Distance() function.
- 1 kudos
- 1159 Views
- 2 replies
- 0 kudos
Cannot use join with Enzyme
I suppose I can use incrementalization on pipelines. Supported operators are listed in here: https://docs.databricks.com/aws/en/optimizations/incremental-refresh#support-for-materialized-view-incremental-refreshHowever, when I run the pipeline, it do...
- 1159 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for your reply. Even though we only do one join, we keep getting this error.
- 0 kudos
- 1546 Views
- 2 replies
- 1 kudos
Resolved! Managing values that change between development and production
Hi all, when moving from development to testing a production one often needs to handle change values like the blob store or database server being differentI have seen that using widgets can be a useful way to have updateable values for Notebooks and ...
- 1546 Views
- 2 replies
- 1 kudos
- 1 kudos
Great, thanks. Speed in this case isn't critical as it's not processing massive amounts of data, well I hope not massive amounts at this time. It'll be some batch processes that can't use dlt.
- 1 kudos
- 4851 Views
- 2 replies
- 1 kudos
Odoo ERP customization Odoo is one of the most popular ERP software. It is widely use by companies. Odoo customization mainly includes changing the sy...
Odoo ERP customizationOdoo is one of the most popular ERP software. It is widely use by companies. Odoo customization mainly includes changing the system by including new features and functionalities in accordance with the business needs of the clien...
- 4851 Views
- 2 replies
- 1 kudos
- 1 kudos
Odoo ERP customization is process to customize module, CRM, website, POS, report and more to meet the specific business requirement.
- 1 kudos
- 7003 Views
- 4 replies
- 2 kudos
Resolved! Data Transfer using Unity Catalog full implementation
I have to share data between Azure A and Azure B . using unity catalog and delta sharing.Every Time Data comes to Azure A, The same Data can be read by AzureB.How to handle Incremental Load. for change records I think I need to use Merge Statement....
- 7003 Views
- 4 replies
- 2 kudos
- 2 kudos
This works well when set up, If you're securely set up in Azure you will need to grant a privatelink to the underlying storage for their service to read data. For enhanced security I'd recommend your catalog for the other party then be in external st...
- 2 kudos
- 942 Views
- 1 replies
- 0 kudos
How can we Implement Conditional Logic on SQL Query Output in Job Workflow
I'm trying to create a job where I define a get data task that executes a SQL query. After that, I want to apply conditional logic using an if-else task based on the query output. Specifically, I want to check each row individually—if a condition is ...
- 942 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, I believe the the fixed parameter option is exists and introduced recently in lake flow declarative pipeline where you need to navigate to the configuration section and add parameters.
- 0 kudos
- 3829 Views
- 2 replies
- 0 kudos
Merge using DLT
I have a requirement where i need only 24 hours data from my delta tablelets call this as latest tablethis latest table should be in sync with sourceso, it should handle all updates and inserts along with delete (if something gets deleted at source, ...
- 3829 Views
- 2 replies
- 0 kudos
- 0 kudos
from pyspark.sql.functions import col, lit, expr, when, to_timestamp, current_timestampfrom pyspark.sql.functions import max as max_import dltfrom pyspark.sql.types import StructType, StructField, StringTypefrom pyspark.sql.utils import AnalysisExcep...
- 0 kudos
- 17140 Views
- 7 replies
- 4 kudos
Resolved! Databricks Autoloader Checkpoint
Hello Databricks Community,I'm encountering an issue with the Databricks Autoloader where, after running successfully for a period of time, it suddenly stops detecting new files in the source directory. This issue only gets resolved when I reset the ...
- 17140 Views
- 7 replies
- 4 kudos
- 4 kudos
I have found that reducing the number of objects in the landing path (via an archive/cleanup process) is the most reliable fix. Auto Loader's file discovery can bog down in big/"long-lived" landing folders—especially in directory-listing mode—so clea...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
