- 1651 Views
- 1 replies
- 0 kudos
UDF already defined error when using it into a DLT pipeline
I'm using Unity catalog and defined some UDFs in my catalog.database, as reported by show functions in main.default main.default.getgender main.default.tointlist main.default.tostrlistI can use them from a start warehouse pro:SELECT main.default.get_...
- 1651 Views
- 1 replies
- 0 kudos
- 0 kudos
Now the Python UDF support is currently in public preview, and it is supported in the current channel as well. Could you please try to re-run the code. https://docs.databricks.com/en/delta-live-tables/unity-catalog.html
- 0 kudos
- 6370 Views
- 1 replies
- 0 kudos
Interactive Notebook with widgets
Dear experts,We need to create a notebook in order the users to insert/update values in a specific table. We have created one using widgets. However the code performed per action selected is visible to the users. Is there a way to have in a the dropd...
- 6370 Views
- 1 replies
- 0 kudos
- 0 kudos
You can separate the user interface (UI) from the code logic by using two notebooks. 1) The first notebook will contain the code that performs the actions based on the widget inputs. 2) The second notebook will contain the widgets that users interact...
- 0 kudos
- 10387 Views
- 3 replies
- 10 kudos
How can I activate enzyme for delta live tables (or dlt serverless) ?
Hi!I am using Delta Live Tables and especially materialized views and i want to run a dlt pipeline but not rerun the whole view that cost time, but rather only update and add only the values that have been changed. I saw that Enzyme and does this job...
- 10387 Views
- 3 replies
- 10 kudos
- 10 kudos
To achieve "PARTITION OVERWRITE" instead of "COMPLETE RECOMPUTE" when running a Delta Live Table (DLT) materialized view, you need to configure your pipeline to use incremental refresh. This can be done by setting up your DLT pipeline to use serverle...
- 10 kudos
- 5331 Views
- 1 replies
- 0 kudos
Data is not loaded when creating two different streaming table from one delta live table pipeline
i am trying to create 2 streaming tables in one DLT pipleine , both read json data from different locations and both have different schema , the pipeline executes but no data is inserted in both the tables.where as when i try to run each table indiv...
- 5331 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta Live Tables (DLT) can indeed process multiple streaming tables within a single pipeline.Here are a few things to check:1) Verify that each streaming table has a unique checkpoint location. Checkpointing is crucial for maintaining the state of s...
- 0 kudos
- 9068 Views
- 7 replies
- 4 kudos
need help with Azure Databricks questions on CTE and SQL syntax within notebooks
Hi amazing community folks,Feel free to share your experience or knowledge regarding below questions:-1.) Can we pass a CTE sql statement into spark jdbc? i tried to do it i couldn't but i can pass normal sql (Select * from ) and it works. i heard th...
- 9068 Views
- 7 replies
- 4 kudos
- 4 kudos
Not a comparison, but there is a DB-SQL cheatsheet at https://www.databricks.com/sites/default/files/2023-09/databricks-sql-cheatsheet.pdf/
- 4 kudos
- 1037 Views
- 1 replies
- 0 kudos
Python function using Splunk SDK works in Python notebook but not in SQL notebook
Background:I've created a small function in a notebook that uses Splunk's splunk-sdk package. The original intention was to call Splunk to execute a search/query, but for the sake of simplicity while testing this issue, the function only prints pr...
- 1037 Views
- 1 replies
- 0 kudos
- 0 kudos
When you run a Python function in a Python cell, it executes in the local Python environment of the notebook. However, when you call a Python function from a SQL cell, it runs as a UDF within the Spark execution environment. You need to define the f...
- 0 kudos
- 1163 Views
- 1 replies
- 1 kudos
Last file in S3 folder using autoloader
Nowadays we already use the autoloader with checkpoint location, but I still wanted to know if it is possible to read only the last updated file within a folder. I know it somewhat loses the purpose of checkpoint locatioAnother question is it possibl...
- 1163 Views
- 1 replies
- 1 kudos
- 1 kudos
Auto loader's scope is limited to incrementally loading files from storage, and there is no such functionality to just load the latest file from a group of files, you'd likely want to have this kind of "last updated" logic in a different layer or in ...
- 1 kudos
- 1226 Views
- 2 replies
- 1 kudos
APPLY AS DELETE without operation
We are performing a full load of API data into the Bronze table (append only), and then using the APPLY CHANGES INTO query to move data from Bronze to Silver using Stream to get only new records. How can we also utilize the APPLY AS DELETE functional...
- 1226 Views
- 2 replies
- 1 kudos
- 1 kudos
@szymon_dybczak is it possible to do directly in Silver layer? We do not have option to go fo Gold layer. I tried to create a TEMP VIEW in Silver DLT pipeline but it gives error for circular dependency as I am comparing data from Silver it self and a...
- 1 kudos
- 3804 Views
- 4 replies
- 1 kudos
Multiple Autoloader reading the same directory path
HiOriginally, I only have 1 pipeline looking to a directory. Now as a test, I cloned the existing pipeline and edited the settings to a different catalog. Now both pipelines is basically reading the same directory path and running continuous mode.Que...
- 3804 Views
- 4 replies
- 1 kudos
- 1 kudos
To answer the original question, autoloader does not use locks when reading files. You are however limited by the underlying storage system, ADLS in this example. Going by what has been mentioned (long batch times, but spark jobs finish really fast) ...
- 1 kudos
- 2634 Views
- 1 replies
- 0 kudos
Autoloader start and end date for ingestion
I have been searching for a way to set up backfilling using autoloader with an option to set a "start_date" or "end_date". I am working on ingesting a massive file system but I don't want to ingest everything from the beginning. I have a start date t...
- 2634 Views
- 1 replies
- 0 kudos
- 0 kudos
If the files have already been loaded by autoloader (like same name and path), this can be tricky. I recommend starting a separate autoloader stream and specifying filters on it to match your start and end dates. If you'd instead like to rely on the ...
- 0 kudos
- 1204 Views
- 1 replies
- 0 kudos
How to Connect Pyspark to Unity Catalog on Kubernetes with Data Stored in MinIO?
https://stackoverflow.com/questions/79177219/how-to-connect-spark-to-unity-catalog-on-kubernetes-with-data-stored-in-minio? I have posted the question on stack overflow.I am trying to register catalog using pyspark.
- 1204 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your question! Although it shouldn't be necessary, could you please try the following: Set the spark.databricks.sql.initial.catalog.name configuration to my_catalog in your Spark session to ensure the correct catalog is initialized. Use ...
- 0 kudos
- 1314 Views
- 1 replies
- 0 kudos
Generate and Deploy Wheel file to the Databricks Cluster from VS Code
I have a scenario where I need to generate a versioned Wheel file and upload it automatically to a Databricks cluster using VS Code, without creating workflows or jobs like those in Databricks Bundles.My use case is to later use the Wheel file as an ...
- 1314 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your question! To automate generating and deploying a versioned Wheel file to a Databricks cluster directly from VS Code, you can try and follow these steps: Use VS Code Tasks[1]: Configure a tasks.json file in your project’s .vscode dire...
- 0 kudos
- 7175 Views
- 1 replies
- 0 kudos
Use unity catalog access connector for autoloader file notification events
We have a databricks access connector, and we have granted it access to file events. But how do we now use that access connector in cloudfiles/autoloader with file-notifications? If I provide the id in the "cloudFiles.clientId" option, I am asked to...
- 7175 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your question! If the access connector is still prompting for a secret or certificate when used in cloudFiles.clientId, this typically indicates that the authentication method is not being properly recognized. Here's what to check: Access...
- 0 kudos
- 977 Views
- 1 replies
- 1 kudos
Resolved! Integrating Databricks Table with Web Page
Hi Guys,We need to integrate Databricks table with Web Page. Where I want to read Databricks table and show it in grid in WebPage and also want to give ability to update this table from web page. Databricks table is in Unity catalog. Have anyone trie...
- 977 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for your question! To integrate a Databricks table with a web page, you can follow these basic steps: Read the Table: Use a Databricks SQL endpoint or JDBC/ODBC driver to query the table from your web application backend. For example, if y...
- 1 kudos
- 3717 Views
- 3 replies
- 1 kudos
Resolved! Translations from T-SQL: TOP 1 OUTER APPLY or LEFT JOIN
Hi All,I am wondering how you would go about translating either of the below to Spark SQL in Databricks. They are more or less equivalent statements in T-SQL.Please note that I am attempting to pair each unique Policy (IPI_ID) record with its highes...
- 3717 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks filipniziol! I'll start running with that when I run into cases where I need an embedded TOP 1.
- 1 kudos
| User | Count |
|---|---|
| 1636 | |
| 791 | |
| 515 | |
| 349 | |
| 287 |
