Hi All, As we "cannot enable Iceberg reads on materialized views and streaming tables", Is there any option in private preview to enable Iceberg reads for materialized views and streaming tables. I tried using the option of DLT Sink API with table c...
Hi Databricks Community,I am encountering an issue when trying to display a DataFrame in a Python notebook using serverless compute. The operation seems to fail after several retries, and I get the following error message:[RETRIES_EXCEEDED] The maxim...
Hi @arjunraja_azure , Below is the best version for your code which will avoid failure:from pyspark.sql.functions import maxdf = spark.read.table('workspace.default.emp')df1 = df.agg(max('sal')) # Aggregate in separate step and also avoid caching bef...
I am upgrading a large number of tables for Iceberg / Uniform compatibility by running REORG TABLE <tablename> APPLY (UPGRADE UNIFORM(ICEBERG_COMPAT_VERSION=2));and finding that some tables take several hours to upgrade - presumably because they are ...
HI @JameDavi_51481 , Hope you tried this approach for enabling iceberg metadata along with delta format :ALTER TABLE internal_poc_iceberg.iceberg_poc.clickstream_gold_sink_dltSET TBLPROPERTIES ('delta.columnMapping.mode' = 'name','delta.enableIceberg...
Hi Everyone, Currently we are facing issue with OPTIMIZE table_name FULL operation. The dataset consists of 150 billion rows of data and it takes 8 hours to optimize the reloaded clustered table. The table is refreshed every month and it needs cluste...
Hi @vsam , Have you tried the Auto liquid clustering with Predictive optimization enabled where you don't need to mention cluster by columns specifically and also the optimization will be handled in the backend by predictive optimization concept.http...
Hey Team,Have below queries :I want to test AI/BI dashboard embeding and want to embed in an html page I have created, but dashboard is not getting loaded can you please help in understanding process.And also I want user to have access even if they ...
Yes, you can and should use a service principal to validate and embed your Databricks AI/BI dashboard in an external website like SharePoint. This is a more secure and robust method than using your personal credentials, especially in a production env...
Hello,Citizen science is transforming how research is conducted by inviting everyday people to contribute to scientific discovery. From tracking wildlife migrations to analyzing space data, volunteers are helping scientists gather and interpret massi...
@james698henry, this feels more like a self-promotion using AI generated content rather than anything actionable or useful to the Data Engineering discussions forum.Please elaborate on how the information you've provided lends itself to Data Engineer...
Hey Team,I tried embedding my AI/BI databricks dashboard in sharepoint and it worked.But i dont want to validate using my credential, can i use service principal to validate.
run into this issue when use serverless environment v3JavaPackage object is not callable V2 works fine, any idea
I have a job with one task which is to run a notebook. The job run is setup with a File arrival trigger with my blob storage as the location. The trigger works and the job will start when a new file arrives in the location, but it does not run for ...
@Sneeze7432 you can also try editing the max concurrent runs in the workflow.
We are attempting to use the POST /api/2.2/jobs/run-now endpoint using oAuth 2.0 Client Credentials authentication.We are finding that when sending a request with an expired token, we receive a HTTP code of 400. This contradicts the documentation on ...
Hello gelvinPlease raise the ticket using this lik https://help.databricks.com/s/contact-us?ReqType=training Please explain the issue clearly so that it will be easy for supoort team to help easily.
I have create a pipeline where i am ingesting the data from bronze to silver and using SCD 1, however when i am trying to create gold table as dlt it is giving me error as "Flow 'user_silver' has FAILED fatally. An error occurred because we detected ...
Streaming tables in Delta Live Tables (DLT) only support append-only operations in the SOURCE.The error occurs because:1. Your silver table uses SCD Type 1, which performs UPDATE and DELETE operations on existing records2. Your gold table is defined ...
We have 3 possible compute clusters that we can run a notebook against.They are varying sizes and the one that the notebook uses will depend on the size of the data being processed.We "t-shirt size" each tenant base on their data size (S, M, L) and c...
Hi @allyallen, just to clarify your use case to see if I can provide a solution:Are you saying you have a single job with multiple tasks, and each of those tasks runs the same notebook (e.g., notebook_1), but you'd like the compute cluster to vary de...
Hi allI've followed the book for creating external location for Azure Data Lake Storage (ADLS V2) using account connectorI've granted all required permissions to the connector:I've created a "stock" container on that above mentioned "devtyremeshare" ...
Hey so Ive just started to use DAB to automatically mange job configs via CICD I had a previously-existing job (lets say ID 123) which was created manually and had this config resources: jobs: My_Job_A: name: My Job A And I wanted to automat...
Thanks so much for the response @Advika !That makes sense!Can you explain why the remote config had a different key when compared to the local one? I guess that was what threw me off and made me want to change the local key to match the remote
Hi, is there any way to apply a expectations only if that column exists? I am creating multiple dlt tables with the same python function so i would like to create diferent expectations based in the table name, currently i only can create expectations...
To apply expectations only if a column exists in Delta Live Tables (DLT), you can use the @Dlt.expect decorator conditionally within your Python function. Here is a step-by-step approach to achieve this: Check if the Column Exists: Before applying th...
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
