I have completed the "Lakehouse Fundamentals" course and assessment and received the certificate instantly. How long will it take for the Badge to generate or What is the process to get it??
AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark is unable to figure out which one. ...
Hi All,the solution for this problem is very strange.this has caused due to the version of the Databricks runtime.We are using Runtime version 7.0 with Apache Spark 3.0.0 version.In PRD we are using Runtime version 11.3LTS with Apache Spark 3.3.0 ver...
Hi,I know how filtering a delta table on a partition column is a very powerful time-saving approach, but what if this column appears as a CONCAT in the where-clause?I explain my case: I have a delta table with only one partition column, say called co...
I did myself a test and the answer is no:with a Concat filter, spark sql does not know I am using a partition-based column, so it scan all the table.
Hi Everyone,I have an ingestion script where I use the delta merge to update and append newly incoming data in dataframe format to an existing delta table.I am experiencing an issue where all the variables that have been used previously loose their d...
We are working on multiline nested ( multilevel).The file is read and flattened using pyspark and the data frame is showing data using display() method. when saving the same dataframe it is giving executor lost failure error.for some files it is givi...
We are looking into enabling Predictive IO on our delta tables. In the ingest process we are using autoloader and I am wondering if autoloader will gett a flag to enable deletion vectors at table creation? Deletion vectors is a requirement for Predic...
I installed hudi maven library org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0 in Dbricks Runtime Ver : 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) with spark config :spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCat...
@Shanmugavel Chandrakasu %sql create table hudi_cow_pt_tbl ( id bigint, name string, ts bigint, dt string, hh string ) using hudi tblproperties ( type = 'cow', primaryKey = 'id', preCombineField = 'ts' ) partitioned by (dt, hh) location '/mnt/data/h...
Hello Everyone,I am thrilled to announce that we have our 6th winner for the raffle contest -@Bolanle Adesanya . Please join me in congratulating her on this remarkable achievement!Your dedication and hard work have paid off, and we are delighted t...
Hello, I'm looking for a way of defining notification destination using API or Pulumi/Terraform providers. However I cannot find it anywhere. Could you please help and advice if i'm missing something or it's not available at the moment?And If it's no...
This issue seems to point to the lack of a public API being the culprit behind the lack of a resource for Terraform.
Hi All,I am trying to do a UNION ALL command with two tables where one table has a few column names that do not match the other table so I am using AS for an alias to match the column names. I am still having trouble getting the two tables to UNION p...
I've been trying to connect to RedShift following Databrick's documentation and validated that I'm using runtime version 11.3 on my cluster and that I have read/write privileges on the tempdir bucket. But, I'm unable to load data from RedShift to a S...
Hi! I have a SQL query that has a where-clause that checks a string attribute not being equal to e.g. 'シミュレータに接続されていません' (Japanese). This works fine when running the query in SQL Editor ad hoc, but creating a view with the same query, the special cha...
A data engineer, User A, has promoted a new pipeline to production by using the REST API to programmatically create several jobs. A DataOps engineer, User B, has configured an external orchestration tool to trigger job runs through the REST API. Both...
@Ajay Pandey II really appreciate your efforts and you are right in terms of UI, but when we carefully see the question we foundWhich statement describes the contents of the workspace audit logs concerning these events?audit logs are generated and...
Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!
Sign Up Now| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 454 | |
| 349 | |
| 287 |
