I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\ .location(target_path)\ .addColumns(df.sche...
Only permissions I can see are select and this gives access to data and that is very unwanted. I only want users to see the metadata, like table/view/column names and descriptions/comments and location and such but not to see any data.
@Uma Maheswara Rao Desula , @Geeta Sai Boddu and @S S ,Thank you for the responses. I have gotten answer from Databricks and it seems this is not possible and this is something that is investigated as a capability.
Hi All,I am trying to do a UNION ALL command with two tables where one table has a few column names that do not match the other table so I am using AS for an alias to match the column names. I am still having trouble getting the two tables to UNION p...
In the sense that, is it possible to only check for column names or column data types or will it always be both?
No, I do not believe that is possible. However, I would be interested in understanding a use case where that is ideal behavior. How Does Schema Enforcement Work?Delta Lake uses schema validation on write, which means that all new writes to a table ar...
Looking at the API for Dataframe.unionAll() when you have 2 different dataframes with different number of columns and names unionAll() doesn't work.How can you do it?One possible solution is using the following function which performs the union of tw...
I'm not sure union is the right tool, if the DataFrames have fundamentally different information in them. If the difference is merely column name, yes, rename. If they don't, then the 'union' contemplated here is really a union of columns as well as ...
