cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

merca
by Valued Contributor II
  • 2948 Views
  • 7 replies
  • 7 kudos

How can I give users permissions to see the objects metadata without access to data

Only permissions I can see are select and this gives access to data and that is very unwanted. I only want users to see the metadata, like table/view/column names and descriptions/comments and location and such but not to see any data.

  • 2948 Views
  • 7 replies
  • 7 kudos
Latest Reply
merca
Valued Contributor II
  • 7 kudos

@Uma Maheswara Rao Desula​ , @Geeta Sai Boddu​  and @S S​ ,Thank you for the responses. I have gotten answer from Databricks and it seems this is not possible and this is something that is investigated as a capability.

  • 7 kudos
6 More Replies
maxutil
by New Contributor II
  • 6896 Views
  • 1 replies
  • 3 kudos

Invalid Characters in Column Names " ,;{}()\n\t="

I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\ .location(target_path)\ .addColumns(df.sche...

  • 6896 Views
  • 1 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Chris Chung​, Can you re-check by trying the below code?df.write.format("delta").option("delta.columnMapping.mode", "name").save("s3://anotherpath")Now you can load it into a Spark dataframe:SELECT * FROM new_table;delta_df = spark.read.format("d...

  • 3 kudos
StephanieRivera
by Valued Contributor II
  • 608 Views
  • 1 replies
  • 0 kudos

Is the delta schema enforcement flexible?

 In the sense that, is it possible to only check for column names or column data types or will it always be both?

  • 608 Views
  • 1 replies
  • 0 kudos
Latest Reply
StephanieRivera
Valued Contributor II
  • 0 kudos

No, I do not believe that is possible. However, I would be interested in understanding a use case where that is ideal behavior. How Does Schema Enforcement Work?Delta Lake uses schema validation on write, which means that all new writes to a table ar...

  • 0 kudos
User15787040559
by New Contributor III
  • 2783 Views
  • 2 replies
  • 0 kudos

How to do a unionAll() when the number and the name of columns are different?

Looking at the API for Dataframe.unionAll() when you have 2 different dataframes with different number of columns and names unionAll() doesn't work.How can you do it?One possible solution is using the following function which performs the union of tw...

  • 2783 Views
  • 2 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

I'm not sure union is the right tool, if the DataFrames have fundamentally different information in them. If the difference is merely column name, yes, rename. If they don't, then the 'union' contemplated here is really a union of columns as well as ...

  • 0 kudos
1 More Replies
Labels