cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
maxutil
by • New Contributor II
  • 16317 Views
  • 6 replies
  • 3 kudos

Resolved! Invalid Characters in Column Names " ,;{}()\n\t="

I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\ .location(target_path)\ .addColumns(df.sche...

Data Engineering
Reply
  • 16317 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 3 kudos

Glad it helped @jb1z , happy to help.

  • 3 kudos
5 More Replies
merca
by • Valued Contributor II
  • 10107 Views
  • 7 replies
  • 7 kudos

How can I give users permissions to see the objects metadata without access to data

Only permissions I can see are select and this gives access to data and that is very unwanted. I only want users to see the metadata, like table/view/column names and descriptions/comments and location and such but not to see any data.

Data Engineering
Reply
  • 10107 Views
  • 7 replies
  • 7 kudos
View Replies
Latest Reply
merca
Valued Contributor II
  • 7 kudos

@Uma Maheswara Rao Desula​ , @Geeta Sai Boddu​  and @S S​ ,Thank you for the responses. I have gotten answer from Databricks and it seems this is not possible and this is something that is investigated as a capability.

  • 7 kudos
6 More Replies
StephanieAlba
by • Databricks Employee
  • 1136 Views
  • 1 replies
  • 0 kudos

Is the delta schema enforcement flexible?

 In the sense that, is it possible to only check for column names or column data types or will it always be both?

Data Engineering
Reply
  • 1136 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
StephanieAlba
Databricks Employee
  • 0 kudos

No, I do not believe that is possible. However, I would be interested in understanding a use case where that is ideal behavior. How Does Schema Enforcement Work?Delta Lake uses schema validation on write, which means that all new writes to a table ar...

  • 0 kudos
User15787040559
by • Databricks Employee
  • 4159 Views
  • 2 replies
  • 0 kudos

How to do a unionAll() when the number and the name of columns are different?

Looking at the API for Dataframe.unionAll() when you have 2 different dataframes with different number of columns and names unionAll() doesn't work.How can you do it?One possible solution is using the following function which performs the union of tw...

Data Engineering
Reply
  • 4159 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
sean_owen
Databricks Employee
  • 0 kudos

I'm not sure union is the right tool, if the DataFrames have fundamentally different information in them. If the difference is merely column name, yes, rename. If they don't, then the 'union' contemplated here is really a union of columns as well as ...

  • 0 kudos
1 More Replies
Labels