cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Anonymous
by • Not applicable
  • 2403 Views
  • 2 replies
  • 2 kudos

Data source V2 streaming is not supported on table acl or credential passthrough clusters

Using:( hostname is hidden )kafka = spark.readStream\    .format("kafka")\    .option("kafka.sasl.mechanism", "SCRAM-SHA-512")\    .option("kafka.security.protocol", "SASL_SSL")\    .option("kafka.sasl.jaas.config", f'org.apache.kafka.common.security...

Data Engineering
Reply
  • 2403 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

With TACL enabled cluster, you got many restrictions, so streaming will not work. Generally, you can read only things registered in metastore; please disable it for your use case,Additionally, remember that the unity catalog doesn't support streaming...

  • 2 kudos
1 More Replies
irfanijaz
by • New Contributor
  • 752 Views
  • 0 replies
  • 0 kudos

Differently named storage accounts in different environments

Hi,I have a solution design question on which I am looking for some help. We have 2 environments in azure (dev and prod), each env has its own ADLS storage account with a different name of course. Within Databricks code we are NOT leveraging the mou...

Data Engineering
Reply
  • 752 Views
  • 0 replies
  • 0 kudos
kjoth
by • Contributor II
  • 10907 Views
  • 9 replies
  • 7 kudos

Resolved! Delete row from table is not working.

I have created External table using spark via below command. (Using Data science & Engineering)df.write.mode("overwrite").format("parquet").saveAsTable(name=f'{db_name}.{table_name}', path="dbfs:/reports/testing")I have tried to delete a row based on...

Data Engineering
Reply
  • 10907 Views
  • 9 replies
  • 7 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 7 kudos

hi @karthick J​ ,Can you try to delete the row and execute your command in a non high concurrency cluster? the reason why im asking this is because we first need to isolate the error message and undertand why is happening to be able to find the best ...

  • 7 kudos
8 More Replies
brickster_2018
by • Databricks Employee
  • 1119 Views
  • 1 replies
  • 0 kudos

Resolved! Does Table ACL support column-level security like Ranger?

I have used Ranger in Apache Hadoop and it works fine for my use case. Now that I am migrating my workloads from Apache Hadoop to Databricks

Data Engineering
Reply
  • 1119 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

Currently, Table ACL does not support column-level security. There are several tools like Privcera which has better integration with Databricks.

  • 0 kudos
Labels