Error accessing Managed Table with Row Level Security using Databricks Cluster

[Soh_m]

Hi Everyone,

We are trying to implement Row Level Security In Delta Table and done testing (i.e. sql execution api,sql editor,sql notebook) using Sql Serverless in Unity Catalog.

But when tried to access the table having RLS in Notebook using Pyspark with all purpose shared cluster, it is throwing error. Tried with Single User as well, as restriction is there, failing as expected. Our multiple workflows running using pyspark notebooks accessing the table.

Does anyone know how to access the table with RLS using All Purpose Cluster. Databricks runtime is 14.3 LTS.

Any help is much appreciated.

[Kaniz]

Hi @Soh_m, When dealing with JSON data in your streaming source, you have a couple of options for extracting fields.

Let’s explore the trade-offs between using the colon sign operator and the schema+from_json function:

1. Colon Sign Operator:

   • The colon sign operator (:) allows you to directly access fields within a JSON string. For example, if your JSON data looks like this: {"name": "Alice", "age": 30}, you can extract the “name” field using json_col.name.
   • Pros:
     • Concise and straightforward syntax.
     • No need to define a schema explicitly.
   • Cons:
     • Performance: Using the colon sign operator can be less efficient, especially for large datasets. It involves parsing the entire JSON string for each record, which can impact performance.
     • Type Inference: The colon sign operator does not perform type inference, so you’ll need to handle type conversions manually.
     • Error Handling: If the JSON structure changes unexpectedly, it may lead to runtime errors.
   • Use Case: Consider using the colon sign operator for quick exploration or when performance is not critical.

2. schema+from_json Function:

   • The schema+from_json approach involves defining a schema explicitly and then using the from_json function to parse the JSON string into a structured format.
   • Pros:
     • Performance: By specifying a schema, you avoid parsing the entire JSON string repeatedly. This can significantly improve performance.
     • Type Safety: The schema enforces type consistency, ensuring that the extracted fields have the expected data types.
     • Error Handling: Schema validation helps catch inconsistencies early.
   • Cons:
     • Requires defining a schema (which can be more verbose).
     • Slightly more complex syntax.
   • Use Case: Use schema+from_json when performance matters and you want better control over data types and error handling.

In summary, if performance is critical and you need type safety, I recommend using the schema+from_json approach. However, for quick exploration or less critical scenarios, the colon sign operator can be convenient. Remember to weigh the trade-offs based on your specific use case! 🚀