cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error accessing Managed Table with Row Level Security using Databricks Cluster

Soh_m
New Contributor

Hi Everyone,

We are trying to implement Row Level Security In Delta Table and done testing (i.e. sql execution api,sql editor,sql notebook) using Sql Serverless in Unity Catalog.

But when tried to access the table having RLS in Notebook using Pyspark with all purpose shared cluster, it is throwing error. Tried with Single User as well, as restriction is there, failing as expected. Our multiple workflows running using pyspark notebooks accessing the table.

Does anyone know how to access the table with RLS using All Purpose Cluster. Databricks runtime is 14.3 LTS.

Any help is much appreciated.

 

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @Soh_mWhen dealing with JSON data in your streaming source, you have a couple of options for extracting fields.

Let’s explore the trade-offs between using the colon sign operator and the schema+from_json function:

  1. Colon Sign Operator:

    • The colon sign operator (:) allows you to directly access fields within a JSON string. For example, if your JSON data looks like this: {"name": "Alice", "age": 30}, you can extract the “name” field using json_col.name.
    • Pros:
      • Concise and straightforward syntax.
      • No need to define a schema explicitly.
    • Cons:
      • Performance: Using the colon sign operator can be less efficient, especially for large datasets. It involves parsing the entire JSON string for each record, which can impact performance.
      • Type Inference: The colon sign operator does not perform type inference, so you’ll need to handle type conversions manually.
      • Error Handling: If the JSON structure changes unexpectedly, it may lead to runtime errors.
    • Use Case: Consider using the colon sign operator for quick exploration or when performance is not critical.
  2. schema+from_json Function:

    • The schema+from_json approach involves defining a schema explicitly and then using the from_json function to parse the JSON string into a structured format.
    • Pros:
      • Performance: By specifying a schema, you avoid parsing the entire JSON string repeatedly. This can significantly improve performance.
      • Type Safety: The schema enforces type consistency, ensuring that the extracted fields have the expected data types.
      • Error Handling: Schema validation helps catch inconsistencies early.
    • Cons:
      • Requires defining a schema (which can be more verbose).
      • Slightly more complex syntax.
    • Use Case: Use schema+from_json when performance matters and you want better control over data types and error handling.

In summary, if performance is critical and you need type safety, I recommend using the schema+from_json approach. However, for quick exploration or less critical scenarios, the colon sign operator can be convenient. Remember to weigh the trade-offs based on your specific use case! 🚀

 
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!