- 2959 Views
- 4 replies
- 1 kudos
Purpose of DLT Table table_properties > quality:medallion
Hi there, silly question here but can anyone help me understand what practical purpose does labelling the table_properties with "quality":"<specific_medallion>"? For example: @Dlt.table( comment="Bronze live streaming table for Test data", name="...
- 2959 Views
- 4 replies
- 1 kudos
- 1 kudos
I'm with the same doubt @ChristianRRL, did you figured out something related to it?My doubt is to check if it's possible to apply any kind of access control based on this property.
- 1 kudos
- 783 Views
- 6 replies
- 3 kudos
Resolved! Plotly Express not rendering in Firefox but fine in Safari
Using a basic example of plotly express i see no output in firefox but is fine in Safari. Any ideas why this may occur? import plotly.express as px import pandas as pd # Create a sample dataframe df = pd.DataFrame({ 'x': range(10), 'y': [2, 3, 5, 7...
- 783 Views
- 6 replies
- 3 kudos
- 3 kudos
UPDATE: I reached out further to Databricks support and they have since deployed a fix. Works fine for me now!
- 3 kudos
- 687 Views
- 1 replies
- 1 kudos
Resolved! Unity Catalog : RDD Issue
In our existing notebooks, the scripts are reliant on RDDs. However, with the upgrade to Unity Catalog, RDDs will no longer be supported. We need to explore alternative approaches or tools to replace the use of RDDs. Could you suggest the best practi...
- 687 Views
- 1 replies
- 1 kudos
- 1 kudos
To transition from using RDDs (Resilient Distributed Datasets) to alternative approaches supported by Unity Catalog, you can follow these best practices and migration strategies: Use DataFrame API: The DataFrame API is the recommended alternative to...
- 1 kudos
- 1045 Views
- 1 replies
- 0 kudos
Best practices for tableau to connect to Databricks
Having problem in connecting to Databrikcs with service principal from tableau . Wanted to how how tableau extracts refreshing connecting to databricks , is it via individual Oauth or service principal
- 1045 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @cheerwthraj, To connect Tableau to Databricks and refresh extracts, you can use either OAuth or service principal authentication. For best practices, please refer to the below link, https://docs.databricks.com/en/partners/bi/tableau.html#best-pr...
- 0 kudos
- 571 Views
- 1 replies
- 1 kudos
New Cluster 90% memory already consumed
Hi, seeing this on all new clusters (single or multi-node) I am creating. As soon as the metrics start showing up, the memory consumption shows 90% already consumed between Used and Cached (something like below). This is the case with higher or lower...
- 571 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @AbhishekNegi I understand your concern. The reason for you to see memory consumption before initiating any task and regarding the comment taking time to execute. This is how Spark internally works. The memory consumption observed in a Spark clust...
- 1 kudos
- 439 Views
- 1 replies
- 0 kudos
SQL table convert to R dataframe
I have a table with ~6 million rows. I am attempting to convert this from a sql table on my catalog to an R dataframe to use the tableone package. I separate my table into 3 tables each containing about 2 million rows then ran it through tbl() and as...
- 439 Views
- 1 replies
- 0 kudos
- 0 kudos
To handle a large SQL table (~6 million rows) and convert it into an R dataframe without splitting it into smaller subsets, you can use more efficient strategies and tools that are optimized for large datasets. Here are some recommendations: 1. Use `...
- 0 kudos
- 261 Views
- 1 replies
- 1 kudos
UC migration : Mount Points in Unity Catalog
Hi All,In my existing notebooks we have used mount points url as /mnt/ and we have notebooks where we have used the above url to fetch the data/file from the container. Now as we are upgrading to unity catalog these url will no longer be supporting a...
- 261 Views
- 1 replies
- 1 kudos
- 1 kudos
Unfortunately no, mount points are no longer supported with UC, so you will need to modify the URL manually on your notebooks.
- 1 kudos
- 330 Views
- 3 replies
- 0 kudos
Issue with Validation After DBFS to Volume Migration in Databricks Workspace
Hello Databricks Community,I have successfully migrated my DBFS (Databricks File System) from a source workspace to a target workspace, moving it from a path in Browse DBFS -> Folders to a Catalog -> Schema -> Volume.Now, I want to validate the migra...
- 330 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Sudheer2, thanks for your comments, you can try using %sh magic to list the folder and sub-directores using unix-like commands for example:
- 0 kudos
- 413 Views
- 3 replies
- 0 kudos
Dashboard sharing in Databriks with Unity Catalog enabled
Hello,I am planning to deploy a workspace with Unity Catalog enabled. Deploying permissions in one place sounds like a good solution. It can even simplify dataset architecture by masking rows and columns.As an architect, I’m concerned about the user’...
- 413 Views
- 3 replies
- 0 kudos
- 0 kudos
I will suggest for you to submit a feature request for this through https://docs.databricks.com/en/resources/ideas.html#ideas
- 0 kudos
- 743 Views
- 3 replies
- 0 kudos
Best practices for optimizing Spark jobs
What are some best practices for optimizing Spark jobs in Databricks, especially when dealing large datasets? Any tips or resources would be greatly appreciated! I’m trying to analyze data on restaurant menu prices so that insights would be especiall...
- 743 Views
- 3 replies
- 0 kudos
- 0 kudos
Optimizing Spark jobs in Databricks can significantly enhance performance. Here are some strategies to consider:Efficient Partitioning: Proper partitioning reduces shuffle times, leading to faster data processing.Caching: Utilize Delta caching inste...
- 0 kudos
- 489 Views
- 0 replies
- 1 kudos
Turn Your Dataframes into an Interactive Tableau-Styled Drag-and-Drop UI for Visual Analysis
You can create Tableau-styled charts without leaving your notebook with just a few lines of code.Imagine this: You’re working within Databricks notebook, trying to explore your Spark/Pandas DataFrame, but visualizing the data or performing Explorator...
- 489 Views
- 0 replies
- 1 kudos
- 549 Views
- 6 replies
- 0 kudos
Migrating Service Principals from Non-Unity to Unity-Enabled Databricks Workspaces - Entitlements Mi
Hello Databricks Community,I am currently in the process of migrating Service Principals from a non-Unity workspace to a Unity-enabled workspace in Databricks. While the Service Principals themselves seem to be migrating correctly, I am facing an is...
- 549 Views
- 6 replies
- 0 kudos
- 0 kudos
Do you have the option to open a support ticket? If not the case I would suggest to run an additional code that disable the entitlement for the new objects, as seems that the entitlement is not being properly passed in the original call
- 0 kudos
- 415 Views
- 0 replies
- 3 kudos
How to make your Service Principal or Group Workspace admin?
Do you require for a Service Principal or a Group to have admin rights to allow automation or reduce the efforts in the process of adding the permission to each user. Solution For Service Principals: You need to be at least Workspace AdminYou can eit...
- 415 Views
- 0 replies
- 3 kudos
- 774 Views
- 2 replies
- 2 kudos
Azure Databricks to GCP Databricks Migration
Hi Team, Can you provide your thoughts on moving Databricks from Azure to GCP? What services are required for the migration, and are there any limitations on GCP compared to Azure? Also, are there any tools that can assist with the migration? Please ...
- 774 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello Team, Adding to @eliana_oviedo comments: Moving Databricks from Azure to GCP involves several steps and considerations. Here are the key points based on the provided context: Services Required for Migration:Cloud Storage Data: Use GCP’s Stora...
- 2 kudos
- 1291 Views
- 1 replies
- 0 kudos
DIAS 2023 -- recommend the training!
Did the SparkUI training yesterday with Mark Ott, and I highly recommend it. It was super helpful and provided a lot of clarity around some of the vaguer terms and metrics, and some surprise penalties.In-memory partition size is the the main thing to...
- 1291 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I can't find this course... can you please share the full name of this course? Thanks in advance
- 0 kudos
