- 1758 Views
- 1 replies
- 0 kudos
Data view in Side Panel
Does anyone know why I cannot see the Data view in the side panel under workspace. I see catalog instead of data. Is this something that has been upgraded?
- 1758 Views
- 1 replies
- 0 kudos
- 0 kudos
@DB_Keith - Data Explorer is renamed to catalog Explorer. Please refer to the release notes. https://learn.microsoft.com/en-us/azure/databricks/release-notes/product/2023/september#data-explorer-is-now-catalog-explorer
- 0 kudos
- 1772 Views
- 1 replies
- 0 kudos
Parsed Logical Plan report UnresolvedHint RANGE_JOIN
I'm new to RANGE_JOIN so this may be completely normal, but I'd like confirmation.Whenever I put a RANGE_JOIN hint in my query SELECT /*+ RANGE_JOIN(pr2, 3600) */ event.FirstIP4Record FROM SCHEMA_NAME_HERE.dnsrequest event INNER JOIN SC...
- 1772 Views
- 1 replies
- 0 kudos
- 0 kudos
@hukel - The query above does not have a range join, the range filter is not a join condition and it is evaluated as a regular filter. Please refer to the criteria on range join optimization for joins. Have a condition that can be interpreted as ...
- 0 kudos
- 2968 Views
- 1 replies
- 1 kudos
DataFrame vs. Spark SQL
When should I use DataFrame over Spark SQL?If DataFrame is better, then why do I need Spark SQL?
- 2968 Views
- 1 replies
- 1 kudos
- 1 kudos
@SamAWS - when you want to query the underlying data in the dataframe by creating a view or table, Spark SQL is preferred. Same can be achieved using pyspark by retaining them as a dataframe. Spark SQL is used by wider audience and easy to develop.
- 1 kudos
- 1578 Views
- 1 replies
- 0 kudos
Unable to create a Unity Catalog
https://accounts.azuredatabricks.net/data/createI am unable to access the above link to start with Unity Catalog.
- 1578 Views
- 1 replies
- 0 kudos
- 0 kudos
Are you an account admin? If you access to https://accounts.azuredatabricks.net/ are you able to see the console or just the workspaces that are currently available for you?
- 0 kudos
- 2123 Views
- 1 replies
- 0 kudos
Is it possible to view Databricks cluster metrics using REST API
I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...
- 2123 Views
- 1 replies
- 0 kudos
- 0 kudos
There is currently no option available to get this metrics available through API, but is coming soon.
- 0 kudos
- 3921 Views
- 1 replies
- 1 kudos
Unable to reconstruct state at version ? as the transaction log has been truncated
We have a small table, which undergoes a merge operation on a daily basis. This causes that currently the table has 83 versions.When trying to query this table, we receive the following error: DeltaFileNotFoundException: dbfs:/mnt/XXXXX/warehous...
- 3921 Views
- 1 replies
- 1 kudos
- 1 kudos
It is difficult to say exactly what caused the issue as there are multiple underlying reasons. But as a general advice, you can try setting the "delta.checkpointRetentionDuration=7 days" and also you can try switching to a different DBR version to se...
- 1 kudos
- 3035 Views
- 1 replies
- 0 kudos
- 3035 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Databricks_Work , Vacuum and Analzye are two separate commands that are used for optimizing the queries but they perform two different operations. Vacuum is used to clear the stale data files in your delta table. Vacuum should be run after a opti...
- 0 kudos
- 3195 Views
- 4 replies
- 0 kudos
Need reattempt to Certified Data Engineer Associate
Hello Team/ @Cert-Team ,I encountered Pathetic experience while attempting my 1st DataBricks certification. My exam got suspended within 10min after starting my exam. Abruptly, Proctor asked me to show my desk, walls, room, wallet, id card, adapter,...
- 3195 Views
- 4 replies
- 0 kudos
- 5096 Views
- 2 replies
- 2 kudos
Resolved! DLT Unity catalog schema no storage location mention details
Hi TeamAs part of an earlier That I did with Databricks team , I got the info for that if one wants to dump the data in unity catalog schema from a DLT pipeline , The specific schema's Storage location must not be specified . else he DLT pipeline wil...
- 5096 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks @Walter_C for the explanation and confirming the understanding. Really Appreaciate
- 2 kudos
- 9272 Views
- 0 replies
- 1 kudos
5 tips to get the most out of your Databricks Assistant
Back in July, we released the public preview of the new Databricks Assistant, a context-aware AI assistant available in Databricks Notebooks, SQL editor and the file editor that makes you more productive within Databricks, including: Generate SQL or ...
- 9272 Views
- 0 replies
- 1 kudos
- 2679 Views
- 0 replies
- 0 kudos
Bringing breakthrough data intelligence to industries
Gen AI for All: Empowering Every Role Across Industries The new frontier of data intelligence is here. As more companies pursue industry-changing transformations, they face the same monumental challenge: how to democratize data and AI. In this new r...
- 2679 Views
- 0 replies
- 0 kudos
- 3031 Views
- 1 replies
- 0 kudos
Spark doesn't register executors when new workers are allocated
Our pipelines sometimes get stuck (example).Some workers get decommissioned due to spot termination and then the new workers get added. However, after (1) Spark doesn't notice new executors: And I don't know why. I don't understand how to debug this,...
- 3031 Views
- 1 replies
- 0 kudos
- 0 kudos
@ivanychev - Firstly, New workers are added and spark notice them hence, there is an init script logging in the event log stating the init script ran on the newly added workers. For debugging, please check the Spark UI - executor tab. Secondly, Fo...
- 0 kudos
- 2427 Views
- 1 replies
- 1 kudos
Ifichangethetimestampformat from yyyy-MM-dd hh-mm-ss to MM-dd-yyyy hh-mm-ssinPostgrestableisthatfine
- 2427 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, You can check the function https://docs.databricks.com/en/sql/language-manual/functions/date_format.html, let us know if this helps.
- 1 kudos
- 6527 Views
- 1 replies
- 0 kudos
How to query sql warehouse tables with spark?
Hey there... I managed to query my data following this guide https://learn.microsoft.com/en-us/azure/databricks/dev-tools/python-sql-connectorusing databricks sql#!/usr/bin/env python3from databricks import sqlwith sql.connect(server_hostname = "adb-...
- 6527 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mobe - Please refer to the github link for more examples - https://github.com/databricks/databricks-sql-python/blob/main/examples. Thanks,Shan
- 0 kudos
- 4191 Views
- 2 replies
- 1 kudos
Getting 'No GCP Marketplace token provided' error while signing up from GCP marketplace.
Hey guys,I was trying to sign up to the 14 day free trial from GCP marketplace. When I click 'SIGN UP WITH DATABRICKS', I get the error below.HTTP ERROR 401Problem accessing /sign-up. Reason: No GCP Marketplace token provided. Please start over fr...
- 4191 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks Walter,I have the IAM permissions in place and also have a valid billing account.However, I keep getting the same error regarding the missing Marketplace token. I am clicking the 'SIGN UP WITH DATABRICKS' button from the GCP UI, so am not sure...
- 1 kudos
