cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

hukel
by Contributor
  • 593 Views
  • 1 replies
  • 0 kudos

Parsed Logical Plan report UnresolvedHint RANGE_JOIN

I'm new to RANGE_JOIN so this may be completely normal, but I'd like confirmation.Whenever I put a RANGE_JOIN hint in my query SELECT /*+ RANGE_JOIN(pr2, 3600) */ event.FirstIP4Record FROM SCHEMA_NAME_HERE.dnsrequest event INNER JOIN SC...

  • 593 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@hukel  - The query above does not have a range join, the range filter is not a join condition and it is evaluated as a regular filter.  Please refer to the criteria on range join optimization for joins.  Have a condition that can be interpreted as ...

  • 0 kudos
Amarjit
by New Contributor
  • 705 Views
  • 1 replies
  • 0 kudos

Unable to create a Unity Catalog

https://accounts.azuredatabricks.net/data/createI am unable to access the above link to start with Unity Catalog.

  • 705 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

Are you an account admin? If you access to https://accounts.azuredatabricks.net/ are you able to see the console or just the workspaces that are currently available for you?

  • 0 kudos
Prashanthkumar
by New Contributor III
  • 1120 Views
  • 1 replies
  • 0 kudos

Is it possible to view Databricks cluster metrics using REST API

I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...

Prashanthkumar_0-1705104692634.png
  • 1120 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

There is currently no option available to get this metrics available through API, but is coming soon.

  • 0 kudos
rgualans
by New Contributor
  • 1434 Views
  • 1 replies
  • 1 kudos

Unable to reconstruct state at version ? as the transaction log has been truncated

We have a small table, which undergoes a merge operation on a daily basis. This causes that currently the table has 83 versions.When trying to query this table, we receive the following error:      DeltaFileNotFoundException: dbfs:/mnt/XXXXX/warehous...

rgualans_0-1705078443409.png
  • 1434 Views
  • 1 replies
  • 1 kudos
Latest Reply
Lakshay
Esteemed Contributor
  • 1 kudos

It is difficult to say exactly what caused the issue as there are multiple underlying reasons. But as a general advice, you can try setting the "delta.checkpointRetentionDuration=7 days" and also you can try switching to a different DBR version to se...

  • 1 kudos
itzme_rahul
by New Contributor II
  • 1484 Views
  • 4 replies
  • 0 kudos

Need reattempt to Certified Data Engineer Associate

Hello Team/ @Cert-Team ,I encountered Pathetic experience while attempting my 1st DataBricks certification. My exam got suspended within 10min  after starting my exam. Abruptly, Proctor asked me to show my desk, walls, room, wallet, id card, adapter,...

  • 1484 Views
  • 4 replies
  • 0 kudos
Latest Reply
KavonToy
New Contributor II
  • 0 kudos

Any update?

  • 0 kudos
3 More Replies
kiko_roy
by Contributor
  • 2859 Views
  • 2 replies
  • 2 kudos

Resolved! DLT Unity catalog schema no storage location mention details

Hi TeamAs part of an earlier That I did with Databricks team , I got the info for that if one wants to dump the data in unity catalog schema from a DLT pipeline , The specific schema's Storage location must not be specified . else he DLT pipeline wil...

  • 2859 Views
  • 2 replies
  • 2 kudos
Latest Reply
kiko_roy
Contributor
  • 2 kudos

Thanks @Walter_C  for the explanation and confirming the understanding. Really Appreaciate

  • 2 kudos
1 More Replies
ivanychev
by Contributor II
  • 981 Views
  • 1 replies
  • 0 kudos

Spark doesn't register executors when new workers are allocated

Our pipelines sometimes get stuck (example).Some workers get decommissioned due to spot termination and then the new workers get added. However, after (1) Spark doesn't notice new executors: And I don't know why. I don't understand how to debug this,...

Screenshot 2023-12-11 at 11.12.05.png Screenshot 2023-12-11 at 11.08.56.png Screenshot 2023-12-11 at 11.48.50.png
Community Platform Discussions
decommission
executor
worker
  • 981 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@ivanychev  - Firstly, New workers are added and spark notice them hence, there is an init script logging in the event log stating the init script ran on the newly added workers.  For debugging, please check the Spark UI - executor tab.  Secondly, Fo...

  • 0 kudos
aman_yadav007
by New Contributor
  • 692 Views
  • 1 replies
  • 0 kudos

Databricks Widget

Hi,I was previously working databricks runtime 10.0 and now just upgraded to 13.0 runtime.I was using dashboard to display the widgets. Before it was just showing the widget label, but now it shows the widget name below it as well. Also it shows the ...

  • 692 Views
  • 1 replies
  • 0 kudos
Latest Reply
JunYang
Contributor
  • 0 kudos

Hi @aman_yadav007, which widget type did you use? Can you please try a different widget type or check the widget type and its arguments from this example: https://docs.databricks.com/en/notebooks/widgets.html#databricks-widgets

  • 0 kudos
jimbender
by New Contributor II
  • 2181 Views
  • 1 replies
  • 1 kudos

Resolved! Is there an automated way to strip notebook outputs prior to pushing to github?

We have a team that works in Azure Databricks on notebooks.We are not allowed to push any data to Github per corporate policy.Instead of everyone having to always remember to clear their notebook outputs prior to commit and push, is there a way this ...

  • 2181 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @jimbender, Yes, there are ways to automate the process of clearing notebook outputs in Azure Databricks.    Here are a couple of options:   Manual Clearing: Azure Databricks provides options to clear the notebook state and outputs. You can select...

  • 1 kudos
marketing2
by New Contributor
  • 362 Views
  • 0 replies
  • 0 kudos

L'importance de Databricks dans le SEO

Le SEO est un domaine dynamique et complexe qui évolue constamment avec les technologies et les algorithmes de recherche. L'utilisation de Databricks, une plateforme d'analyse basée sur le cloud, a révolutionné la manière dont les spécialistes du SEO...

pexels-oleksandr-p-9822732.jpg
  • 362 Views
  • 0 replies
  • 0 kudos
Wijnand
by New Contributor II
  • 1166 Views
  • 1 replies
  • 1 kudos

Resolved! Updates on a column in delta table with downstream autoloader

I've got the following questions:1. Can I pause autoloader jobs, delete cluster that was used to run these jobs, create a new cluster and run jobs with newer version cluster?2. I have one autoloader job that ingests JSONs and transforms this to a del...

  • 1166 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Wijnand, Pausing Autoloader Jobs and Updating Clusters: Yes, you can pause autoloader jobs, delete the cluster that was used to run these jobs, create a new cluster, and run jobs with a newer version cluster. You can pause a job in the Jobs UI in...

  • 1 kudos
Sujitha
by Community Manager
  • 9764 Views
  • 2 replies
  • 0 kudos

Creating High Quality RAG Applications with Databricks

Retrieval-Augmented-Generation (RAG) has quickly emerged as a powerful way to incorporate proprietary, real-time data into Large Language Model (LLM) applications. Today we are excited to launch a suite of RAG tools to help Databricks users build hig...

Screenshot 2023-12-06 at 11.41.22 PM.png
  • 9764 Views
  • 2 replies
  • 0 kudos
Latest Reply
antsdispute
New Contributor II
  • 0 kudos

It seems like you're sharing an announcement or promotional content related to Databricks and their launch of a suite of tools for Retrieval-Augmented-Generation (RAG) applications. These tools are aimed at helping Databricks users build high-quality...

  • 0 kudos
1 More Replies
SethParker
by New Contributor II
  • 2864 Views
  • 2 replies
  • 1 kudos

Power BI Import Model Refresh from Databricks SQL Whse - Query has been timed out due to inactivity

We have an intermittant issue where occasionally a partition in our Power BI Import Dataset times out at 5 hours.  When I look at Query History in Databricks SQL, I see a query that failed with the following error message:  "Query has been timed out ...

  • 2864 Views
  • 2 replies
  • 1 kudos
Latest Reply
SethParker
New Contributor II
  • 1 kudos

The only solution we have been able to come up with was to create a Notebook in Databricks that uses the Power BI API to check the status of a Refresh.  We schedule it a bit after we expect the Refresh to complete.  If it is still running, we kill th...

  • 1 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors