cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
ClaudeR
by • New Contributor III
  • 6411 Views
  • 5 replies
  • 1 kudos

Resolved! Can someone help me understand how compute pricing works.

Im looking at using Databricks internally for some Data Science projects. I am however very confused to how the pricing works and would like to obviously avoid high spending right now. Internal documentation and within Databricks All-Purpose Compute...

Data Engineering
Reply
  • 6411 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
GuillermoM
New Contributor II
  • 1 kudos

Hello,I was able to get a very precise cost of Azure Databricks Clusters and Computers jobs, using the Microsoft API and Databricks APIThen I wrote a simple tool to extract and manipulate the API results and generate detailed cost reports that can be...

  • 1 kudos
4 More Replies
Pragati_17
by • New Contributor II
  • 3884 Views
  • 2 replies
  • 1 kudos

Setting Date Range Picker to some default date

I have a date range picker filter in Databricks Lakeview Dashboard, so when i open dashboard there is no date selected and i want to set a default date. Is that possible with lakeview dashboard filters?

Data Engineering
databricks lakeview dashboard
date range pciker filter
default date set
Reply
  • 3884 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
miranda_luna_db
Databricks Employee
  • 1 kudos

  Hi there - We're working on default filter values actively and that will help here. For now, when you change filter values you'll notice the URL changes. You can always bookmark the URL or share that modified one with other and when they open it, t...

  • 1 kudos
1 More Replies
Gutek
by • New Contributor II
  • 5111 Views
  • 4 replies
  • 1 kudos

Resolved! How to import a Lakeview Dashboard programmatically (API or CLI)?

I'm trying to import a Lakeview Dashboard that I've originally exported through the CLI (version 0.213.0). The exported file has extension .lvdash.json and is a single line json file.I can't get it to work, I tried this command: databricks workspace ...

Data Engineering
Reply
  • 5111 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
miranda_luna_db
Databricks Employee
  • 1 kudos

Glad you've got everything up and running!

  • 1 kudos
3 More Replies
Akshith_Rajesh
by • New Contributor III
  • 12096 Views
  • 5 replies
  • 6 kudos

Resolved! Call a Stored Procedure in Azure Synapse with input and output Params

driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager connection = driver_manager.getConnection(mssql_url, mssql_user, mssql_pass) connection.prepareCall("EXEC sys.sp_tables").execute() connection.close()The above code works fine but however...

Data Engineering
Reply
  • 12096 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
judyy
New Contributor III
  • 6 kudos

This blog helped me with the output of the stored procedure: https://medium.com/@judy3.yang/how-to-run-sql-procedure-in-databricks-notebook-e28023555565

  • 6 kudos
4 More Replies
vk217
by • Contributor
  • 11214 Views
  • 3 replies
  • 1 kudos

ModuleNotFoundError: No module named 'pyspark.dbutils'

I have a class in a python file like this from pyspark.sql import SparkSession from pyspark.dbutils import DBUtils class DatabricksUtils: def __init__(self‌‌): self.spark = SparkSession.getActiveSession() self.dbutils = DBUtil...

Data Engineering
Reply
  • 11214 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Jarkrung
New Contributor II
  • 1 kudos

Hi, we are also in the same exact situation. Were you able to solve the problem? Or a workaround maybe.

  • 1 kudos
2 More Replies
EdemSeitkh
by • New Contributor III
  • 5528 Views
  • 5 replies
  • 0 kudos

Resolved! Pass catalog/schema/table name as a parameter to sql task

Hi, i am trying to pass catalog name as a parameter into query for sql task, and it pastes it with single quotes, which results in error. Is there a way to pass raw value or other possible workarounds? query:INSERT INTO {{ catalog }}.pas.product_snap...

Data Engineering
Reply
  • 5528 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
lathaniel
New Contributor III
  • 0 kudos

@EdemSeitkh  can you elaborate on your workaround? Curious how you were able to implement an enum paramter in DBSQL.I'm running into this same issue now.

  • 0 kudos
4 More Replies
Abhikchoudhury
by • New Contributor II
  • 1145 Views
  • 0 replies
  • 1 kudos

Unable to see / download course materials in Advanced Data Engineering with Databricks ID: E-VDG8QV

I am trying to download course materials(.dbc file and the presentation slides) in Advanced Data Engineering with Databricks ID: E-VDG8QV. However I do not see those materials even when I scroll all the way down in that page. I have tried multiple br...

Data Engineering
Reply
  • 1145 Views
  • 0 replies
  • 1 kudos
gianni77
by • New Contributor
  • 58041 Views
  • 13 replies
  • 4 kudos

How can I export a result of a SQL query from a databricks notebook?

The "Download CSV" button in the notebook seems to work only for results <=1000 entries. How can I export larger result-sets as CSV?

Data Engineering
Reply
  • 58041 Views
  • 13 replies
  • 4 kudos
View Replies
Latest Reply
igorstar
New Contributor III
  • 4 kudos

If you have a large dataset, you might want to export it to a bucket in parquet format from your notebook:%python df = spark.sql("select * from your_table_name") df.write.parquet(your_s3_path) 

  • 4 kudos
12 More Replies
Mits
by • New Contributor II
  • 2813 Views
  • 4 replies
  • 3 kudos

Sending email alerts to non-databricks user

I am trying to send email alerts to a non databricks user. I am using Alerts feature available in SQL. Can someone help me with the steps.Do I first need to first add Notification Destination through Admin settings and then use this newly added desti...

Data Engineering
Reply
  • 2813 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Mitali Lad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 3 kudos
3 More Replies
Phani1
by • Valued Contributor II
  • 650 Views
  • 1 replies
  • 0 kudos

integrating Azure Databricks with AAD

Hi Team, Could you please provide the details/process for integrating Azure Databricks - Unity Catalog and AAD? Regards,Phani

Data Engineering
delta
Reply
  • 650 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
raphaelblg
Databricks Employee
  • 0 kudos

Hello @Phani1 ,These doc pages might be useful for you: Set up and manage Unity CatalogSync users and groups from Microsoft Entra ID 

  • 0 kudos
ismaelhenzel
by • Contributor
  • 849 Views
  • 1 replies
  • 1 kudos

Upsert into a Delta Lake table with merge when using row masking function

I'm using databricks rls functions on my tables, and i need to make some merges into, but tables with rls functions does not support merge operations (https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html#limitation...

ismaelhenzel_0-1716979371091.png
Data Engineering
Reply
  • 849 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
raphaelblg
Databricks Employee
  • 1 kudos

Hi @ismaelhenzel, if you want to use the "MERGE INTO" sql command, you must turn-off rls. This is by design.

  • 1 kudos
Mathias
by • New Contributor II
  • 517 Views
  • 1 replies
  • 0 kudos

Delay rows coming into DLT pipeline

Backgroundand requirements: We are reading data from our factory and storing it in a DLT table called telemetry with columns sensorid, timestamp and value. We need to get rows where sensorid is “qrreader-x” and join with some other data from that sam...

Data Engineering
Reply
  • 517 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
raphaelblg
Databricks Employee
  • 0 kudos

Hi @Mathias,  I'd say that watermarking might be a good solution for your use case. Please check Control late data threshold with multiple watermark policy in Structured Streaming.  If you want to dig-in further there's also: Spark Structured Streami...

  • 0 kudos
surband
by • New Contributor III
  • 1294 Views
  • 1 replies
  • 0 kudos

Databricks Run Notebook GitHub Action

The GitHub action databricks/run-notebook to deploy and run a notebook from GitHub to DBX awaits the completion of the Job. The pulsar streaming job that I have is a long running job due to which the Action times out when the access token it uses to ...

Data Engineering
Reply
  • 1294 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
surband
New Contributor III
  • 0 kudos

https://github.com/databricks/run-notebook/issues/53#issue-2321682696

  • 0 kudos
190809
by • Contributor
  • 2144 Views
  • 3 replies
  • 2 kudos

Is there a way to add a date parameter to the jobs run API call?

Hi there I am currently making a call to the Databricks API jobs run endpoint. I would like to make this call on a daily basis to get data on the jobs run in the past 24 hours and add this to my delta table. Is there a way to set a GTE value in the A...

Data Engineering
Reply
  • 2144 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
AdrianC
New Contributor II
  • 2 kudos

Actually the "start_time_to" parameter doesn't seem to work at all. Neither alone or together with "start_time_from" (Whenever used the api call returns nothing). I'd like to report this as an issue as we want to automate our cluster usage monitoring...

  • 2 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels