cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MetaRossiVinli
by Contributor
  • 6959 Views
  • 2 replies
  • 2 kudos

Resolved! DLT data quality UI was present last week. Now absent. Did I change a setting?

Last week, I started running a DLT pipeline with expectations that dropped rows on streaming live tables. In the side bar for a table, I saw a nice circular chart with Written/Dropped rows and Failed records stats.Today, I ran a similar DLT pipeline ...

  • 6959 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Kevin Rossi​ :The circular chart with Written/Dropped rows and Failed records stats that you saw in the sidebar of a table in Delta Live Tables (DLT) is a built-in feature called "Data Quality Metrics" that provides a visual representation of the da...

  • 2 kudos
1 More Replies
karthik_p
by Esteemed Contributor
  • 1216 Views
  • 1 replies
  • 4 kudos

Resolved! while creating serverless warehouse we are receiving below message <workspaceId> is no longer eligible for Serverless Compute. Please reach out to your administrator

HI team,As far as limitations and pre-requisites we have met all, able to create warehouse in other workspace which are part of same account, for one of the workspaces we are seeing above issue. we don't any clear error log other than <workspaceId >i...

  • 1216 Views
  • 1 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@karthik p​ Your workspace might have blocked from serverless feature if there are some uncleared bills. If that's not the case, please file a support case to us.

  • 4 kudos
yzhang
by New Contributor III
  • 1504 Views
  • 1 replies
  • 1 kudos

Is it possible to have customized task parameter variables

I found some Task parameter variables in this document, https://community.databricks.com/s/feed/0D58Y0000A7AUWhSQO. Sounds like it will be much flexible with double curly braces for Task parameter variables. However, from the documentation, seems onl...

  • 1504 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Yanan Zhang​ :As per the documentation you shared, Databricks Task parameter variables are used to parameterize notebook tasks in a Databricks workspace. These variables are used to pass values from the parent notebook to the child notebook that is ...

  • 1 kudos
Anonymous
by Not applicable
  • 3431 Views
  • 1 replies
  • 0 kudos

I am getting an exception "RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive."

I have a parquet dataframe df. I first add a column using df.withColumn("version",lit(currentTimestamp)) and append it a table db.tbl with format parquet and partitioned by the "version" column. I then ran MSCK REPAIR TABLE db.tbl. I have then create...

Image
  • 3431 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@vikashk84The exception "RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive" typically occurs when there is an issue with Hive metadata related to partitioning in Databricks. Here are a few steps you ...

  • 0 kudos
kll
by New Contributor III
  • 2354 Views
  • 2 replies
  • 0 kudos

geojson file doesn't plot points on mapbox in jupyter notebook

I am running jupyter notebook on a Databricks cluster and attempting to render a map. I tried the example from the documentation to test and it doesn't plot the points.   # Load data from sample csv  data_url = 'https://raw.githubusercontent.com/mapb...

  • 2354 Views
  • 2 replies
  • 0 kudos
Latest Reply
kll
New Contributor III
  • 0 kudos

@Kaniz Fatma​  Do you have any updates? were you able to reproduce the error? i am not able to plot within jupyter notebook.

  • 0 kudos
1 More Replies
Tico23
by Contributor
  • 4648 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks Lakehouse Fundamentals Training -- Voucher

I just attended the Databricks Lakehouse Fundamentals Training (28 March 2023) and I am still wondering if I should expect any email with infos regarding a survey I should do and the voucher I should get. The Webcast finished without giving any infos...

dbl_webcast
  • 4648 Views
  • 5 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Alexander Mora Araya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 2 kudos
4 More Replies
Herkimer
by New Contributor II
  • 1286 Views
  • 1 replies
  • 0 kudos

Acceptable pythons to use with databricks-sql-cli (dbsqlcli)?

I have recently had to rebuild my laptop. Before it crashed I was able to use the dbsqlcli and have a set of scripts which I was using to do simple extracts into text files.The rebuild shows Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 ...

  • 1286 Views
  • 1 replies
  • 0 kudos
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

@John Zajic​ you need -Python 3.7 or higher. To check whether you have Python installed, run the command Install Python, if you do not have it already installed.I believe one version should be sufficient.

  • 0 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 944 Views
  • 1 replies
  • 8 kudos

Databricks will release a new navigation experience to enhance user efficiency and convenience. The update includes a single, unified homepage with a ...

Databricks will release a new navigation experience to enhance user efficiency and convenience. The update includes a single, unified homepage with a list of recently viewed objects, a single search box, and common Lakehouse categories at the top of ...

nav-2-graphic
  • 944 Views
  • 1 replies
  • 8 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 8 kudos

Thank you for sharing @Hubert Dudek​ 

  • 8 kudos
Tjadi
by New Contributor III
  • 1751 Views
  • 2 replies
  • 4 kudos

Specifying cluster on running a job

Hi,Let's say that I am starting jobs with different parameters at a certain time each day in the following manner:response = requests.post( "https://%s/api/2.0/jobs/run-now" % (DOMAIN), headers={"Authorization": "Bearer %s" % TOKEN}, json={ ...

  • 1751 Views
  • 2 replies
  • 4 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 4 kudos

@Tjadi Peeters​ You can select option Autoscaling/Enhanced Scaling in workflows which will scale based on workload

  • 4 kudos
1 More Replies
Aasu
by New Contributor II
  • 1967 Views
  • 3 replies
  • 1 kudos

In azure databricks custom alert i am able to preview but when it goes to mail data gets missing

In azure databricks custom alert i am able to preview but when it goes to mail data gets missing

  • 1967 Views
  • 3 replies
  • 1 kudos
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, what is the custom alert about? Also, could you please raise an Azure support case on the same to triage this further? Please tag @Debayan​ with your next response which will notify me. Thank you!

  • 1 kudos
2 More Replies
Eric-JoelBlanco
by New Contributor II
  • 2253 Views
  • 1 replies
  • 2 kudos

Execute code on Application End

Hello,I want to execute a custom code onApplicationEnd. Outisde Databricks, I have used the Spark Listener onApplicationEnd without problems. But it is not working on Databricks (I tried listener onJobEnd and this one worked).I have also tried Spark ...

  • 2253 Views
  • 1 replies
  • 2 kudos
Latest Reply
abhilash
New Contributor II
  • 2 kudos

Did you find any solution?

  • 2 kudos
Chris_Shehu
by Valued Contributor III
  • 4518 Views
  • 7 replies
  • 4 kudos

On-Behalf of tokens disabled for Azure Environments?

While trying to setup a Power BI connection to the Azure Delta Lake we ran into several issues around Service Principals. ​1) The API listed on the learn.microsoft site (link 1 below) indicates that there is an API you can use to create SP tokens. Wh...

  • 4518 Views
  • 7 replies
  • 4 kudos
Latest Reply
meetskorun
New Contributor II
  • 4 kudos

hello,i am new here from india, here to share some thoughts with you all

  • 4 kudos
6 More Replies
Simmy
by New Contributor II
  • 2493 Views
  • 3 replies
  • 1 kudos

Databricks Repos are pushing all changes to GitHub

When I make any changes within a repo, when I go to commit and push to GitHub if I uncheck any changes that I don't want pushed, still get pushed to Github. Any help would be appreciated

  • 2493 Views
  • 3 replies
  • 1 kudos
Latest Reply
Simmy
New Contributor II
  • 1 kudos

Hi @Vidula Khanna​ Problem is now resolved thanks, didn't have to do anything different, the functionality just started working as expected.

  • 1 kudos
2 More Replies
dcantos
by New Contributor II
  • 6037 Views
  • 5 replies
  • 0 kudos

IP Access list Databricks

Hello, good morning everyone, I have a problem, I have IP access List activated in my Databricks workspace and I have connections to powerBI and Azure DevOps, in Power BI I have already added all the public IP ranges that Azure gives me, which are ab...

  • 6037 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @David Cantos​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
4 More Replies
uv
by New Contributor II
  • 6102 Views
  • 3 replies
  • 2 kudos

Parquet to csv delta file

Hi Team, I have a parquet file in s3 bucket which is a delta file I am able to read it but I am unable to write it as a csv file.​getting the following error when i am trying to write:​A transaction log for Databricks Delta was found at `s3://path/a...

  • 6102 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @yuvesh kotiala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels