cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

yalei
by New Contributor
  • 5193 Views
  • 2 replies
  • 0 kudos

leaflet not works in notebook(R language)

I saw this notebook: htmlwidgets-azure - Databricks (microsoft.com)However, it is not reproducible. I got a lot errors:there is no package called ‘R.utils’. This is easy to fix, just install the package "R.utils""can not be unloaded". This is not ...

  • 5193 Views
  • 2 replies
  • 0 kudos
Latest Reply
KAdamatzky
New Contributor II
  • 0 kudos

Hi yalei,  Did you have any luck fixing this issue? I am also trying to replicate the htmlwidgets notebook and am running into the same error.Unfortunately, the suggestions provided by Kaniz_Fatma below did not work.

  • 0 kudos
1 More Replies
ksenija
by Contributor
  • 200 Views
  • 3 replies
  • 1 kudos

Resolved! DLT pipeline - silver table, joining streaming data

Hello!I'm trying to do my modeling in DLT pipelines. For bronze, I created 3 streaming views. When I try to join them to create silver table, I got an error that I can't join stream and stream without watermarks. I tried adding them but then I got no...

  • 200 Views
  • 3 replies
  • 1 kudos
Latest Reply
Ravivarma
New Contributor III
  • 1 kudos

Hello @ksenija , Greetings! Streaming uses watermarks to control the threshold for how long to continue processing updates for a given state entity. Common examples of state entities include: Aggregations over a time window. Unique keys in a join b...

  • 1 kudos
2 More Replies
ShankarM
by New Contributor III
  • 113 Views
  • 1 replies
  • 1 kudos

Resolved! Serverless feature audit in data engg.

As recently announced in the summit that notebooks, jobs, workflows will run in serverless mode, how do we track/debug the compute cluster metrics in this case especially when there are performance issues while running jobs/workflows.

  • 113 Views
  • 1 replies
  • 1 kudos
Latest Reply
imsabarinath
New Contributor II
  • 1 kudos

Databricks is planning to enable some system tables to capture some of these metrics and same can be leveraged for troubleshooting as starting point is my view

  • 1 kudos
OliverCadman
by New Contributor III
  • 9888 Views
  • 10 replies
  • 5 kudos

'File not found' error when executing %run magic command

I'm just walking through a simple exercise presented in the Databricks Platform Lab notebook, in which I'm executing a remote notebook from within using the %run command. The remote notebook resides in the same directory as the Platform Lab notebook,...

Data Engineering
%file_not_found
%magic_commands
%run
  • 9888 Views
  • 10 replies
  • 5 kudos
Latest Reply
MuthuLakshmi
New Contributor III
  • 5 kudos

The %run command is a specific Jupyter magic command. The ipykernel used in Databricks examines the initial line of code to determine the appropriate compiler or language for execution. To minimize the likelihood of encountering errors, it is advisab...

  • 5 kudos
9 More Replies
Oliver_Angelil
by Valued Contributor II
  • 6336 Views
  • 9 replies
  • 6 kudos

Resolved! Confusion about Data storage: Data Asset within Databricks vs Hive Metastore vs Delta Lake vs Lakehouse vs DBFS vs Unity Catalogue vs Azure Blob

Hi thereIt seems there are many different ways to store / manage data in Databricks.This is the Data asset in Databricks: However data can also be stored (hyperlinks included to relevant pages):in a Lakehousein Delta Lakeon Azure Blob storagein the D...

Screenshot 2023-05-09 at 17.02.04
  • 6336 Views
  • 9 replies
  • 6 kudos
Latest Reply
Rahul_S
New Contributor II
  • 6 kudos

Informative.

  • 6 kudos
8 More Replies
jwilliam
by Contributor
  • 2285 Views
  • 4 replies
  • 7 kudos

Resolved! Has Unity Catalog been available in Azure Gov Cloud?

We are using Databricks with Premium Tier in Azure Gov Cloud. We check the Data section but don't see any options to Create Metastore.

  • 2285 Views
  • 4 replies
  • 7 kudos
Latest Reply
User16672493709
New Contributor III
  • 7 kudos

Azure.gov does not have Unity Catalog (as of July 2024). I think previous responses missed the context of government cloud in OP's question. UC has been open sourced since this question was asked, and is a more comprehensive solution in commercial cl...

  • 7 kudos
3 More Replies
DmitriyLamzin
by New Contributor
  • 3620 Views
  • 4 replies
  • 0 kudos

applyInPandas started to hang on the runtime 13.3 LTS ML and above

Hello, recently I've tried to upgrade my runtime env to the 13.3 LTS ML and found that it breaks my workload during applyInPandas.My job started to hang during the applyInPandas execution. Thread dump shows that it hangs on direct memory allocation: ...

Data Engineering
pandas udf
  • 3620 Views
  • 4 replies
  • 0 kudos
Latest Reply
Daisy98
New Contributor II
  • 0 kudos

The applyInPandas function may hang on Databricks Runtime 13.3 LTS ML and later versions owing to changes or inefficiencies in how the runtime handles parallel processing. Consider evaluating recent revisions or implementing alternative DataFrame ope...

  • 0 kudos
3 More Replies
bricksdata
by New Contributor
  • 6546 Views
  • 4 replies
  • 0 kudos

Unable to authenticate against https://accounts.cloud.databricks.com as an account admin.

ProblemI'm unable to authenticate against the https://accounts.cloud.databricks.com endpoint even though I'm an account admin. I need it to assign account level groups to workspaces via the workspace assignment api (https://api-docs.databricks.com/re...

  • 6546 Views
  • 4 replies
  • 0 kudos
Latest Reply
137292
New Contributor II
  • 0 kudos

From this doc: To automate Databricks account-level functionality, you cannot use Databricks personal access tokens. Instead, you must use either OAuth tokens for Databricks account admin users or service principals. For more information, see:Use a s...

  • 0 kudos
3 More Replies
John_Rotenstein
by New Contributor II
  • 6687 Views
  • 8 replies
  • 4 kudos

Retrieve job-level parameters in Python

Parameters can be passed to Tasks and the values can be retrieved with:dbutils.widgets.get("parameter_name")More recently, we have been given the ability to add parameters to Jobs.However, the parameters cannot be retrieved like Task parameters.Quest...

  • 6687 Views
  • 8 replies
  • 4 kudos
Latest Reply
xiangzhu
Contributor II
  • 4 kudos

ah sorry, the thread asked for notebooks too.nevertheless, I'm search for getting job params in pure python jobs

  • 4 kudos
7 More Replies
Phani1
by Valued Contributor
  • 4961 Views
  • 8 replies
  • 8 kudos

Delta Live Table name dynamically

Hi Team,Can we pass Delta Live Table name dynamically [from a configuration file, instead of hardcoding the table name]? We would like to build a metadata-driven pipeline.

  • 4961 Views
  • 8 replies
  • 8 kudos
Latest Reply
Vic01
New Contributor II
  • 8 kudos

Hello, I wonder if there is any update for this feature?Thanks

  • 8 kudos
7 More Replies
rk1994
by New Contributor
  • 182 Views
  • 2 replies
  • 0 kudos

Incrementally ingesting from a static db into a Delta Table

Hello everyone,I’m very new to Delta Live Tables (and Delta Tables too), so please forgive me if this question has been asked here before.Some context: I have over 100M records stored in a Postgres table. I can connect to this table using the convent...

  • 182 Views
  • 2 replies
  • 0 kudos
Latest Reply
TPSteve
New Contributor II
  • 0 kudos

First, you need to understand why your current solution is failing.Materialized views and views in DLT don't differ conceptually from mat. views and views in PostgreSQL. Every time the pipeline is run, both the mat. view and the view will be recalcul...

  • 0 kudos
1 More Replies
Shivam_Pawar
by New Contributor III
  • 9773 Views
  • 12 replies
  • 4 kudos

Databricks Lakehouse Fundamentals Badge

I have successfully passed the test after completion of the course with 95%. But I have'nt recieved any badge from your side as promised. I have been provided with a certificate which looks fake by itself. I need to post my credentials on Linkedin wi...

  • 9773 Views
  • 12 replies
  • 4 kudos
Latest Reply
Elham
New Contributor II
  • 4 kudos

hello, I'm trying to login to following URL :https://v2.accounts.accredible.com/login?app=recipient-portal&origin=https:%2F%2Fcredentials.databricks.com%2Fissuer%2F45847%2Fcredentials&language=enbut I received an error and the message is :Sorry, we c...

  • 4 kudos
11 More Replies
AWS1567
by New Contributor III
  • 15108 Views
  • 10 replies
  • 5 kudos

We've encountered an error logging you in.

I'm trying to login for past two days and i'm still facing this error: "We've encountered an error logging you in." I've tried to reset the password multiple times and nothing happened. My friend is also not able to login. I request you to resolve t...

Databricks_login_issue
  • 15108 Views
  • 10 replies
  • 5 kudos
Latest Reply
AlxMares
New Contributor II
  • 5 kudos

I had to sign up again using the same email. In my case, the error seemed to be related to my Azure account. So, when Databricks asks you to choose your cloud provider to sync with, you should select the community version instead.

  • 5 kudos
9 More Replies
Avinash_Narala
by Contributor
  • 159 Views
  • 2 replies
  • 1 kudos

Custom Endpoints for AI functions In Databricks

Hi Community.Recently I gone through the AI Functions and amazed by the results.I just wanted to know whether can we use our custom endpoints(instead of databricks foundational models) and leverage this AI Functions(ai_classify, ai_mask, etc)https://...

  • 159 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Avinash_Narala, I’m glad to hear that you’re excited about AI Functions in Databricks.  The article provides more details on setting up web endpoints for use with Custom Commands in Azure. While this is specific to Azure, the general principles o...

  • 1 kudos
1 More Replies
rimaissa
by New Contributor II
  • 166 Views
  • 2 replies
  • 0 kudos

Autoloader file notification mode error using UC

We have a DLT pipeline we've created that is using autoloader file notification mode. The pipeline ran fine before moving it to UC. Now that we're using UC, we are getting an AWS permissions issue when the autoloader file notification mode is set to ...

  • 166 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @rimaissa,  Ensure that the user or service principal running the DLT pipeline has the necessary permissions to access the S3 bucket and set up the required cloud resources (SNS, SQS) in the Unity Catalog context. This may require additional permi...

  • 0 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels