cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

brickster_2018
by Databricks Employee
  • 5423 Views
  • 2 replies
  • 0 kudos

Resolved! The job fails with HTTP 403

My jobs that are running for more than 48 hours are failing with HTTP 403 error

  • 5423 Views
  • 2 replies
  • 0 kudos
Latest Reply
willjoe
New Contributor III
  • 0 kudos

Check for URL errors and make sure you're specifying an actual web page file name and extension, not just a directory. Most websites are configured to disallow directory browsing, so a 403 Forbidden message when trying to display a folder instead of ...

  • 0 kudos
1 More Replies
chinoiserie
by New Contributor II
  • 5513 Views
  • 6 replies
  • 5 kudos

406 error code

https://databricks.com/try-databricks not acceptable, 406 error codeWhat should I do?

  • 5513 Views
  • 6 replies
  • 5 kudos
Latest Reply
willjoe
New Contributor III
  • 5 kudos

How Do I Fix HTTP 406 Not Acceptable?Troubleshooting on the Client-Side.Check Your URL. Rollback Recent Upgrades.Uninstall New Extensions, Modules, or Plugins.Check for Unexpected Database Changes. Troubleshooting on the Server-Side. Check the Config...

  • 5 kudos
5 More Replies
spdteng
by New Contributor III
  • 2067 Views
  • 2 replies
  • 3 kudos

Resolved! Voucher code error

Hi,I received a voucher in Nov 2020 for 'Databricks Certified Associate Developer for Apache Spark 3.0 exam' with an expiry date in Nov 2022. However I receive an error stating the promotion code is used. I have never used the code. Any idea who to c...

  • 2067 Views
  • 2 replies
  • 3 kudos
Latest Reply
willjoe
New Contributor III
  • 3 kudos

If your voucher code is not working, it could be due to the following reasons:The voucher code has expiredProducts in your shopping cart are not eligible for the discountTerms and conditions are not metIf applying a "new user only voucher" and our sy...

  • 3 kudos
1 More Replies
Atul_Sharan
by New Contributor II
  • 5532 Views
  • 3 replies
  • 3 kudos

Resolved! Error Code: 3206 - Processed HTTP request failed.

The ADF(Azur Data Factory) pipelines jobs executing several Databricks Notebook activities in parallel have been failing regularly with the following error "Error Code: 3206 - Processed HTTP request failed." The issue gets resolved on its own upon re...

  • 5532 Views
  • 3 replies
  • 3 kudos
Latest Reply
willjoe
New Contributor III
  • 3 kudos

Method 1 - Close Conflicting Programsdown voteWhen you get a runtime error, keep in mind that it is happening due to programs that are conflicting with each other. The first thing you can do to resolve the problem is to stop these conflicting program...

  • 3 kudos
2 More Replies
Pavan1
by New Contributor II
  • 2480 Views
  • 1 replies
  • 2 kudos

Does Spark MLlib support Generalized Additive Modeling? How does one go about implementing GAM models in Spark?

I want to implement GAM (Generalized additive model) model in Spark. Based on my research on online forums, I could not find the implementation of GAM models on Spark. Has anyone in this community attempted this? Does Spark MLlib support GAM?

  • 2480 Views
  • 1 replies
  • 2 kudos
Latest Reply
Pavan1
New Contributor II
  • 2 kudos

Hi @Kaniz Fatma​, thanks for sharing this. We ended up using pyGAM library in Python for this. This PDF is a good introduction. I will share my learnings once we complete our experiments.

  • 2 kudos
BradSheridan
by Valued Contributor
  • 2257 Views
  • 4 replies
  • 0 kudos

Resolved! Add custom tags to DLT cluster

We have a policy in our AWS account that whenever an EC2 instance is created, there are 5 mandatory tags that need to be added. When we create an All-Purpose cluster in the Databricks console, we can easily add these under Advance Options > Tags. How...

  • 2257 Views
  • 4 replies
  • 0 kudos
Latest Reply
BradSheridan
Valued Contributor
  • 0 kudos

@Tomasz Bacewicz​ Perfect!! worked like a charm...thank you!!do you think auto loader or DLT is a better approach to implementing a CDC pipeline in Databricks?

  • 0 kudos
3 More Replies
abhay_sud
by New Contributor II
  • 4942 Views
  • 3 replies
  • 3 kudos

Delta Live Table Pipeline Load Error

I am getting this error when running Delta Live Table Pipelines pulling from a source in our sandbox's local dbfs folder. It says the user is not authorized to perform this operation, whereas I am able to see the data when I run a simple select state...

image image image
  • 4942 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Abhay Sudhakaran​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution and mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...

  • 3 kudos
2 More Replies
smanchem
by New Contributor III
  • 2593 Views
  • 3 replies
  • 8 kudos

Resolved! AWS Private Link for DB workspace

What's the best way to set up a private link connection to Databricks without loosing the current settins? workspace

  • 2593 Views
  • 3 replies
  • 8 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 8 kudos

At the data and AI summit, there was a nice talk about AWS private link. Here is a link if you have access:https://dataaisummit.com/session-virtual/?v2477da705118cc74fd14460db021e1784e2eed5a7982c6482ec95cb2e86d259644b8741959f52a49e0e6908b82a9d860=C22...

  • 8 kudos
2 More Replies
Kaher
by New Contributor
  • 2955 Views
  • 3 replies
  • 3 kudos
  • 2955 Views
  • 3 replies
  • 3 kudos
Latest Reply
Rheiman
Contributor II
  • 3 kudos

For general cluster decision making refer to this article https://docs.microsoft.com/en-gb/azure/databricks/clusters/cluster-config-best-practicesOnce you've selected a cluster that makes sense, run it and check your ganglia metrics to see whether yo...

  • 3 kudos
2 More Replies
Deepak_Goldwyn
by New Contributor III
  • 3345 Views
  • 5 replies
  • 2 kudos

Resolved! DLT Pipeline and Job Cluster

We have written few python functions(methods within a class) and packaged them as a wheel library.In the as-is situation we use to install that wheel library in All-Purpose cluster that we already have created. It works fine.In the to-be situtation(D...

  • 3345 Views
  • 5 replies
  • 2 kudos
Latest Reply
tomasz
Databricks Employee
  • 2 kudos

Does it give you an error when running the DLT pipeline specifically on the %pip command or does it not work in some other way? If it's the former, could you share the path format that you're using for the %pip command path?

  • 2 kudos
4 More Replies
sage5616
by Valued Contributor
  • 5046 Views
  • 3 replies
  • 4 kudos

Resolved! Spark persistent view on a partition parquet file

In Spark, is it possible to create a persistent view on a partitioned parquet file in Azure BLOB? The view must be available when the cluster restarted, without having to re-create that view, hence it cannot be a temp view.I can create a temp view, b...

  • 5046 Views
  • 3 replies
  • 4 kudos
Latest Reply
sage5616
Valued Contributor
  • 4 kudos

Here is what worked for me. Hope this helps someone else: https://stackoverflow.com/questions/72913913/spark-persistent-view-on-a-partition-parquet-file/72914245#72914245CREATE VIEW test as select * from parquet.`/mnt/folder-with-parquet-file(s)/`@Hu...

  • 4 kudos
2 More Replies
ishantjain194
by New Contributor II
  • 1011 Views
  • 0 replies
  • 0 kudos

BEGINNER

Hey i am beginner and i am intrested to learn data engineering path for azure databrics.Can some one please provide some guidance

  • 1011 Views
  • 0 replies
  • 0 kudos
bala_kowsalya
by New Contributor II
  • 1475 Views
  • 1 replies
  • 2 kudos

How to get the normalized hours value for the Databricks EMR cluster?

Hi Hubert!I'm working on a usecase to get the compute usage stats. I used boto3 code and describe_cluster() function to get the normalized instance hours value for the EMR cluster. I would like to know if there is an equivalent for this normalized in...

  • 1475 Views
  • 1 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Hi Bala, nice to meet you.Yes, there is an equivalent. For example, in Databricks, you need to use the get method in cluster API or CLI implementation of API.https://docs.databricks.com/dev-tools/api/latest/clusters.html#get

  • 2 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels