cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

db-avengers2rul
by • Contributor II
  • 3708 Views
  • 3 replies
  • 2 kudos

Resolved! Documentation - notebook not working

Dear Team,While practising few examples i have noticed the below notebook is not fetching the full dataset and also no schema is fetched https://docs.databricks.com/_static/notebooks/widget-demo.htmlcan you please re try and let me know the results N...

  • 3708 Views
  • 3 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Databricks MVP
  • 2 kudos

I think https://health.data.ny.gov/api/views/myeu-hzra/rows.csv was a public dataset, but now it shows authentication_required, so this error is independent of databricks.But the good news is that I was able to generate a new URL on the New Your heal...

  • 2 kudos
2 More Replies
subhransu02
by • New Contributor II
  • 2260 Views
  • 2 replies
  • 2 kudos

Databricks Lakehouse Fundamentals badge not received

I have completed and passed the short assessment for Lakehouse fundamentals but I didn't receive any badge. I have also checked in credentials.databricks.com but I don't see any badge.

  • 2260 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vartika
Databricks Employee
  • 2 kudos

Hey @Subhransu Ranjan Sankhua​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly. 

  • 2 kudos
1 More Replies
Anonymous
by • Not applicable
  • 9256 Views
  • 4 replies
  • 1 kudos

Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel is not whitelisted when mounting a s3 bucket

Hello all, I'm experiencing this issueConstructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel is not whitelisted when I'm trying to mount a s3 bucket. %python dbutils.fs.mount("s3a://dd-databricks-staging-storage/data/staging/datalak...

  • 9256 Views
  • 4 replies
  • 1 kudos
Latest Reply
leonids2005
New Contributor II
  • 1 kudos

WE have this problem running cluster with 11.2 and shared access mode. spark.databricks.pyspark.enablePy4JSecurity false - this does not help because it says spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing access modehere is ...

  • 1 kudos
3 More Replies
sgarcia
by • Databricks Partner
  • 6069 Views
  • 4 replies
  • 1 kudos

Call scala application jar in notebook

Hi,Is there any way to execute jar scala-spark application inside the notebook, without using jobs?I have different jars for different intakes and I want to call them from a notebook, so I could call them in a parameterized way.Thanks

  • 6069 Views
  • 4 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @Sergio Garccia​ ,Just a friendly follow-up. Do you still need help? have you check our docs? This might help https://docs.databricks.com/workflows/jobs/jobs.html#jar-jobs-1

  • 1 kudos
3 More Replies
djfliu
by • New Contributor III
  • 2202 Views
  • 0 replies
  • 3 kudos

Getting a com.databricks.s3commit.S3CommitRejectException exception error when using structured streaming to write to a delta table on s3.

The full error is below: An error occurred while calling o95098.execute.: com.databricks.s3commit.S3CommitRejectException: rejected by server26 times at com.databricks.s3commit.S3CommitClientImpl.commit(S3CommitClient.scala:303)It was a one-off inci...

  • 2202 Views
  • 0 replies
  • 3 kudos
VinayEmmadi
by • New Contributor
  • 1674 Views
  • 0 replies
  • 0 kudos

%run not working as expected

I have a quick question about %run <notebook path>. I am using the %run command to import functions from a notebook. It works fine when I run %run once. But when I run two %run commands, I lose the reference from the first %run. I get NameError when ...

  • 1674 Views
  • 0 replies
  • 0 kudos
marco_almeida
by • New Contributor II
  • 3094 Views
  • 2 replies
  • 2 kudos

I can't import an library like the example

I read this article and I created a notebook to use like a library but when I tried to import it in other notebook I received this error: No module named 'lib.lib_test' No module named 'lib.lib_*****'

  • 3094 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey @Marco Antônio de Almeida Fernandes​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love...

  • 2 kudos
1 More Replies
MaverickF14
by • New Contributor II
  • 4632 Views
  • 6 replies
  • 2 kudos

From a noob Databrickser... concerning Python programming in databricks

The following...We,ve got clients working with us in contracts. Per client several contracts of a certain type with start- and end dates. If I need aggregated info per client in one record like:how many different contract did the client haveof which ...

  • 4632 Views
  • 6 replies
  • 2 kudos
Latest Reply
MaverickF14
New Contributor II
  • 2 kudos

Yeah, Thanks for all the help!

  • 2 kudos
5 More Replies
alexgv12
by • New Contributor III
  • 14485 Views
  • 2 replies
  • 0 kudos

How can I somehow run spark.something in a worker? - rdd foreach spark.context

i am using rdd to parallelize a function, in this function i format the record i want to save, how can i store from this function the record with a dataframe? because every time i use spark..... an error is generated Caused by: org.apache.spark.api.p...

  • 14485 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @alexander grajales vanegas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear ...

  • 0 kudos
1 More Replies
HariharaSam
by • Databricks Partner
  • 5389 Views
  • 2 replies
  • 3 kudos

Enabling Fair Scheduler from Databricks Notebook

How to enable fair scheduler from Databricks notebook using python commands?

  • 5389 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Hariharan Sambath​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 3 kudos
1 More Replies
Braxx
by • Contributor II
  • 2846 Views
  • 2 replies
  • 3 kudos

List mounts for specyfic scope

I have like 40 mounts and some of them were done based on the scopes which now I want to remove. How can I list the mounts for specyfic scope or even better, get the list of all the mounts alongside the scopes?

  • 2846 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Bartosz Wachocki​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fro...

  • 3 kudos
1 More Replies
Pat
by • Esteemed Contributor
  • 25642 Views
  • 7 replies
  • 18 kudos

Resolved! Cluster Modes - High Concurrency

It took me quite some time to find the option to create a cluster in High Concurrency mode. It was hidden in the new UI.What should be the way to access the data with TAC?What is the equivalent mode to work with TAC ?Does it mean that we are being pu...

image image.png image
  • 25642 Views
  • 7 replies
  • 18 kudos
Latest Reply
Prabakar
Databricks Employee
  • 18 kudos

Thanks. Always happy to help.

  • 18 kudos
6 More Replies
RamaSantosh
by • New Contributor II
  • 6565 Views
  • 2 replies
  • 3 kudos

Data load from Azure databricks dataframe to cosmos db container

I am trying to load data from Azure databricks dataframe to cosmos db container using below commandcfg = { "spark.cosmos.accountEndpoint" : cosmosEndpoint, "spark.cosmos.accountKey" : cosmosMasterKey, "spark.cosmos.database" : cosmosDatabaseName, "sp...

  • 6565 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey @Rama Santosh Ravada​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 3 kudos
1 More Replies
Praveen2609
by • New Contributor
  • 3949 Views
  • 2 replies
  • 0 kudos

dbfs access for job clusters and interactive cluster

Hi All,I am new to databricks need some understanding for my requirement .our requirement:a: we have zip file in azure blob storage and we are bringing that file to dbfs and unzip that file and executing our transformations in multiple steps (3 steps...

  • 3949 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @praveen rajak​ Does @Debayan Mukherjee​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
Trey
by • New Contributor III
  • 3392 Views
  • 2 replies
  • 3 kudos

Where do you usually store and manage "JDBC credentials" to use on databricks notebook?

Hi all,I would like to improve the way I use JDBC credenditial information (ID/PW, host, port, etc)Where do you guys usually store and use the jdbc credentials?Thanks for your help in advance!

  • 3392 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Kwangwon Yi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 3 kudos
1 More Replies
Labels