cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

angel_ba
by New Contributor II
  • 3211 Views
  • 2 replies
  • 2 kudos

File Trigger using azure file share in unity Catalog

Hello, I have got the unity catalog eanbled in my workspace. the file srae manually copied by customers in azure file share(domain joint account, wabs) on adhoc basis. I would like to add a file trigger on the job so that as soon as file arrives in t...

  • 3211 Views
  • 2 replies
  • 2 kudos
Latest Reply
adriennn
Contributor III
  • 2 kudos

@Diego33 Kaniz is half-bot half-human, but unfortunately not gracing us with "sorry for the confusion" responses.After a quick search, I thought that maybe there's a possiblity to find use the web terminal and do a manual mount with the bash script t...

  • 2 kudos
1 More Replies
stevenayers-bge
by Contributor
  • 1045 Views
  • 2 replies
  • 2 kudos

DBUtils from databricks-connect and runtime are quite different libraries....

If you find yourself using dbutils in any of your code, and you're testing locally vs running on a cluster, there's a few gotchas to be very careful of when it comes to listing files in Volumes or files on DBFS.The DBUtils you'll use locally installe...

  • 1045 Views
  • 2 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 2 kudos

Hi @stevenayers-bge ,Thanks for sharing. I didn't know that these interfaces aren't align with each other.

  • 2 kudos
1 More Replies
tdk
by New Contributor III
  • 856 Views
  • 2 replies
  • 0 kudos

Resolved! Cannot install jar to cluster: invalid authority.

Hi allI want to access on-prem Oracle Database data from the python notebooks. However, the install of the jar (ojdbc8.jar) results in an error, which occurs while the cluster is starting up.The error message:"Library installation attempted on the dr...

  • 856 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

The error message suggests that the jar file located at abfss:/jars/ojdbc8.jar has an invalid authority. This could be due to a number of reasons such as incorrect file path, insufficient permissions, or network restrictions. Here are a few steps you...

  • 0 kudos
1 More Replies
juan_barreto
by New Contributor III
  • 1551 Views
  • 6 replies
  • 9 kudos

Problem with dropDuplicates in Databricks runtime 15.4LTS

Hi,I'm testing the latest version of the databricks runtime but I'm getting errors doing a simple dropDuplicates.Using the following codedata = spark.read.table("some_table") data.dropDuplicates(subset=['SOME_COLUMN']).count() I'm getting this error....

juan_barreto_0-1726153266526.png
  • 1551 Views
  • 6 replies
  • 9 kudos
Latest Reply
Witold
Honored Contributor
  • 9 kudos

Unless is was communicated as a breaking changes between major updates, it would be OK. But I can't find anything in the release notes, so it's a bug.

  • 9 kudos
5 More Replies
ossinova
by Contributor II
  • 1347 Views
  • 3 replies
  • 2 kudos

Reading data from S3 in Azure Databricks

Is it possible to create an external volume in Azure Databricks that points to an external S3 bucket so that I can read files for processing? Or is it only limited to ADLSv2?

  • 1347 Views
  • 3 replies
  • 2 kudos
Latest Reply
Ashley1
Contributor
  • 2 kudos

Yep, I'm keen to see this functionality as well.I think it is reasonable to expect external locations can be on diverse storage types (at least the big players). I can nicely control access to azure storage in UC but not S3.

  • 2 kudos
2 More Replies
del1000
by New Contributor III
  • 19089 Views
  • 8 replies
  • 3 kudos

Resolved! Is it possible to passthrough job's parameters to variable?

Scenario:I tried to run notebook_primary as a job with same parameters' map. This notebook is orchestrator for notebooks_sec_1, notebooks_sec_2, and notebooks_sec_3 and next. I run them by dbutils.notebook.run(path, timeout, arguments) function.So ho...

  • 19089 Views
  • 8 replies
  • 3 kudos
Latest Reply
nnalla
New Contributor II
  • 3 kudos

I am using getCurrentBindings(), but it returns an empty dictionary even though I passed parameters. I am running it in a scheduled workflow job

  • 3 kudos
7 More Replies
rendorHaevyn
by New Contributor III
  • 10843 Views
  • 5 replies
  • 0 kudos

Databricks SQL Warehouse did not auto stop after specified 90 minute interval - why not?

In this specific case, we're running a 2XSmall SQL Warehouse on Databricks SQL.In looking at the SQL Warehouse monitoring log for this cluster, we noticed:final query executed by user at 10:26 on 2023-06-20no activity for some time, yet cluster remai...

  • 10843 Views
  • 5 replies
  • 0 kudos
Latest Reply
jfidanquec
New Contributor II
  • 0 kudos

Also dealing with the same issue! Anybody has any idea how to check it? There is no sort of logs and no actual query happens

  • 0 kudos
4 More Replies
VeeruK
by New Contributor III
  • 2608 Views
  • 7 replies
  • 0 kudos

Databricks Lakehouse Fundamentals BadgeI have successfully passed the test after completion of the course "Databricks Lakehouse Fundamentals&quot...

Databricks Lakehouse Fundamentals BadgeI have successfully passed the test after completion of the course "Databricks Lakehouse Fundamentals". But I have'nt recieved any badge. I have been provided with a certificate only. Please provide me with th...

  • 2608 Views
  • 7 replies
  • 0 kudos
Latest Reply
data_learner
New Contributor II
  • 0 kudos

I'm having the same issue

  • 0 kudos
6 More Replies
Sangram
by New Contributor III
  • 3857 Views
  • 4 replies
  • 2 kudos

Turn on full screen for databricks training videos

It seems full screen option for databricks training videos are turned off. How to turn it on ?

  • 3857 Views
  • 4 replies
  • 2 kudos
Latest Reply
bennner
New Contributor II
  • 2 kudos

It sounds like the full-screen option is disabled by the platform hosting the Databricks training videos. If that's the case, it may be out of your control. However, you could try these workarounds:Browser Zoom: Use the zoom feature (Ctrl + "+" on Wi...

  • 2 kudos
3 More Replies
Mario_D
by New Contributor III
  • 941 Views
  • 2 replies
  • 1 kudos

Resolved! Foreign key constraint in a dlt pipeline

As primary/foreign key constraints are now supported/available in Databricks, how are foreign key constraints handled in a dlt pipeline, i.e if a foreign key constraint is violated, is the record logged as a data quality issue and still added to the ...

  • 941 Views
  • 2 replies
  • 1 kudos
Latest Reply
RCo
New Contributor III
  • 1 kudos

Hi @Mario_D!While primary & foreign key constraints are generally available in Databricks Runtime 15.2 and above, they are strictly informational only.This means that a primary key will not prevent duplicates from being added to a table and a foreign...

  • 1 kudos
1 More Replies
FG
by New Contributor II
  • 10348 Views
  • 5 replies
  • 1 kudos

Running unit tests from a different notebook (using Python unittest package) doesn't produce output (can't discover the test files)

I have a test file (test_transforms.py) which has a series of tests running using Python's unittest package. I can successfully run the tests inside of the file with expected output. But when I try to run this test file from a different notebook (run...

image.png image
  • 10348 Views
  • 5 replies
  • 1 kudos
Latest Reply
SpaceDC
New Contributor II
  • 1 kudos

Hello, I have exactly the same issue.In my case, using the ipytest library from Databricks clusters, this is the error that occurs when I try to run the tests:EEEEE [100%]============================================== ERRORS =========================...

  • 1 kudos
4 More Replies
epps
by New Contributor
  • 2112 Views
  • 1 replies
  • 0 kudos

400 Unable to load OAuth Config

I've enabled SSO for my Databricks account with Okta as the identity provider and tested the integration is working. I'm now trying to implement an on-behalf-of token exchange so that my API can make authenticate requests to Databricks's API (e.g. ) ...

  • 2112 Views
  • 1 replies
  • 0 kudos
Latest Reply
riyadh-ruhr
New Contributor II
  • 0 kudos

Hello ,Were able to fix the issue ? I'm trying to implement the same thing

  • 0 kudos
JUPin
by New Contributor II
  • 2219 Views
  • 3 replies
  • 0 kudos

REST API for Pipeline Events does not return all records

I'm using the REST API to retrieve Pipeline Events per the documentation:https://docs.databricks.com/api/workspace/pipelines/listpipelineeventsI am able to retrieve some records but the API stops after a call or two.  I verified the number of rows us...

  • 2219 Views
  • 3 replies
  • 0 kudos
Latest Reply
wise_owl
New Contributor III
  • 0 kudos

You can leverage this code base. It works as expected using "next_page_token" parameter-Don't forget to mark this solution as correct if this helped you  import requests token = 'your token' url = 'your URL' params = {'expand_tasks': 'true'} header...

  • 0 kudos
2 More Replies
himanshu_k
by New Contributor
  • 2484 Views
  • 1 replies
  • 0 kudos

Clarification Needed: Ensuring Correct Pagination with Offset and Limit in PySpark

Hi community,I hope you're all doing well. I'm currently engaged in a PySpark project where I'm implementing pagination-like functionality using the offset and limit functions. My aim is to retrieve data between a specified starting_index and ending_...

  • 2484 Views
  • 1 replies
  • 0 kudos
Latest Reply
wise_owl
New Contributor III
  • 0 kudos

You can leverage this code base. It works as expected using "next_page_token" parameter-Don't forget to mark this solution as correct if this helped you  import requests token = 'your token' url = 'your URL' params = {'expand_tasks': 'true'} header...

  • 0 kudos
tingwei
by New Contributor II
  • 3356 Views
  • 4 replies
  • 5 kudos

ISOLATION_STARTUP_FAILURE

Hi I'm getting error in my data pipeline[ISOLATION_STARTUP_FAILURE] Failed to start isolated execution environment. Please contact Databricks support. SQLSTATE: XXKSSit was working fine and suddenly it is keep failing. Please advice. 

  • 3356 Views
  • 4 replies
  • 5 kudos
Latest Reply
SP2
New Contributor II
  • 5 kudos

Hello Team, I'm unable to run UDF by using this this DBR. Has issue been fixed?

  • 5 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels