cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Fnazar
by New Contributor II
  • 1005 Views
  • 3 replies
  • 0 kudos

Resolved! Migrate delta live tables from a non-uc to uc enabled databricks workspace

Hello Team,I have a usecase where I need to migrate Delta Live Tables from a non-Unity Catalog Databricks workspace to a Unity Catalog enabled Databricks workspace. What will be the efficient approach to achieve this scenario, please suggest?

  • 1005 Views
  • 3 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @Fnazar ,You can try to use UCX by Databricks Labs. It automates whole process. databrickslabs/ucx: Automated migrations to Unity Catalog (github.com)

  • 0 kudos
2 More Replies
varshini_reddy
by New Contributor III
  • 2734 Views
  • 2 replies
  • 1 kudos

Databricks Jobs For Each Functionality

Hi Team, I have two questions, please clarigy.1. can we use multiple tasks inside a for each for databricks jobs2. is nested for each functionality available?

  • 2734 Views
  • 2 replies
  • 1 kudos
Latest Reply
filipniziol
Contributor III
  • 1 kudos

Hi @varshini_reddy,While running multiple tasks within a for each loop is not available, there is a simple workaround.Workaround: create a main notebook to run multiple child notebooks:Suppose you need to run notebook1, notebook2, and notebook3 withi...

  • 1 kudos
1 More Replies
Ravikumashi
by Contributor
  • 3693 Views
  • 4 replies
  • 0 kudos

issue with azure databricks workspace after we disable public network access

Hi All,We had azure databricks workspaces created thru terraform with public network access enabled to true and everything was working great. recently we have disabled the public network access and started to face issues.terraform is uanble to add us...

  • 3693 Views
  • 4 replies
  • 0 kudos
Latest Reply
Mohamednazeer
New Contributor III
  • 0 kudos

Do you got any solution for this issue? I am also facing the same issue.

  • 0 kudos
3 More Replies
Henrik_
by New Contributor III
  • 757 Views
  • 4 replies
  • 4 kudos

Dashboard use case - order of bars

On a spark dataframe, is there any smart way to set the order of a categorical feature explicitly, equivalent to Categorical(ordered=list) in Pandas? The use case here is a dashboard in Databricks, and I want the bars to be arranged in certain order....

  • 757 Views
  • 4 replies
  • 4 kudos
Latest Reply
Henrik_
New Contributor III
  • 4 kudos

Thanks for your effort!

  • 4 kudos
3 More Replies
prabhjot
by New Contributor III
  • 3584 Views
  • 4 replies
  • 2 kudos

Resolved! Data lineage graph is not working

Hi Team,The issue - Data lineage graph is not working (16-feb, 17-18 Feb) –  I created the below tables but when I click the lineage graph not able to see the upstream or downstream table .... the + sign goes away after a few sec but not able to clic...

  • 3584 Views
  • 4 replies
  • 2 kudos
Latest Reply
Sikki
New Contributor III
  • 2 kudos

 Hi Kaniz,We're encountering the same issue where the lineage is not getting populated for a few tables. Could you let us know if a fix has been implemented in any runtime?"We are uaing job cluster 12.2.x .

  • 2 kudos
3 More Replies
kjoth
by Contributor II
  • 19684 Views
  • 9 replies
  • 7 kudos

How to make the job fail via code after handling exception

Hi , We are capturing the exception if an error occurs using try except. But we want the job status to be failed once we got the exception. Whats the best way to do that. We are using pyspark.

  • 19684 Views
  • 9 replies
  • 7 kudos
Latest Reply
kumar_ravi
New Contributor III
  • 7 kudos

you can do some hack arround   dbutils = get_dbutils(spark)    tables_with_exceptions = []    for table_config in table_configs:        try:            process(spark, table_config)        except Exception as e:            exception_detail = f"Error p...

  • 7 kudos
8 More Replies
monil
by New Contributor II
  • 953 Views
  • 3 replies
  • 1 kudos

/api/2.1/jobs/runs/get-output api response

/api/2.1/jobs/runs/get-output what are the possbile status or state values of the api?I am trying to check the status of my job run based on run id. but there is not enough detail on the response body which contains the status of the run.

  • 953 Views
  • 3 replies
  • 1 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 1 kudos

@monil It's documented well in the API documentation.https://docs.databricks.com/api/workspace/jobs/getrunoutput 

  • 1 kudos
2 More Replies
janwoj
by New Contributor II
  • 10273 Views
  • 4 replies
  • 1 kudos

PowerApps connection to Azure Databricks

Hello,​I would like to read Databricks delta table to show the data on the screen using PowerApps gallery and insert new records to the same table also. What is the best method to achieve an efficient connection and perform above?​​Cheers

  • 10273 Views
  • 4 replies
  • 1 kudos
Latest Reply
Chris_Shehu
Valued Contributor III
  • 1 kudos

Anyone find a solution to this yet? I'm currently investigating the same issue. Currently the only one I can find is paying for a third-party tool to set it up. Thanks,

  • 1 kudos
3 More Replies
PiotrU
by Contributor II
  • 2012 Views
  • 6 replies
  • 1 kudos

Resolved! Adding extra libraries to databricks (rosbag)

HelloI have interesting challenge, I am required to install few libraries which are part of rosbag packages, for allowing some data deserialization tasks.While creating cluster I do use init_script that install this software using apt    sudo apt upd...

lakime_0-1717597430889.png lakime_1-1717597470819.png
  • 2012 Views
  • 6 replies
  • 1 kudos
Latest Reply
amandaK
New Contributor II
  • 1 kudos

@PiotrU did adding the path to sys.path resolve all of your ModuleNotFoundErrors? i'm trying to do something similar and adding the path to the sys.path resolved ModuleNotFoundError for rclpy, but i continue to see others related to ros

  • 1 kudos
5 More Replies
pranathisg97
by New Contributor III
  • 6339 Views
  • 3 replies
  • 0 kudos

Control query caching using SQL statement execution API

I want to execute this statement using databricks SQL Statement Execution API. curl -X POST -H 'Authorization: Bearer <access-token>' -H 'Content-Type: application/json' -d '{"warehouse_id": "<warehouse_id>", "statement": "set us...

image.png
  • 6339 Views
  • 3 replies
  • 0 kudos
Latest Reply
SumitAgrawal454
New Contributor II
  • 0 kudos

Looking for solution as I am facing the exact same problem

  • 0 kudos
2 More Replies
LauJohansson
by Contributor
  • 850 Views
  • 3 replies
  • 3 kudos

Resolved! Delta live table: Retrieve CDF columns

I have want to use the apply_changes feature from a bronze table to a silver table.The bronze table have no "natural" sequence_by column. Therefore, I want to use the CDF column "_commit_timestamp" as the sequence_by.How do I retrieve the columns in ...

  • 850 Views
  • 3 replies
  • 3 kudos
Latest Reply
LauJohansson
Contributor
  • 3 kudos

Thank you @raphaelblg!I chose to write an article on the subject after this discussion: https://www.linkedin.com/pulse/databricks-delta-live-tables-merging-lau-johansson-cdtce/?trackingId=L872gj0yQouXgJudM75gdw%3D%3D

  • 3 kudos
2 More Replies
BillMarshall
by New Contributor
  • 1395 Views
  • 2 replies
  • 0 kudos

workflow permissions errors

I have a notebook that outputs an Excel file. Through trial and error, and after consulting with various forums I discovered  the .xlsx file needed to be written to a temp file and then copied to the volume in Unity Catalog.When I run the notebook by...

  • 1395 Views
  • 2 replies
  • 0 kudos
Latest Reply
emora
New Contributor III
  • 0 kudos

Hello, yes of course you need to write the excel file in the tmp folder, but then you can move it to whatever you want without problem. In my current project we implemented this method to create the file in the tmp folder, and then move it to one spe...

  • 0 kudos
1 More Replies
Subhasis
by New Contributor II
  • 799 Views
  • 5 replies
  • 0 kudos

Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all data

Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all data. I want to load all the data which are not processed . I don't want to relaod all the data.

  • 799 Views
  • 5 replies
  • 0 kudos
Latest Reply
Subhasis
New Contributor II
  • 0 kudos

Do checkpoint has some benchmark capacity after that it stops writing data? 

  • 0 kudos
4 More Replies
SowmyaDesai
by New Contributor II
  • 761 Views
  • 3 replies
  • 2 kudos

Run pyspark queries from outside databricks

I have written a Notebook which would execute pyspark query. I then execute it remotely from outside databricks environment using /api/2.1/jobs/run-now, which would then run the notebook. I also want to retrieve the results from this job execution. H...

  • 761 Views
  • 3 replies
  • 2 kudos
Latest Reply
SowmyaDesai
New Contributor II
  • 2 kudos

Thanks for responding. I did go through this link. It talks about executing on SQL warehouse though. Is there a way we can execute queries on Databricks clusters instead?Databricks has this connector for SQL https://docs.databricks.com/en/dev-tools/p...

  • 2 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels