cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Ian_Neft
by New Contributor
  • 5789 Views
  • 2 replies
  • 0 kudos

Data Lineage in Unity Catalog not Populating

I have been trying to get the data lineage to populate with the simplest of queries on a unity enabled catalog with a unity enabled cluster.  I am essentially running the example provided with more data to see how it works with various aggregates dow...

  • 5789 Views
  • 2 replies
  • 0 kudos
Latest Reply
davood_NL
New Contributor II
  • 0 kudos

I have exactly the same issue, create the sample tables , everything seems fine but I can not see any data in lineage tab page in databricks .can anybody say some other thing to help me, because that answer doesn't work for me

  • 0 kudos
1 More Replies
SureshKumarDV
by New Contributor II
  • 1147 Views
  • 2 replies
  • 1 kudos

Not able to find the DBAcademyDLT policy to create the DLT Pipeline

Hi,Greetings of the day, I am preparing for the Databricks Engineering Associate Certification and following the Databricks Academy V3 course. As part of this course, trying to create the DLT pipelines but couldn't able to see the DBAcademyDLT policy...

  • 1147 Views
  • 2 replies
  • 1 kudos
Latest Reply
ManyPixels
New Contributor II
  • 1 kudos

I had to go to Admin Settings > Advanced (Workspace settings) and enable "Cluster, Pool and Jobs Access Control". Afterwards I was able to create an instance pool.

  • 1 kudos
1 More Replies
dataguru
by New Contributor II
  • 1354 Views
  • 2 replies
  • 1 kudos

Hello unable to start using the notebook

   I get this error when I run the Databricks training notebook%run ../Includes/Classroom-Setup-01 Resetting the learning environment: | dropping the catalog "***_53sh_da"...(0 seconds) Skipping install of existing datasets to "dbfs:/mnt/dbacademy-da...

  • 1354 Views
  • 2 replies
  • 1 kudos
Latest Reply
nferran
New Contributor II
  • 1 kudos

Could you fixed it? I have the same problem

  • 1 kudos
1 More Replies
Mist3
by New Contributor II
  • 3584 Views
  • 9 replies
  • 4 kudos

Dashboard API - Create a dashboard object doesn't work

I am trying to copy a dashboard object from one workspace to another using API. I am using Get dashboard objects (/api/2.0/preview/sql/dashboards GET method), then Retrieve a definition (/api/2.0/preview/sql/dashboards/{dashboard_id} GET method) and ...

  • 3584 Views
  • 9 replies
  • 4 kudos
Latest Reply
eason_gao_db
New Contributor II
  • 4 kudos

Hi @markusk, unfortunately the issue you're running into is a limitation of the outgoing SQL dashboard APIs. You can create blank dashboards, but you cannot programmatically insert existing definitions. The good news is we're currently running a prev...

  • 4 kudos
8 More Replies
SJR
by New Contributor III
  • 2931 Views
  • 4 replies
  • 2 kudos

Resolved! Problem when updating Databricks Repo through DevOps Pipeline

Hello all!I've been working on integrating a Databricks Repos update API call to a DevOps Pipeline so that the Databricks local repo stays up to date with the remote staging branch (Pipeline executes whenever there's a new commit in to the staging br...

Data Engineering
CICD
Data_Engineering
DevOps
pipelines
repo
  • 2931 Views
  • 4 replies
  • 2 kudos
Latest Reply
SJR
New Contributor III
  • 2 kudos

@BookerE1 I found it!. There was already another thread related to this problem and someone else helped me find the solution (Problem was the pool that I was using for the pipeline)This is the link to the other thread: https://community.databricks.co...

  • 2 kudos
3 More Replies
costi9992
by New Contributor III
  • 1004 Views
  • 2 replies
  • 0 kudos

Pipeline API documentation issue

 In List Pipelines API documentation, at response is specified that response is  statuses Array of objects The list of events matching the request criteria. next_page_token string If present, a token to fetch the next page of events. But if we retri...

  • 1004 Views
  • 2 replies
  • 0 kudos
Latest Reply
arpit
Valued Contributor
  • 0 kudos

@costi9992 This has been documented now: https://docs.databricks.com/api/workspace/jobs/listruns

  • 0 kudos
1 More Replies
Hal
by New Contributor II
  • 873 Views
  • 1 replies
  • 3 kudos

Connecting Power BI on Azure to Databricks on AWS?

Can someone share with me the proper way to connect Power BI running on Azure to Databricks running on AWS?

  • 873 Views
  • 1 replies
  • 3 kudos
Latest Reply
bhanadi
New Contributor II
  • 3 kudos

Have the same question. Do we have to take care of any specific tasks to make it work. Anyone who implemented it?

  • 3 kudos
costi9992
by New Contributor III
  • 1629 Views
  • 2 replies
  • 0 kudos

Access Databricks API using IDP token

Hello,We have a databricks account & workspace, provided by AWS with SSO enabled. Is there any way to access databricks workspace API ( jobs/clusters, etc ) using a token retrieved from IdentityProvider ? We can access databricks workspace API with A...

  • 1629 Views
  • 2 replies
  • 0 kudos
Latest Reply
fpopa
New Contributor II
  • 0 kudos

Hey - Costin and Anonymous user, have you managed to get this working, do you have examples by any chance?I'm also trying something similar but I haven't been able to make it work.> authenticate and access the Databricks REST API by setting the Autho...

  • 0 kudos
1 More Replies
srjchoubey2
by New Contributor
  • 2277 Views
  • 1 replies
  • 0 kudos

How to import excel files xls/xlsx file into Databricks python notebook?

Method 1: Using "com.crealytics.spark.excel" package, how do I import the package?Method 2: Using pandas I tried the possible paths, but file not found it shows, nor while uploading the xls/xlsx file it shows options for importing the dataframe.Help ...

Data Engineering
excel
import
pyspark
python
  • 2277 Views
  • 1 replies
  • 0 kudos
Latest Reply
vishwanath_1
New Contributor III
  • 0 kudos

import pandas as pd ExcelData = pd.read_excel("/dbfs"+FilePath, sheetName) #  make sure you add /dbfs to FilePath 

  • 0 kudos
AurelioGesino
by New Contributor II
  • 1185 Views
  • 2 replies
  • 0 kudos

Bug in unity catalog registering external database that is case sensitive

I successfully registered in my Unity Catalog an external Database ```dwcore``` that is hosted on SQL server.I first added the connection in "External Data": tested the connection and it was successful.I then added the database on top: tested the con...

  • 1185 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @AurelioGesino, It seems you’ve encountered an issue with table names when connecting to an external SQL Server database in Databricks. Let’s break down the situation and explore potential solutions: Table Name Case Sensitivity: You’ve correc...

  • 0 kudos
1 More Replies
MoJaMa
by Valued Contributor II
  • 5819 Views
  • 8 replies
  • 2 kudos
  • 5819 Views
  • 8 replies
  • 2 kudos
Latest Reply
User15848365773
New Contributor II
  • 2 kudos

Hi @amitca71 @atanu .. yes you can associate as many vpcs(workspace deployment fundamental) across regions and aws accounts to one single databricks aws account infact its one of the super powers of databricks platform and you can even track all thei...

  • 2 kudos
7 More Replies
Milliman
by New Contributor
  • 753 Views
  • 1 replies
  • 0 kudos

How could we automatically re run the complete job if any of its associted task fails.?

I need to re run the compete job automatically if any of its associated task gets failed, any help would be appreciable. Thanks

  • 753 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Milliman, In Databricks, you can automate the re-run of a job if any of its associated tasks fail. Here are some steps to achieve this: Conditional Task Execution: You can specify “Run if dependencies” to run a task based on the run status o...

  • 0 kudos
creditorwatch
by New Contributor
  • 533 Views
  • 1 replies
  • 0 kudos

Load data from Aurora to Databricks directly

Hi,Does anyone know how to link Aurora to Databricks directly and load data into Databricks automatically on a schedule without any third-party tools in the middle?

  • 533 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @creditorwatch, To ingest data into Databricks directly from Amazon Aurora and automate the process on a schedule, you have a few options.    Let’s explore them:   Auto Loader (Recommended): Auto Loader is a powerful feature in Databricks that eff...

  • 0 kudos
Bas1
by New Contributor III
  • 8408 Views
  • 17 replies
  • 20 kudos

Resolved! network security for DBFS storage account

In Azure Databricks the DBFS storage account is open to all networks. Changing that to use a private endpoint or minimizing access to selected networks is not allowed.Is there any way to add network security to this storage account? Alternatively, is...

  • 8408 Views
  • 17 replies
  • 20 kudos
Latest Reply
Odee79
New Contributor II
  • 20 kudos

How can we secure the storage account in the managed resource group which holds the DBFS with restricted network access, since access from all networks is blocked by our Azure storage account policy?

  • 20 kudos
16 More Replies
alm
by New Contributor III
  • 4763 Views
  • 6 replies
  • 1 kudos

Resolved! How to grant access to views without granting access to underlying tables

I have a medallion architecture: Bronze layer: Raw data in tablesSilver layer: Refined data in views created from the bronze layerGold layer: Data products as views created from the silver layerCurrently I have a data scientist that needs access to d...

  • 4763 Views
  • 6 replies
  • 1 kudos
Latest Reply
MoJaMa
Valued Contributor II
  • 1 kudos

Single-user clusters use a different security mode which is the reason for this difference. On single-user/assigned clusters, you'll need the Fine Grained Access Control service (which is a Serverless service) - that is the solution to this problem (...

  • 1 kudos
5 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels