- 3781 Views
- 7 replies
- 3 kudos
Resolved! Azure Databricks Unity Catalog - cannot access managed volume in notebook
We have set up Azure Databricks with Unity Catalog (Metastore).Used Managed Identity (Databricks Access Connector) for connection from workspace(s) to ADLS Gen2ADLS Gen2 storage account has Storage Blob Data Contributor and Storage Queue Data Contrib...
- 3781 Views
- 7 replies
- 3 kudos
- 3 kudos
@AlbertWang @VAMSaha22 Since you want private connectivity I assume you have a vnet and a PE associated with the gen2 account. That PE needs to have a sub-resource of type dfs when the storage account is gen2/hierarchical namespace. You might want to...
- 3 kudos
- 9 Views
- 0 replies
- 0 kudos
Sharing Databricks Notebook Functionality Without Revealing Source Code
Hi All,I have a unique scenario in Databricks and would appreciate your insights.I’ve developed functionality in Databricks notebooks, and I’d like to share this with other developers within the same workspace. My goal is to allow colleagues to impor...
- 9 Views
- 0 replies
- 0 kudos
- 15 Views
- 0 replies
- 0 kudos
Error filtering by datetime Lakehouse Federated SQL Server table
In unity catalog, I have a connection to a SQL Server database. When I try to filter by a datetime column using a datetime with fractional seconds, Databricks gives me this error:Job aborted due to stage failure: com.microsoft.sqlserver.jdbc.SQLServe...
- 15 Views
- 0 replies
- 0 kudos
- 1752 Views
- 1 replies
- 0 kudos
Parameter "expand_tasks" on List job runs request seams not to be working (databricsk api)
I'm setting it as True, but it doesn't return the cluster_instance info. Here is the function I'm using:def get_job_runs(job_id): """ Fetches job runs for a specific job from Databricks Jobs API. """ headers = { "Authorization...
- 1752 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mildred The way you passed the data for the expand_tasks parameter is wrong: data = { data = { "job_id": job_id, "expand_tasks": "true" } It should not be passed as Python boolean values, but as a string "true" or "false" Once you do that will...
- 0 kudos
- 1797 Views
- 3 replies
- 0 kudos
Executing Bash Scripts or Binaries Directly in Databricks Jobs on Single Node Cluster
Hi,Is it possible to directly execute a Bash script or a binary executable from the operating system of a Databricks job compute node using a single node cluster?I’m using databricks asset bundels for job initialization and execution. When the job s...
- 1797 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @jorperort , I did some research internally and have some tips/suggestions for you to consider: Based on the research and available documentation, it is not possible to directly execute a Bash script or binary executable from the operating sy...
- 0 kudos
- 1527 Views
- 1 replies
- 0 kudos
How to Read and Wrire Data between 2 seperate instance of Databricks
How to Read and Wrire Data between 2 seperate instance of Databricks.I want to have bi-directional data read and write between Databricks A and Databricks B. Both are not in same instance.Please help
- 1527 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Pratikmsbsvm , I want to better understand what you mean by "Instance"? Do you mean two seperate workspace within the same ADB account or do you mean two different ADB accounts? Please clarify so I can provide guidance. Regards, Louis.
- 0 kudos
- 71 Views
- 3 replies
- 0 kudos
ExcelWriter and local files
I have a couple things going on here.First, to explain what I'm doing, I'm passing an array of objects in to a function that contain a dataframe per item. I want to write those dataframes to an excel workbook - one dataframe per worksheet. That part ...
- 71 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey @data-grassroots , I did some digging with our internal docs and have some suggestions/tips to help you further diagnose the issue: You're following the recommended Databricks approach for editing Excel files: copying the template to a local pa...
- 0 kudos
- 3277 Views
- 4 replies
- 0 kudos
Connect Oracle Fusion (ERP / HCM) to Databricks
Any suggestion to connect Oracle Fusion (ERP/HCM) to Databricks?I have explored a few options including the use of Oracle Integration Cloud but it requires a lot of customization.
- 3277 Views
- 4 replies
- 0 kudos
- 0 kudos
I used Fivetran Oracle Fusion Connector in past. It is a fully managed ELT connector that extracts data from Oracle Fusion and loads it into Databricks.
- 0 kudos
- 47 Views
- 2 replies
- 0 kudos
Resolved! Getting NO_TABLES_IN_PIPELINE error in Lakeflow Declarative Pipelines
Yesterday (10/1) starting around 12 PM EST we starting getting the following error in our Lakeflow Declarative Pipelines (LDP) process. We get this in environments where none of our code has changed. I found some info on the serverless compute abou...
- 47 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @cpollock Check the “Event log” and “Pipeline logs” in the Databricks UI for any clues.also can you please share screenshot as pasted in window, attachment is not really working and only scanning
- 0 kudos
- 42 Views
- 1 replies
- 0 kudos
DLT Flow Redeclaration Error After Service Upgrade
Hi, our delta live tables(Lakeflow declarative pipelines) pipeline started failing after the Sep 30 / Oct 1 service upgrade with the following error :AnalysisException: Cannot have multiple queries named `<table_name>_realtime_flow` for `<table_name>...
- 42 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @DiskoSuperStar IT seems you’ve run into a recently enforced change in Databricks DLT/Lakeflow:Multiple flows (append or otherwise) targeting the same table must have unique names. actually it looks correct on your code. Check if your table_info ...
- 0 kudos
- 1022 Views
- 4 replies
- 1 kudos
How to select performance mode for Databricks Delta Live Tables
Hi everyone,Based on the official link,For triggered pipelines, you can select the serverless compute performance mode using the Performance optimized setting in the pipeline scheduler. When this setting is disabled, the pipeline uses standard perfor...
- 1022 Views
- 4 replies
- 1 kudos
- 1 kudos
I would like an answer to his question also, I need to see how to turn this off, but any check box relating to performance optimization in my serverless pipeline does not show up.
- 1 kudos
- 92 Views
- 3 replies
- 4 kudos
Need Help to learn scala
Hi Enthusiasts, recently i joined company in that company they used to develope databricks notebook with Scala programming language perviously, i worked on Pyspark it was very easy for me by the way i have 3 years of experence in DE i need help to wh...
- 92 Views
- 3 replies
- 4 kudos
- 4 kudos
fwiw: you do not have to be a scala wiz to work on spark in scala.Older spark articles are often about scala in spark (before python took over).You will notice it is a lot like pyspark, but way way better. typing, immutability, things like leftfold ...
- 4 kudos
- 93 Views
- 5 replies
- 1 kudos
Why ETL Pipelines and Jobs
I do notice that ETL Pipelines let's you run declarative SQL syntax such as DLT tables but you can do the same with Jobs if you use SQL as your task type. So why and when to use ETL Pipelines?
- 93 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @john77 SQL Task Type : Simple, one-off SQL operations or batch jobs + you need to orchestrate a mix of notebooks, Python/Scala code, and SQL in a single workflowLakeflow Declarative Pipelines : Complex , production ETL jobs requires lineage , mon...
- 1 kudos
- 26 Views
- 1 replies
- 0 kudos
Databricks to Dataverse migration via ADF copy data
Hi team,I need to load data from databricks delta tables to dataverse tables and I have one unique id column which I am ensuring via mapping. Its datatype is GUID in dataverse and string in delta table. I ensured that column holds unique values. Sinc...
- 26 Views
- 1 replies
- 0 kudos
- 0 kudos
That is not a valid guid.Dataverse will check this.http://guid.us/test/guid
- 0 kudos
- 40 Views
- 1 replies
- 2 kudos
How Databricks Helped Me See Data Engineering Differently
Over the years working as a data engineer, I’ve started to see my role very differently. In the beginning, most of my focus was on building pipelines—extracting, transforming, and loading data so it could land in the right place. Pipelines were the g...
- 40 Views
- 1 replies
- 2 kudos
- 2 kudos
@Brahmareddy thanks for this! .Think you've nailed it on the head there. If the stakeholders trust the data and there's integrity, governance, and a single source of truth, you've got a recipe for a great product! Love this take @Brahmareddy . Really...
- 2 kudos
| User | Count |
|---|---|
| 1616 | |
| 788 | |
| 404 | |
| 349 | |
| 287 |
