Join the Databricks Learning Festival (Virtual)! Mark your calendars from 10 October - 31 October 2024! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have elevated their career w...
- 2625 Views
- 0 replies
- 5 kudos
Tuesday
Shifting to customized GenAI that deeply understands your data AMER October 8 / 10 AM PTEMEA October 9 / 9 AM BST / 10 AM CESTAPJ October 10 / 12 PM SGT Click here to check out the agenda and speakers and register now! Why are 9 out of 10 organiz...
- 7 Views
- 0 replies
- 0 kudos
19m ago
Get practical guidance, notebooks, code snippets You know this better than anyone: the best GenAI models in the world will not succeed without good data. That’s why data engineers are even more critical today. The challenge is staying ahead of the ra...
- 7 Views
- 0 replies
- 0 kudos
14m ago
Community Activity
- 7 Views
- 0 replies
- 0 kudos
Big Book of Data Engineering — 3rd Edition
Get practical guidance, notebooks, code snippets You know this better than anyone: the best GenAI models in the world will not succeed without good data. That’s why data engineers are even more critical today. The challenge is staying ahead of the ra...
- 7 Views
- 0 replies
- 0 kudos
- 6 Views
- 0 replies
- 0 kudos
How to fully enable row-level concurrency on Databricks 14.1
Hey guys,I hope whoever's reading this is doing well.We're trying to enable row-level concurrency on Databricks 14.1. However, the documentation seems a bit contradictory as to whether this is possible, and whether the whole capability of row-level c...
- 6 Views
- 0 replies
- 0 kudos
- 80 Views
- 1 replies
- 0 kudos
" ResourceNotFound" error is coming on connecting devops repo to databricks workflow(job).
I have a .py file in a repo in azure devops,I want to add it in a workflow in databricks and these are the values I have provided. And the source is this:I have provided all the values correctly but getting this error: " ResourceNotFound". Can someon...
- 80 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Can you try cloning the DevOps repo as a Git folder? The git folder clone interface should ask you to set up a Git credential if it's not already there.
- 0 kudos
- 384 Views
- 2 replies
- 0 kudos
Resolved! Standardized Framework to update Databricks job definition using CI/CD
Hi Databricks support, I am looking for a standardized Databricks framework to update job definition using DevOps from non-production till it get productionized. Our current process of updating the Databricks job definition is as follows:In our sourc...
- 384 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi from the Git folders/Repos PM: DAB is the way to go, and we are working on an integration to author DABs directly in the workspace. Here's a DAIS talk where the DAB PM and I demo'ed some recommendations for source controlling jobs: https://www.da...
- 0 kudos
- 59 Views
- 1 replies
- 0 kudos
Lost untracked and gitignore files when push new commit
Hi guys, our team uses Databricks clusters to develop with Python using Jupyter Notebooks.Recently we had a serious problem in our repos folder ( mainly Jupyter Notebook and Python scripts ).After my teammate committed in the branch, all the tracked ...
- 59 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi from the Git folder/Repos PM: > all the tracked files were restored to the version which I believe is from the last commit ( which was a long time ago ), untracked files and .gitignore files disappeared, This sounds unexpected. If you have a supp...
- 0 kudos
- 1447 Views
- 3 replies
- 1 kudos
Resolved! Hard reset programatically
Is it possible to trigger a git reset --hard programatically?I'm running a platform service where, as part of CI/CD, repos get deployed into the Databricks workspace. Normally, our developers work with upstream repos both from their local IDEs and fr...
- 1447 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi from the Git folders/Repos PM: We don't currently have a solution to hard reset programmatically. This is on our roadmap. If you orchestrate production jobs using the Databricks workflows, there is an option to use version-controlled source code i...
- 1 kudos
- 5156 Views
- 3 replies
- 3 kudos
Resolved! Import from repo
Hi all,I am trying the new "git folder" feature, with a repo that works fine from the "Repos". In the new folder location, my imports from my own repo don't work anymore. Anyone faced something similar?Thanks in advance for sharing your experience
- 5156 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi from the Git folder product manager - If you use DBR 14.3+, the Git folder root is automatically added to the Python `sys.path`. This is documented here. Unfortunately we could not backport this behavior to earlier DBR versions. Hope this helps!...
- 3 kudos
- 2035 Views
- 4 replies
- 0 kudos
DBUtils commands do not work on shared access mode clusters
Hi there,I am trying to upload a file to an s3 bucket. However, none of dbutils commands seem to work neither does the boto3 library. For clusters that have the configuration, except for the shared access mode, seem to work fine.Those are the error m...
- 2035 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
I am encountering very similar behavior to drii_cavalcanti. When I use a Shared cluster with an IAM Role specified I can verify that the aws cli is installed but when I run aws sts get-caller-identity I receive the error "Unable to locate credential...
- 0 kudos
- 25 Views
- 0 replies
- 0 kudos
luv scrub amazon
Exfoliating Washcloth Face & Body Scrub Towel — Japanese Exfoliating Towel with Hexagon Fibers, Exfoliating Body Scrubber with 2 Sides for Scrubbing & Washing — 1 Extra Long TowelReveals Softer & Smoother Skin — Get silky soft skin that glows with ou...
- 25 Views
- 0 replies
- 0 kudos
- 34 Views
- 0 replies
- 0 kudos
Resource organization in a large company
Hello,We are using Azure Databricks in a single tenant. We will have many teams working in multiple (Unity Enabled) Workspaces using a variety of Catalogs, External Locations, Storage Credentials, ect. Some of those resources will be shared (e.g., an...
- 34 Views
- 0 replies
- 0 kudos
- 202 Views
- 3 replies
- 0 kudos
How to stop a continuous pipeline which is set to RETRY on FAILURE and failing for some reason
I have created a pipeline which is continuous and set to RETRY on FAILURE. For some reason it keeps failing and retrying. Is there any way I can stop it. Hitting Stop button throws an error.
- 202 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @szymon_dybczak I already tried to remove it via REST API but got same error as in the pipeline logs. Eventually, I had to remove workspace to get rid of it.
- 0 kudos
- 46 Views
- 4 replies
- 1 kudos
How to get catalog and schema from system query table
Hi,We are querying the system.query table to parse query history. If the table in the query is not fully qualified with its catalog and schema, how can we derive the catalog and schema?Thanks,Jen
- 46 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
There is no straightforward method to get this data. Run the query to check the defaults:SELECT current_catalog() AS default_catalog, current_schema() AS default_schema;Catalog and schema may be changed in the query, so if you have query text you...
- 1 kudos
- 28 Views
- 0 replies
- 0 kudos
Bring data from databricks to sharepoint list using the Power Automate
Good afternoon to all and I am new to this community.We are trying to bring data from databricks to sharepoint list using the Power Automate app (create workflow and trigger it when there is new record or exising record is modified in source table in...
- 28 Views
- 0 replies
- 0 kudos
- 2129 Views
- 4 replies
- 2 kudos
Editor lags
Does anyone else experiencing lags on the Databricks notebook editor?The typing is sometimes very slow causes the cell to hang for a moment. (Using python)
- 2129 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
The notebook editor has been sluggish for me regardless of splitting up between cells and the language in use. Very frustrating.
- 2 kudos
- 26 Views
- 0 replies
- 0 kudos
Issues with Delta Sharing API when using Service Principal Token
Hello,I am currently working with the Delta Sharing API and have encountered an issue when using a Service Principal token for authentication. The API call returns the following error:[CANNOT_INFER_EMPTY_SCHEMA] Can not infer schema from empty datase...
- 26 Views
- 0 replies
- 0 kudos
Latest from our Blog
From Raw to Refined: Processing Overture Maps Geospatial Data on Databricks - Part 1
This is the first part of a two-part series blog on geospatial data processing on Databricks. The first part will cover ingesting and processing Overture Maps data on Databricks, while the second part...
MLOps Gym - Version control best practices
In the first part of this article, we explored the critical role that version control plays in data science, particularly when using Git within the Databricks environment. We covered the foundational ...
6 Migration Mistakes You Don’t Want To Make: Part 2
Welcome to Part 2 of this series. If you haven’t read Part 1, we recommend you start there. To recap so far: Between us, we have worked on over 30 customer migrations leading us to develop this list o...
GenAI is for Data Engineers Too
Large Language Models (LLMs) or GenAI have taken center stage in their ability to summarize large text, write code, articles, poems, explain complex topics, and power now ubiquitous chatbots. Data Eng...
