- 10544 Views
- 4 replies
- 9 kudos
Choosing between Azure Data Factory (ADF) and Databricks PySpark notebooks
I’m working on a project where I need to pull large datasets from Cosmos DB into Databricks for further processing, and I’m trying to decide whether to use Azure Data Factory (ADF) or Databricks PySpark notebooks for the extraction and processing tas...
- 10544 Views
- 4 replies
- 9 kudos
- 9 kudos
Hi @DebIT2011,You're facing a classic architectural decision between orchestration with ADF versus direct transformation using Databricks PySpark notebooks. Both tools are powerful but serve different purposes depending on your project needs. Below i...
- 9 kudos
- 904 Views
- 1 replies
- 0 kudos
How can I leverage Databricks for building end-to-end machine learning pipelines?
I’ve been following practical tutorials on makerandcoder, which often showcase hands-on machine learning projects using Python, scikit-learn, and Spark. I’m looking to scale my projects using the Databricks platform for better collaboration, data han...
- 904 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks enables the creation of scalable, end-to-end machine learning (ML) pipelines by providing a comprehensive and collaborative platform that integrates key components for data handling, experimentation, and model deployment. Here’s how Databr...
- 0 kudos
- 2501 Views
- 1 replies
- 0 kudos
Optimal Strategies for downloading large query results with Databricks API
Hi everyone,I'm currently facing an issue with handling a large amount of data using the Databricks API. Specifically, I have a query that returns a significant volume of data, sometimes resulting in over 200 chunks.My initial approach was to retriev...
- 2501 Views
- 1 replies
- 0 kudos
- 0 kudos
I am also facing the same issue now one approach tomorrow i will try I will create a job that using serverless job cluster. Then whenever user will click on download button from UI. This should trigger the job now this job. Will read the table as dat...
- 0 kudos
- 973 Views
- 3 replies
- 0 kudos
S3 limited bucket permissions
Hi,can I run Databricks on limited/restricted S3 bucket folder, no access to bucket root level as it is restricted per project folder in IAM?i.e s3://mybucket/myproject_abc/Now I configured all permissions as per documentationhttps://docs.databricks....
- 973 Views
- 3 replies
- 0 kudos
- 19287 Views
- 6 replies
- 1 kudos
Insufficient Permissions Issue on Databricks
I have encountered a technical issue on Databricks.While executing commands both in Spark and SQL within the Databricks environment, I’ve run into permission-related errors from selecting files from DBFS. "org.apache.spark.SparkSecurityException: [IN...
- 19287 Views
- 6 replies
- 1 kudos
- 1 kudos
Please refer to some of the other community articles with the no module error https://community.databricks.com/t5/data-engineering/udf-importing-from-other-modules/td-p/58988
- 1 kudos
- 4847 Views
- 4 replies
- 1 kudos
DLT Pipeline unable to find custom Libraries/Wheel packages
We have our DLT pipeline and we need to import our custom libraries packaged in wheel files.We are on Azure DBX and we are using Az DevOps CI/CD to build and deploy the wheel packages on our DBX environment. In the top of our DLT notebook we are impo...
- 4847 Views
- 4 replies
- 1 kudos
- 1 kudos
You might want to verify the file path and permissions within your CI/CD process—sometimes the context in which the pipeline runs lacks proper DBFS mount visibility. We've encountered similar visibility inconsistencies while working on data aggregati...
- 1 kudos
- 643 Views
- 1 replies
- 0 kudos
How to Migrate Legacy Dashboards from hive_metastore to Unity Catalog using Python
Hi all,After updating the legacy dashboard APIs, I’m looking to migrate legacy dashboards from the hive_metastore to Unity Catalog in Databricks. Specifically, I want to programmatically:Migrate SQL queries used in dashboardsRetain or recreate the as...
- 643 Views
- 1 replies
- 0 kudos
- 0 kudos
For your consideration: To migrate legacy dashboards from the Hive Metastore to Unity Catalog in Databricks programmatically while retaining SQL queries, data visualizations, and ensuring compatibility with Unity Catalog schemas and tables using Py...
- 0 kudos
- 5945 Views
- 3 replies
- 4 kudos
ArcGIS Connection
Hi,I am trying to connect to an ArcGIS instance using Data bricks. Is this possible? After connecting, I am trying to read the data into a Data fame.Please help me with this request. If its not possible to connect , please provide an alternative.Than...
- 5945 Views
- 3 replies
- 4 kudos
- 4 kudos
I am trying to set up an ODBC or JDBC direct connection from ArcGIS Pro and ArcGIS Enterprise Server; has anyone done this successfully? I was able to make DSN successful connection, but no tables are being delivered; I did not use a connection stri...
- 4 kudos
- 3736 Views
- 11 replies
- 7 kudos
Facing issues while upgrading DBR version from 9.1 LTS to 15.4 LTS
Dear all,I am upgrading DBR version from 9.1 LTS to 15.4 LTS in Azure Databricks. for that I have created a new cluster with 15.4 DBR attached init script for installing application dependencies. Cluster has started successfully but it takes 30 min. ...
- 3736 Views
- 11 replies
- 7 kudos
- 7 kudos
ust to add, you might also want to check the cluster logs (driver and init script logs) for any hidden errors or timeouts during startup. Sometimes dependencies silently fail to install,even if the cluster appears to be running. If possible, try brea...
- 7 kudos
- 2512 Views
- 3 replies
- 2 kudos
Calling all Chennai residents! Join the Chennai User Group on Community!
Calling all Chennai residents! Join the Chennai User Group on Community! Are you passionate about our vibrant city of Chennai? Do you love connecting with like-minded individuals, expanding your knowledge, and contributing to a thriving Community...
- 2512 Views
- 3 replies
- 2 kudos
- 2 kudos
Thank you, will that be happening in future or any what's up group is available now?
- 2 kudos
- 809 Views
- 1 replies
- 0 kudos
Gcs and databricks community
Hello,I would like to know if it is possible to connect my Databricks community account with a Google cloud storage account via a notebook.I tried to connect it via the json key of my gcs service account but the notebook always gives this error when ...
- 809 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @oricaruso! It looks like this post duplicates the one you posted earlier. A response has already been provided in the original thread. I recommend continuing the discussion there to keep the conversation focused and organized.
- 0 kudos
- 828 Views
- 1 replies
- 0 kudos
How to create and manage daily partitioned tables in Databricks Unity Catalog for Bronze layer with
I’m working on a project to practice using Databricks, Lakehouse architecture, Medallion architecture, and data lakes. Setup: I have an Azure Data Lake with three containers: bronze, silver, and gold. In the bronze container, I store daily customer C...
- 828 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @OmarSEKkOUTI , you can leverage DLT feature to do so. Please check: https://docs.databricks.com/aws/en/dlt/transform https://docs.databricks.com/aws/en/dlt/stateful-processing Here is the step-by-step tutorial: https://docs.databricks.com/aws/en...
- 0 kudos
- 1566 Views
- 1 replies
- 1 kudos
Resolved! Cleared DE associate exam ..need to clear GenAI associate in next 5 days.
Hi fellow members,Hope you're all doing well.I’m seeking advice from those who’ve recently cleared the Databricks Generative AI Associate certification. I’ve spent 20 years in data and analytics consulting, and I'm currently in the process of resetti...
- 1566 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @gmglitch, If you have an account at the Databricks employee academy, please take the GEN AI certification course. You can go through the below medium articles: https://medium.com/@chandadipendu/databricks-generative-ai-engineer-associate-certifi...
- 1 kudos
- 5445 Views
- 5 replies
- 3 kudos
Failure when deploying a custom serving endpoint LLLM
I'm currently experimenting with vector search using Databricks. Everything runs smoothly when I load the model deployed in Unity Catalog into a notebook session and ask questions using Python. However, when I attempt to serve it, I encounter a gener...
- 5445 Views
- 5 replies
- 3 kudos
- 3 kudos
Get your Ehsaas Program 8171 online registration and CNIC check online now. Enter your National ID card number & get Rs. 25000/12000 grant.8171 web portal
- 3 kudos
- 3155 Views
- 2 replies
- 0 kudos
Workflow UI not showing all jobs that were run
I found a scenario where the workflow UI seems to not list all the jobs that ran.The objective is to keep trying to run a job, using the job rest api, when the service returns a status of "skipped" while another run is currently running, until the jo...
- 3155 Views
- 2 replies
- 0 kudos
- 0 kudos
I'm.facing the same issue I used to see the jobs ran via databricks bundle but now I can't see those jobs ? Can ay please specify wat went wrong ?
- 0 kudos
