- 971 Views
- 2 replies
- 0 kudos
feature store
i need to build for data science team feature store that will return one big df after one hot encoding for almost each dimension,join and group by. should I create one feature store for final output that contain all the relevant data or create featur...
- 971 Views
- 2 replies
- 0 kudos
- 0 kudos
Here are some things to consider: The best practice for designing a feature store in your scenario depends on balancing scalability, maintainability, and the dynamic nature of some dimensions like doctor names. Here's an outlined recommendation bas...
- 0 kudos
- 1357 Views
- 2 replies
- 0 kudos
Databricks DLT ADLS Access issue
We have a DLT pipeline configure with spn inside the notebook, which was working fine. Now after credentials expiry, we created new one and updated the same in notebook. Now we are pipeline is not able to read from ADLS.SPN and my UserId is having co...
- 1357 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @VigneshJaisanka The issue likely comes from a permissions or configuration mismatch. Here are a few things worth checking:Make sure the SPN is set as the pipeline owner and has the necessary permissions on the ADLS resource.If you’re using Unity ...
- 0 kudos
- 402 Views
- 1 replies
- 0 kudos
Delta Sharing & UC: Understanding the Initial Empty Predicate Query
We're testing our Delta Sharing server with Unity Catalog (UC) and noticed a behavior where a simple query like SELECT COUNT(1) FROM table_name WHERE col1 = 'value' triggers two /query requests to our server.The initial request arrives with empty pre...
- 402 Views
- 1 replies
- 0 kudos
- 0 kudos
The initial /query request during a Delta Sharing operation with Unity Catalog serves a critical purpose in the query lifecycle. It is intended to retrieve the schema and basic metadata of the table, which helps in query planning and optimization. Th...
- 0 kudos
- 1135 Views
- 2 replies
- 0 kudos
Migration of PowerBI reports from Synapse to Databricks sql (DBSQL)
We have 250 powerbi reports build on top of Azure Synapse, now we are migrating from Azure Synapse to Databricks (DB SQL). How to plan for cutover and strategy for PowerBII just seeking high level points we have to take care for planning. Any techie ...
- 1135 Views
- 2 replies
- 0 kudos
- 0 kudos
While your account Solution Architect (SA) will be able to guide you, if you still want to check what peers did here https://community.databricks.com/t5/warehousing-analytics/migrate-azure-synapse-analytics-data-to-databricks/td-p/90663 and here http...
- 0 kudos
- 9130 Views
- 15 replies
- 5 kudos
Unable to Login - Account Verification Loop
I'm having trouble logging in to my Databricks account at databricks.com. Here's what happens:I enter my email address and password.I receive an account verification code via email.I enter the verification code on the login page.Instead of logging me...
- 9130 Views
- 15 replies
- 5 kudos
- 5 kudos
Same problem. Has anyone found a solution yet?
- 5 kudos
- 2114 Views
- 3 replies
- 1 kudos
Resolved! Delta Live Table Pipeline
I have the error message when try to create a delta live table pipeline.My error is: com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 1207-112912-8e84v9h5: Encountered Quota Exhaustion issue in ...
- 2114 Views
- 3 replies
- 1 kudos
- 1558 Views
- 1 replies
- 0 kudos
Databricks User Group Meetups
Are there any Databricks User Group Meetups in Charlotte?
- 1558 Views
- 1 replies
- 0 kudos
- 0 kudos
I am interested to participate in this group if it is available
- 0 kudos
- 8255 Views
- 4 replies
- 9 kudos
Choosing between Azure Data Factory (ADF) and Databricks PySpark notebooks
I’m working on a project where I need to pull large datasets from Cosmos DB into Databricks for further processing, and I’m trying to decide whether to use Azure Data Factory (ADF) or Databricks PySpark notebooks for the extraction and processing tas...
- 8255 Views
- 4 replies
- 9 kudos
- 9 kudos
Hi @DebIT2011,You're facing a classic architectural decision between orchestration with ADF versus direct transformation using Databricks PySpark notebooks. Both tools are powerful but serve different purposes depending on your project needs. Below i...
- 9 kudos
- 581 Views
- 1 replies
- 0 kudos
How can I leverage Databricks for building end-to-end machine learning pipelines?
I’ve been following practical tutorials on makerandcoder, which often showcase hands-on machine learning projects using Python, scikit-learn, and Spark. I’m looking to scale my projects using the Databricks platform for better collaboration, data han...
- 581 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks enables the creation of scalable, end-to-end machine learning (ML) pipelines by providing a comprehensive and collaborative platform that integrates key components for data handling, experimentation, and model deployment. Here’s how Databr...
- 0 kudos
- 2170 Views
- 1 replies
- 0 kudos
Optimal Strategies for downloading large query results with Databricks API
Hi everyone,I'm currently facing an issue with handling a large amount of data using the Databricks API. Specifically, I have a query that returns a significant volume of data, sometimes resulting in over 200 chunks.My initial approach was to retriev...
- 2170 Views
- 1 replies
- 0 kudos
- 0 kudos
I am also facing the same issue now one approach tomorrow i will try I will create a job that using serverless job cluster. Then whenever user will click on download button from UI. This should trigger the job now this job. Will read the table as dat...
- 0 kudos
- 720 Views
- 3 replies
- 0 kudos
S3 limited bucket permissions
Hi,can I run Databricks on limited/restricted S3 bucket folder, no access to bucket root level as it is restricted per project folder in IAM?i.e s3://mybucket/myproject_abc/Now I configured all permissions as per documentationhttps://docs.databricks....
- 720 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks, but no thanks, spam resides in JUNK folder
- 0 kudos
- 2732 Views
- 8 replies
- 0 kudos
Resolved! Programatic selection of serverless compute for notebooks environment version
Hello,I have a case where I am executing notebooks from an external system using databricks api /api/2.2/jobs/runs/submit. This has always been non problematic with the job compute, but due to the quite recent serverless for notebooks support being i...
- 2732 Views
- 8 replies
- 0 kudos
- 0 kudos
@tts, did you manage to get this resolved? Battling the same issue here and cannot find a way to make it work. Even though the default API is 2 now, it still seems to randomly assign an API when deploy via DAB. @JakubSkibicki, tried this too but hitt...
- 0 kudos
- 11665 Views
- 3 replies
- 0 kudos
Resolved! Unable to add column comment on a View. Any way to update comments on multiple columns in bulk?
I noticed that unlike "Alter Table" there is no "Alter View" command to add comment on a column in the existing view. This is a regular view created on Tables (and not Materialized view). If the underlying table column has comment then the View inh...
- 11665 Views
- 3 replies
- 0 kudos
- 0 kudos
Use COMMENT ONCOMMENT ON | Databricks on AWS
- 0 kudos
- 16119 Views
- 6 replies
- 1 kudos
Insufficient Permissions Issue on Databricks
I have encountered a technical issue on Databricks.While executing commands both in Spark and SQL within the Databricks environment, I’ve run into permission-related errors from selecting files from DBFS. "org.apache.spark.SparkSecurityException: [IN...
- 16119 Views
- 6 replies
- 1 kudos
- 1 kudos
Please refer to some of the other community articles with the no module error https://community.databricks.com/t5/data-engineering/udf-importing-from-other-modules/td-p/58988
- 1 kudos
- 4393 Views
- 4 replies
- 1 kudos
DLT Pipeline unable to find custom Libraries/Wheel packages
We have our DLT pipeline and we need to import our custom libraries packaged in wheel files.We are on Azure DBX and we are using Az DevOps CI/CD to build and deploy the wheel packages on our DBX environment. In the top of our DLT notebook we are impo...
- 4393 Views
- 4 replies
- 1 kudos
- 1 kudos
You might want to verify the file path and permissions within your CI/CD process—sometimes the context in which the pipeline runs lacks proper DBFS mount visibility. We've encountered similar visibility inconsistencies while working on data aggregati...
- 1 kudos
Join Us as a Local Community Builder!
Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!
Sign Up Now-
.CSV
1 -
Access Data
2 -
Access Databricks
1 -
Access Delta Tables
2 -
Account reset
1 -
ADF Pipeline
1 -
ADLS Gen2 With ABFSS
1 -
Advanced Data Engineering
1 -
AI
1 -
Analytics
1 -
Apache spark
1 -
Apache Spark 3.0
1 -
API Documentation
3 -
Architecture
1 -
asset bundle
1 -
Asset Bundles
2 -
Auto-loader
1 -
Autoloader
4 -
AWS
3 -
AWS security token
1 -
AWSDatabricksCluster
1 -
Azure
5 -
Azure data disk
1 -
Azure databricks
14 -
Azure Databricks SQL
6 -
Azure databricks workspace
1 -
Azure Delta Lake
1 -
Azure Unity Catalog
5 -
Azure-databricks
1 -
AzureDatabricks
1 -
AzureDevopsRepo
1 -
Big Data Solutions
1 -
Billing
1 -
Billing and Cost Management
1 -
Blackduck
1 -
Bronze Layer
1 -
Certification
3 -
Certification Exam
1 -
Certification Voucher
3 -
CICDForDatabricksWorkflows
1 -
Cloud_files_state
1 -
CloudFiles
1 -
Cluster
3 -
Community Edition
3 -
Community Event
1 -
Community Group
2 -
Community Members
1 -
Compute
3 -
Compute Instances
1 -
conditional tasks
1 -
Connection
1 -
Contest
1 -
Credentials
1 -
CustomLibrary
1 -
Data
1 -
Data + AI Summit
1 -
Data Engineering
3 -
Data Explorer
1 -
Data Ingestion & connectivity
1 -
Databrick add-on for Splunk
1 -
databricks
2 -
Databricks Academy
1 -
Databricks AI + Data Summit
1 -
Databricks Alerts
1 -
Databricks Assistant
1 -
Databricks Certification
1 -
Databricks Cluster
2 -
Databricks Clusters
1 -
Databricks Community
10 -
Databricks community edition
3 -
Databricks Community Edition Account
1 -
Databricks Community Rewards Store
3 -
Databricks connect
1 -
Databricks Dashboard
2 -
Databricks delta
2 -
Databricks Delta Table
2 -
Databricks Demo Center
1 -
Databricks Documentation
2 -
Databricks genAI associate
1 -
Databricks JDBC Driver
1 -
Databricks Job
1 -
Databricks Lakehouse Platform
6 -
Databricks Migration
1 -
Databricks notebook
2 -
Databricks Notebooks
3 -
Databricks Platform
2 -
Databricks Pyspark
1 -
Databricks Python Notebook
1 -
Databricks Repo
1 -
Databricks Runtime
1 -
Databricks SQL
5 -
Databricks SQL Alerts
1 -
Databricks SQL Warehouse
1 -
Databricks Terraform
1 -
Databricks UI
1 -
Databricks Unity Catalog
4 -
Databricks Workflow
2 -
Databricks Workflows
2 -
Databricks workspace
3 -
Databricks-connect
1 -
DatabricksJobCluster
1 -
DataDays
1 -
Datagrip
1 -
DataMasking
2 -
DataVersioning
1 -
dbdemos
2 -
DBFS
1 -
DBRuntime
1 -
DBSQL
1 -
DDL
1 -
Dear Community
1 -
deduplication
1 -
Delt Lake
1 -
Delta
22 -
Delta Live Pipeline
3 -
Delta Live Table
5 -
Delta Live Table Pipeline
5 -
Delta Live Table Pipelines
4 -
Delta Live Tables
7 -
Delta Sharing
2 -
deltaSharing
1 -
Deny assignment
1 -
Development
1 -
Devops
1 -
DLT
10 -
DLT Pipeline
7 -
DLT Pipelines
5 -
Dolly
1 -
Download files
1 -
Dynamic Variables
1 -
Engineering With Databricks
1 -
env
1 -
ETL Pipelines
1 -
External Sources
1 -
External Storage
2 -
FAQ for Databricks Learning Festival
2 -
Feature Store
2 -
Filenotfoundexception
1 -
Free trial
1 -
GCP Databricks
1 -
GenAI
1 -
Getting started
2 -
Google Bigquery
1 -
HIPAA
1 -
import
1 -
Integration
1 -
JDBC Connections
1 -
JDBC Connector
1 -
Job Task
1 -
Lineage
1 -
LLM
1 -
Login
1 -
Login Account
1 -
Machine Learning
2 -
MachineLearning
1 -
Materialized Tables
2 -
Medallion Architecture
1 -
Migration
1 -
ML Model
2 -
MlFlow
2 -
Model Training
1 -
Module
1 -
Networking
1 -
Notebook
1 -
Onboarding Trainings
1 -
Pandas udf
1 -
Permissions
1 -
personalcompute
1 -
Pipeline
2 -
Plotly
1 -
PostgresSQL
1 -
Pricing
1 -
Pyspark
1 -
Python
5 -
Python Code
1 -
Python Wheel
1 -
Quickstart
1 -
Read data
1 -
Repos Support
1 -
Reset
1 -
Rewards Store
2 -
Schedule
1 -
Serverless
3 -
Session
1 -
Sign Up Issues
2 -
Spark
3 -
Spark Connect
1 -
sparkui
2 -
Splunk
2 -
SQL
8 -
Summit23
7 -
Support Tickets
1 -
Sydney
2 -
Table Download
1 -
Tags
1 -
Training
2 -
Troubleshooting
1 -
Unity Catalog
4 -
Unity Catalog Metastore
2 -
Update
1 -
user groups
1 -
Venicold
3 -
Voucher Not Recieved
1 -
Watermark
1 -
Weekly Documentation Update
1 -
Weekly Release Notes
2 -
Women
1 -
Workflow
2 -
Workspace
3
- « Previous
- Next »
User | Count |
---|---|
133 | |
97 | |
52 | |
42 | |
30 |