Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
879 Posts
Activity in Databricks Platform Discussions
- 21 Views
- 0 replies
- 0 kudos
Very long vacuum on s3
Since we've moved from azure to aws, a specific job has extremely long vacuum runs, is there a specific flag/configuration for the s3 storage that is needed to support faster vacuum.How can I research what's going on?Note, it's not ALL jobs, but a sp...
- 21 Views
- 0 replies
- 0 kudos
- 40 Views
- 0 replies
- 0 kudos
Issue reading Vertica table into Databricks - Numeric value out of range
I am trying to read a Vertica table into a Spark DataFrame using JDBC in Databricks.Here is my sample code:hostname = ""username = ""password = ""database_port = ""database_name = ""qry_col_level = f"""SELECT * FROM analytics_DS.ansh_units_cum_dash""...
- 40 Views
- 0 replies
- 0 kudos
- 22378 Views
- 13 replies
- 23 kudos
Databricks Announces the Industry’s First Generative AI Engineer Learning Pathway and Certification
Today, we are announcing the industry's first Generative AI Engineer learning pathway and certification to help ensure that data and AI practitioners have the resources to be successful with generative AI. At Databricks, we recognize that generative ...
- 22378 Views
- 13 replies
- 23 kudos
- 871 Views
- 2 replies
- 0 kudos
Implementing Federated Governance in Databricks Unity Catalog
Hi,I am working for a large company that is implementing a Databricks solution. We have multiple domains, each responsible for its own data products, following a data mesh approach.As part of a federated governance model, we need a way to communicate...
- 871 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Tuno986 You can implement a notification-driven system by having each domain team register new data products in a Delta table or send an event. This triggers automated compliance checks (schema, lineage, ACLs) using Databricks workflows. Governan...
- 0 kudos
- 23 Views
- 0 replies
- 0 kudos
learning Data Analysis with Databricks
Hi Team, Im unable to access below mentioed URL page. - https://customer-academy.databricks.com/learn/course/external/view/elearning/2916/data-analysis-with-databricks can you pls help.
- 23 Views
- 0 replies
- 0 kudos
- 184 Views
- 4 replies
- 2 kudos
Is databricks community edtion cluster having issue?
i am trying to run a piece of code and everytime i am getting "Notebook detached×Exception when creating execution context: java.net.SocketTimeoutException: Connect Timeout". Can anyone please tell if there is any issue going on for the databrics com...
- 184 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
I am running into the same issue. I am just trying to initialize a spark session. Please let me know if there is a fix.
- 2 kudos
- 53 Views
- 0 replies
- 0 kudos
Issue with Verification Code Input on Login
Hello, I hope you're well.​I'd like to report a bug encountered when entering the verification code to log into the platform. When I type the code without Caps Lock enabled, the input field displays the characters in uppercase, but the code isn't acc...
- 53 Views
- 0 replies
- 0 kudos
- 1607 Views
- 1 replies
- 0 kudos
DEV and PROD
"SELECT * FROM' data call on my table in PROD is giving all the rows of data (historical data), but a call on my table in DEV is giving me just one row of data (current one row of historical data). what could be the problem??
- 1607 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Please don't cross post. Thanks, Louis.
- 0 kudos
- 72 Views
- 1 replies
- 0 kudos
Is it possible restore a deleted catalog and schema
Is it possible restore a deleted catalog and schema.if CASCADE is used even though schemas and tables are present in catalog, catalog will be dropped.Is it possible to restore catalog or is possible to restrict the use of CACADE command.Thank you.
- 72 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
It is not possible to directly restore a deleted catalog or schema if they were dropped with the CASCADE option, especially in Databricks Unity Catalog. When a catalog or schema is dropped with CASCADE, all its dependent objects, such as schemas and ...
- 0 kudos
- 97 Views
- 1 replies
- 0 kudos
Is it possible expand/extend subnet CIDR of an existing azure databricks workspace
Is it possible expand/extend subnet CIDR of an existing azure databricks workspace. Currently our workspace is maxed out, Is it possible expand/extend subnet CIDR of an existing azure databricks workspace without having create a new one
- 97 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Yes, it is possible to expand or extend the subnet CIDR of an existing Azure Databricks workspace without creating a new one, but this capability is specifically applicable if the workspace is deployed with VNet injection. For workspaces that use V...
- 0 kudos
- 181 Views
- 6 replies
- 1 kudos
Resolved! GET /api/2.2/jobs/list Ordering
Hi there!I am calling the job list API (via the Python SDK):GET /api/2.2/jobs/listdocs.databricks.com/api/workspace/jobs/listDoes anyone know what ordering is applied / calculated for the list of jobs? Is it consistent or random?Is it by creation tim...
- 181 Views
- 6 replies
- 1 kudos
- 169 Views
- 3 replies
- 1 kudos
Azure Databricks Status
Dear all,I wanted to check if anyone implemented the solution of capturing information from Databricks status page in real-time 24x7 and load that into a log or table...https://learn.microsoft.com/en-us/azure/databricks/resources/statuswhat is the be...
- 169 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
It seems that the webhook is the way!There is nothing about system status in Databricks REST API.There is nothing about system status in the System Tables schema.
- 1 kudos
- 461 Views
- 7 replies
- 0 kudos
Google Pub Sub and Delta live table
I am using delta live table and pub sub to ingest message from 30 different topics in parallel. I noticed that initialization time can be very long around 15 minutes. Does someone knows how to reduced initialization time in dlt ? Thanks You
- 461 Views
- 7 replies
- 0 kudos
Latest Reply
- 0 kudos
Classic clusters can take up to seven minutes to be acquired, configured, and deployed, with most of this time spent waiting for the cloud service to allocate virtual machines. In contrast, serverless clusters typically start in under eight seconds. ...
- 0 kudos
- 338 Views
- 1 replies
- 0 kudos
How to do NLP against PDFs in Databricks? Can be done in Snowflake very easily.
I’d love to build a quick "art of the possible" demo showing how easy it is to query unstructured PDFs using natural language. In Snowflake, I wired up a similar solution in ~2 hours just by following their tutorial guide.Does anyone know the best wa...
- 338 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
To query unstructured PDF files using natural language in Databricks, you can leverage an approach similar to the "Retrieval Augmented Generation (RAG) and DBRX" demo. Although the specific demo you referenced (https://notebooks.databricks.com/demos/...
- 0 kudos
- 101 Views
- 1 replies
- 0 kudos
Error when uploading MLFlow artifacts to DBFS
Hi everyone,I'm attempting to use MLFlow experiment tracking from a local machine, but I'm encountering difficulties in uploading artifacts.I've tried a sample code as simple as the following.import mlflow import os os.environ["DATABRICKS_HOST"] = "...
- 101 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
A couple things: 1. If you don't own the mlflow experiment you need ot have edit permissions on the experiment (needed for logging). Default artifact locations in DBFS (`dbfs:/databricks/mlflow-tracking/`) require explicit write permissions 2. The lo...
- 0 kudos
