Hello everyone,I am a doctoral researcher at the University of the Cumberlands and seeking 2–3 volunteers for a 20–25-minute field test for my dissertation research on Lakehouse platform adoption.The field test will be conducted over Zoom or Microsof...
Given free edition data bricks has serverless compute only is there any work around to chose ML Specific computes like belowis paying for it the only option ?
Hi,In May 2026, I have tried with the environment v5 and still the same issue. However, looks like a Databricks employee answered short before, that in environment v4 it was available again, but not working for me, neither v4 nor v5.https://www.reddi...
Has anyone been experiencing the issue of R plots not rendering in notebooks, starting a few days ago?t's not related to splarkly or plotly, or specifc data types, or anything. For example in base R: plot(1:3, 5:7) calculates without error, but does ...
The device to file workaround doesn't work for complex plots from ggarrange. I also stopped seeing plots in all old notebooks. Hope this is fixed soon.
Saving logs from an all-purpose cluster to Volume or S3 is not consistent, because stderr, stdout, and log4j-active.log get overwritten when the cluster is restarted between minutes 01 and 59.Tested case:A job is configured to start every 20 minutes,...
Hi @ccsalt , This is a known limitation. Log rotation (renaming to log4j-YYYY-MM-DD-HH.log.gz) only happens on the hour boundary. The active log file log4j-active.log has always the same name and is overwritten if a cluster restart happens within one...
I'm looking for technical references to connect an Azure Databricks serverless workspace to an S3 bucket over a private site-to-site VPN connection. Found the following to connect AWS (consumer) to Azure (provider), but I'm looking for the other way....
Hi,I'm looking at moving from job clusters to serverless environments. Ideally to reduce cost and improve start up time.I can see that it is now possible to specify a custom environment .yaml file - and specify Python packages to be installed.Is ther...
Hi @AlexM There isn’t currently a way to bring a pre-built container image into serverless notebooks/jobs. Serverless supports custom environment YAML files and dependency installation/caching, but Databricks Container Services isn’t supported on ser...
Hi everyone,I'm using Azure Databricks with a customer who has a SQL Server database federated on the Unity Catalog.It seems that, while converting some date functions to the SQL Server dialect, Databricks uses the function "extract", which is not re...
Hi @Alessio_F ,This happens because in Databricks SQL both year and month functions are just aliases over following patterns:- extract (YEAR FROM expr)- extract(MONTH FROM expr) When Databricks pushes a predicate or expression down to the remote SQL ...
Hi everyone,I am looking to create a notebook that, when executed by a user, performs the following actions:Retrieves all Databricks jobs created by the current userChecks whether a specific role already has permissions on those jobsAutomatically add...
Hi @Raj_DB You can use databricks SDK to retrieve all jobs filter them by selecting only those where owner is current usersomething like thisfrom databricks.sdk import WorkspaceClient w = WorkspaceClient() # Specify the user email/username you want...
Hi experts,I'm having a problem when trying to create an external catalog with my PostgreSQL database. The connection is fine. But the database name that I want to connect has dashes and colon (eg. my-db-prod:all). When trying to connect with it, I a...
Hi @micheloh, From what we’ve seen, this is currently a limitation of Lakehouse Federation foreign catalog creation rather than a problem with the connection itself. The PostgreSQL connection can succeed, but the database value used when creating the...
Hi folks,I would like to ask for best practises concerning the topic of parametrizing queries in Databricks Asset Bundle deployments.This topic is relevant to differentiate between deployments on different environments as well as [dev]-deployments vs...
Came across this thread as I was facing the same exact issue as @malterializedvw and I want to comment my fix in case anyone else tears their hair out on this problem.In my databricks.yml file I put in a gold_catalog and a silver_catalog variable and...
I'm looking to know programatically how many files a delta table is made of.I know I can do %sqlDESCRIBE DETAIL my_tableBut that would only give me the number of files of the current version. I am looking to know the total number of files (basically ...
The best way to get this is executing the following statement:ANALYZE TABLE [table_name] COMPUTE STORAGE METRICS;Applies to: Databricks Runtime 18.0 and above
One thing that really changed how I think about BI platforms happened while I was working in a large enterprise environment heavily invested in Tableau.On paper, the environment looked mature: lots of dashboards, lots of business areas onboarded, and...
I'm trying to run backfill with the following parameter. How can I access this in the Python script?Do I need to change anything in the yml?I usually set task parameters the following way:These are then parsed using argparse Python module.
Found the following solution:Add job level parameters:parameters: - name: run_timestamp default: "some_default_value" Reference in task level parameters:tasks: - task_key: my_task spark_python_task: python_file: ../../script.py ...
In ingestion pipeline, for query based connector there is option of selecting batch snapshot instead of column name under dropdown - Cursor column. If I choose batch snapshot, will the databricks engine run select * from my source table, say Sql serv...
@rbtv It may execute something very similar to a `SELECT *` on the source table unless the platform adds its own partitioning or optimisation behind the scenes. From what I've observed, selecting batch snapshot often means the connector handles each ...
Hi,The option to enable custom and community connectors does not seem to be available on the Previews page, how can this be enabled? Feature I'm referencing: Community connectors in Lakeflow Connect - Azure Databricks | Microsoft Learn
Hi @koen_hai, The Community Connectors feature is controlled from the workspace-level Previews page by a workspace admin. If you don’t see that option there, the workspace likely hasn’t been enrolled for the preview yet. In that case, please contact ...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
