- 990 Views
- 1 replies
- 0 kudos
Describe history operationMetrics['materializeSourceTimeMs']
Hi, during some checks on MERGE execution , I was running the describe history command and in the operationMetrics column I noticed this information : operationMetrics['materializeSourceTimeMs'] .I haven't found that metric in the documentation so I...
- 990 Views
- 1 replies
- 0 kudos
- 0 kudos
@roberta_cereda If it’s specific to “materializeSourceTimeMs” then it’s “time taken to materialize source (or determine it's not needed)”
- 0 kudos
- 2323 Views
- 3 replies
- 1 kudos
Resolved! Error while loading mosaic in notebook - TimeoutException: Futures timed out after [80 seconds]
I am working on reading spatial data with mosaic and gdal Previously I used databricks mosaic = 0.3.9 version with databricks cluster = 12.2 LTS version With following command - %pip install databricks-mosaic==0.3.9 --quiet Now It's giving timeout er...
- 2323 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @pranav_k1,Thanks for confirming it worked for you now!I see that the usual %pip install databricks-mosaic cannot install due to the fact that it has thus far allowed geopandas to essentially install the latest... As of geopandas==0.14.4, the vers...
- 1 kudos
- 4882 Views
- 2 replies
- 1 kudos
applyInPandas function hangs in runtime 13.3 LTS ML and above
Hello, recently I've tried to upgrade my runtime env to the 13.3 LTS ML and found that it breaks my workload during applyInPandas.My job started to hang during the applyInPandas execution. Thread dump shows that it hangs on direct memory allocation: ...
- 4882 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Debayan the link just redirects to the same thread? Is there any update on this issue?We share some similar issue on job hanging using mapInPandas.
- 1 kudos
- 1592 Views
- 3 replies
- 1 kudos
Unverified Commits via Databricks Repos: Seeking Solution for GitHub Verification
The team is integrating Databricks Repos with Personal Access Tokens (PAT) to commit code directly to GitHub. Our organization requires signed commits for verification purposes.Issue: When committing via Databricks Repos, the commits appear as unveri...
- 1592 Views
- 3 replies
- 1 kudos
- 1 kudos
Can you please share the link to this doc DB-I-3082. I couldn't find it.
- 1 kudos
- 1085 Views
- 1 replies
- 0 kudos
UI improvement - open multiple workspace notebooks
Hi all,I have an idea for a feature to open multiple workbooks. Currently, right-clicking a notebook in the Workspace will allow you to "Open in new tab". If I multi-select notebooks, I only have option to Move or Move to trash. Why not allow a us...
- 1085 Views
- 1 replies
- 0 kudos
- 0 kudos
Many thanks for your feedback and great idea. We have created idea DBE-I-1544, this will be analyzed by our team and if approved it can be implemented in the near future.
- 0 kudos
- 6919 Views
- 1 replies
- 0 kudos
DBT or Delta Live Tables
Quick question, when would people use DBT over Delta Live Tables? Or better yet can you use DBT to create Delta Live Tables?
- 6919 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta Live Tables (DLT): DLT is an ETL (Extract, Transform, Load) framework designed to simplify the creation and management of data pipelines. It uses a declarative approach to build reliable data pipelines and automatically manages infrastructure a...
- 0 kudos
- 2290 Views
- 2 replies
- 0 kudos
Photon plan invariant violated Error
We've run into a niche error where we get the below message only on our non prod environment, with the same data, with the same code as our prod environment.org.apache.spark.sql.execution.adaptive.InvalidAQEPlanException: Photon plan invariant violat...
- 2290 Views
- 2 replies
- 0 kudos
- 5343 Views
- 4 replies
- 2 kudos
Resolved! Error while Deploying Asset Bundle using Azure Devops
Hi,I'm trying to deploy the Asset Bundle using Azure DevOps, it is giving me this error Step: databricks bundle validate -t dev========================== Starting Command Output ===========================2024-09-02T05:41:19.9113254Z Error: failed du...
- 5343 Views
- 4 replies
- 2 kudos
- 2 kudos
I had similar error message but then using correct environment variables in the pipeline solved the problem. Especially setting DATABRICKS_HOST point to the account. More detailed description is here Databricks Asset Bundle OAuth Authentication in Az...
- 2 kudos
- 9866 Views
- 3 replies
- 2 kudos
Resolved! Import from repo
Hi all,I am trying the new "git folder" feature, with a repo that works fine from the "Repos". In the new folder location, my imports from my own repo don't work anymore. Anyone faced something similar?Thanks in advance for sharing your experience
- 9866 Views
- 3 replies
- 2 kudos
- 2 kudos
Hello,When I had the same issue even using 14.3+, I tried this codes.```pythonimport sysimport pprintpprint.pprint(sys.path)```And I noticed that the path was throwed by old(Legacy) Repos folder. I managed the same name of the folder & Repos.Then I r...
- 2 kudos
- 797 Views
- 1 replies
- 0 kudos
Error Creating Table from delta location dbfs
[UC_FILE_SCHEME_FOR_TABLE_CREATION_NOT_SUPPORTED] Creating table in Unity Catalog with file scheme dbfs is not supported. Instead, please create a federated data source connection using the CREATE CONNECTION command for the same table provider, then ...
- 797 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @vyasakhilesh,The error you're seeing, [UC_FILE_SCHEME_FOR_TABLE_CREATION_NOT_SUPPORTED], occurs because Unity Catalog in Databricks does not support creating tables directly from DBFS (Databricks File System) locations.In this case, you're trying...
- 0 kudos
- 1667 Views
- 6 replies
- 0 kudos
Resolved! SQL View Formatting in Catalog - Can you turn it off?
It appears as though Databricks now formats SQL View definitions when showing them in the Catalog. Our solution is based on views, and we have comment tags in those views. We format these views so that it is easy for us to find and update parts of ...
- 1667 Views
- 6 replies
- 0 kudos
- 0 kudos
Thank you! I will submit that request.In case anyone else stumbles upon this post, here is a function you can add that will return the view definition from information_schema, unformatted with an ALTER statement at the top:DROP FUNCTION IF EXISTS <c...
- 0 kudos
- 6766 Views
- 7 replies
- 2 kudos
ODBC on Windows -- Where to specify Catalog name?
We are attempting to connect a Windows ODBC application to Unity Catalog.The Configure the Databricks ODBC and JDBC drivers documentation has a section titled "ODBC configuration and connection parameters" that mentions a configuration parameter call...
- 6766 Views
- 7 replies
- 2 kudos
- 2 kudos
It's quite interesting - I am using Mac, Simba spark ODBC 2.8.2 - and, If I will not add "Catalog" parameter - in UI, I will only see default one (If I will have access to it) - that doesn't mean I cannot query other one, It's just not listed in the ...
- 2 kudos
- 7509 Views
- 9 replies
- 0 kudos
DLT piplines with UC
I try to run a (very simple) DLT pipeline in with a resulting materialized table is published in UC schema with a managed storage location defined (within an existing EXTERNAL LOCATION). Accoding to the documentation: Publishing to schemas that speci...
- 7509 Views
- 9 replies
- 0 kudos
- 29768 Views
- 5 replies
- 11 kudos
Resolved! How to import a function to another notebook using Repos without %run?
Hi all,I was reading the Repos documentation: https://docs.databricks.com/repos.html#migrate-from-run-commandsIt is explained that, one advantage of Repos is no longer necessary to use %run magic command to make funcions available in one notebook to ...
- 29768 Views
- 5 replies
- 11 kudos
- 11 kudos
Due to new functionalies in Runtime 16.0 regarding autoload i came across this autoload. Performaed a practical test. It works. However had some problems at first.As in solution the key was that definitions are places in a file.py not a notebook.
- 11 kudos
- 2459 Views
- 3 replies
- 0 kudos
Create a simple Geospatial Table with Geography type column
Hello AllI am looking for guidance on creating an simple US states table with shape as Geography column type. I do not want to use Apache Sedona (due to cluster limitations). I am going to create an Node JS application which is going to query this ge...
- 2459 Views
- 3 replies
- 0 kudos
- 0 kudos
Thank you for prompt response. Yes, I have gone through the blog. I followed the same process ; for examples, buildings table have geometry which stores as binary type. My question, how to store geometry in 'Geography' type. The GEOGRAPHY data type ...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
