- 1889 Views
- 1 replies
- 0 kudos
- 1889 Views
- 1 replies
- 0 kudos
- 0 kudos
You can now use Delta Sharing to share notebook files securely using the Databricks-to-Databricks sharing flow.Sharing notebooks empowers users to collaborate across metastores and accounts, and enables providers to demonstrate use cases and visualiz...
- 0 kudos
- 8010 Views
- 3 replies
- 0 kudos
Resolved! Unable to install R geospatial libraries raster, terra, sf, ncdf4, etc
When trying to install any of this R libraries from a cmd cell/block on a notebook, or from the UI in the cluster I receive the same error,seeming that are unable to install dependecies.Warning in utils::install.packages(pkgs, ...) : installation of ...
- 8010 Views
- 3 replies
- 0 kudos
- 0 kudos
For the next soul looking for an answer, I managed to solve the issue with the next 2 Init scripts, it gets tricky in the apt or apt-get, that was the issue, sometimes it did update, sometimes it didn't, making it possible to find libmysqlclient21.1)...
- 0 kudos
- 1211 Views
- 0 replies
- 0 kudos
Number of rows displayed in sql cell
Even though the default limit on rows displayed is 10,000, the SQL cell is showing rows less than the limit when my resultant has more rows than 10k.It should alteast show the default limit .
- 1211 Views
- 0 replies
- 0 kudos
- 7738 Views
- 4 replies
- 2 kudos
Resolved! how to make databricks job to fail when the application has already given "exit code 1"?
object OurMainObject extends LazyLogging with IOApp { def run(args: List[String]): IO[ExitCode] = { logger.info("Started the application") val conf = defaultOverrides.withFallback(defaultApplication).withFallback(defaultReference) val...
- 7738 Views
- 4 replies
- 2 kudos
- 2 kudos
my workaround now is to make the code like below, so the databricks jobs becomes failure. case Left(ex) => { IO(logger.error("Glue failure", ex)).map(_ => ExitCode.Error) IO.raiseError(ex) }
- 2 kudos
- 8281 Views
- 3 replies
- 3 kudos
Resolved! Roll back to previous version of an AutoLoader checkpoint file
I know to "reset" AutoLoader, you can delete the checkpoint file entirely. I was wondering if it's possible to and how would someone :Get the checkpoint file to a previous version so I can reload certain files that were already processedDelete certai...
- 8281 Views
- 3 replies
- 3 kudos
- 2357 Views
- 0 replies
- 4 kudos
databricks Runtime 13.1 has added the sql_keywords() function, which lists all SQL keywords. It is a good practice to refrain from using these keyword...
databricks Runtime 13.1 has added the sql_keywords() function, which lists all SQL keywords. It is a good practice to refrain from using these keywords as names for tables or fields, although, in standard ANSI false mode, it will work without problem...
- 2357 Views
- 0 replies
- 4 kudos
- 5367 Views
- 2 replies
- 1 kudos
Resolved! Notebook activity is getting timed out in ADF pipeline.
Notebook activity is getting timed out after certain time of running (5 hours) in ADF pipeline and getting timeout error.Its just simply getting timed out error. Problem is this will process TB of data daily. can anyone have any idea to fix this.
- 5367 Views
- 2 replies
- 1 kudos
- 7104 Views
- 2 replies
- 9 kudos
You can use apache hudi in databricks without a problem:Â - in cluster settings, install Maven library org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0...
You can use apache hudi in databricks without a problem:- in cluster settings, install Maven library org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0 for Databricks 12.2 LTS- in cluster spark config, add three lines:spark.sql.extensions org.apache.sp...
- 7104 Views
- 2 replies
- 9 kudos
- 9 kudos
I tried installing library and configuring spark configs, restarted the cluster and then in notebook ran the create cmd but it gives me error stating java.io.FileNotFoundException: No such file or directory: s3://incred-databricks-data/hudi_dms_data/...
- 9 kudos
- 2402 Views
- 1 replies
- 2 kudos
Resolved! Shallow copy in databricks
Hi, I am new to Databricks. I need to setup a non-prod environment for which I need data of prod to be cloned in non-prod. Explored some and got to know about shallow copy. Is it possible to do shallow copy across environments? or Is it possible to d...
- 2402 Views
- 1 replies
- 2 kudos
- 2 kudos
@deepak prasad​ I'm not sure it's possible to do that. Even with Unity Catalog enabled, you cannot use shallow clone.You can do two things here:Without UC - just simply recreate an empty table in your non-prod environment and do SELECT * from prod st...
- 2 kudos
- 2276 Views
- 1 replies
- 2 kudos
Resolved! Databricks Account
Hi,In my org, we are using Azure Databricks. As an Azure AD user, I and my project team have access to Databricks workspaces. In our context, what's exactly meant as Databricks Account? I understand it's a group of workspaces used for billing, but at...
- 2276 Views
- 1 replies
- 2 kudos
- 2 kudos
@Senthilnathan J​ Databricks Account is like a top level of administration layer for everything that's going on your tenant.
- 2 kudos
- 3052 Views
- 1 replies
- 2 kudos
Resolved! Version Controlling SQL Query snippets
Hello, I have a suite of SQL Queries for creating table views (i.e)``` CREATE OR REPLACE VIEW silver.filtered_samples.metadata_table AS ( SELECT * FROM bronze.samples.table WHERE sample_status ='pass' )```I have tried moving these a repo but I ...
- 3052 Views
- 1 replies
- 2 kudos
- 2 kudos
@Alexis Lucattini​ You're right, the best approach would be to use notebooks with a single %sql magic block.
- 2 kudos
- 2826 Views
- 1 replies
- 1 kudos
Resolved! Databricks query history api with filter_by warehouse_id
Hi, Im trying to pull query history filtered by warehouse id but my url is not working. Do you have an example on how the url will looks like?I tried this --> https://**.cloud.databricks.com/api/2.0/sql/history/queries?filter_by={"warehouse_id":"193b...
- 2826 Views
- 1 replies
- 1 kudos
- 1 kudos
Oh, looks like I need to add this raw data. { "filter_by": { "warehouse_ids": "193b15a590ed23d2" }}
- 1 kudos
- 1377 Views
- 0 replies
- 0 kudos
Spark DataFrame apply Databricks geospatial indexing functions
I have a spark DataFrame with `h3` hex ids and I am trying to obtain the polygon geometries. from pyspark.sql import SparkSession from pyspark.sql.functions import col, expr from pyspark.databricks.sql.functions import * from mosaic import enable_m...
- 1377 Views
- 0 replies
- 0 kudos
- 13567 Views
- 2 replies
- 6 kudos
METASTORE_DOWN: Cannot connect to metastore
I am trying to view databases and tables, default as well user created but it looks like the cluster created is not able to connect. I am using databricks default hive metastore. Viewing cluster logs provide following ventMETASTORE_DOWN Metastore is...
- 13567 Views
- 2 replies
- 6 kudos
- 6 kudos
@rami​ :If the metastore is down, it means that the Databricks cluster is not able to connect to the metastore. Here are a few things you can try to resolve the issue:Check if the Hive metastore is up and running. You can try to connect to the metast...
- 6 kudos
- 3638 Views
- 2 replies
- 2 kudos
Resolved! Error 95 when importing one Notebook into another
When I follow the instructions Modularize your code using files I get the following error:I am on azure, use DBRT 12.2 LTS, use ADLS as storage, I am happy to provide more details if needed. My research suggest that the reason is that the dfbs fuse...
- 3638 Views
- 2 replies
- 2 kudos
- 2 kudos
import works for .py files..%run is for notebooks.is lib a .py file or a notebook?
- 2 kudos
| User | Count |
|---|---|
| 1638 | |
| 791 | |
| 536 | |
| 349 | |
| 287 |
