- 9652 Views
- 3 replies
- 3 kudos
Resolved! Roll back to previous version of an AutoLoader checkpoint file
I know to "reset" AutoLoader, you can delete the checkpoint file entirely. I was wondering if it's possible to and how would someone :Get the checkpoint file to a previous version so I can reload certain files that were already processedDelete certai...
- 9652 Views
- 3 replies
- 3 kudos
- 2649 Views
- 0 replies
- 4 kudos
databricks Runtime 13.1 has added the sql_keywords() function, which lists all SQL keywords. It is a good practice to refrain from using these keyword...
databricks Runtime 13.1 has added the sql_keywords() function, which lists all SQL keywords. It is a good practice to refrain from using these keywords as names for tables or fields, although, in standard ANSI false mode, it will work without problem...
- 2649 Views
- 0 replies
- 4 kudos
- 6472 Views
- 2 replies
- 1 kudos
Resolved! Notebook activity is getting timed out in ADF pipeline.
Notebook activity is getting timed out after certain time of running (5 hours) in ADF pipeline and getting timeout error.Its just simply getting timed out error. Problem is this will process TB of data daily. can anyone have any idea to fix this.
- 6472 Views
- 2 replies
- 1 kudos
- 9230 Views
- 2 replies
- 9 kudos
You can use apache hudi in databricks without a problem:Â - in cluster settings, install Maven library org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0...
You can use apache hudi in databricks without a problem:- in cluster settings, install Maven library org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0 for Databricks 12.2 LTS- in cluster spark config, add three lines:spark.sql.extensions org.apache.sp...
- 9230 Views
- 2 replies
- 9 kudos
- 9 kudos
I tried installing library and configuring spark configs, restarted the cluster and then in notebook ran the create cmd but it gives me error stating java.io.FileNotFoundException: No such file or directory: s3://incred-databricks-data/hudi_dms_data/...
- 9 kudos
- 3062 Views
- 1 replies
- 2 kudos
Resolved! Shallow copy in databricks
Hi, I am new to Databricks. I need to setup a non-prod environment for which I need data of prod to be cloned in non-prod. Explored some and got to know about shallow copy. Is it possible to do shallow copy across environments? or Is it possible to d...
- 3062 Views
- 1 replies
- 2 kudos
- 2 kudos
@deepak prasad​ I'm not sure it's possible to do that. Even with Unity Catalog enabled, you cannot use shallow clone.You can do two things here:Without UC - just simply recreate an empty table in your non-prod environment and do SELECT * from prod st...
- 2 kudos
- 3421 Views
- 1 replies
- 2 kudos
Resolved! Databricks Account
Hi,In my org, we are using Azure Databricks. As an Azure AD user, I and my project team have access to Databricks workspaces. In our context, what's exactly meant as Databricks Account? I understand it's a group of workspaces used for billing, but at...
- 3421 Views
- 1 replies
- 2 kudos
- 2 kudos
@Senthilnathan J​ Databricks Account is like a top level of administration layer for everything that's going on your tenant.
- 2 kudos
- 3770 Views
- 1 replies
- 2 kudos
Resolved! Version Controlling SQL Query snippets
Hello, I have a suite of SQL Queries for creating table views (i.e)``` CREATE OR REPLACE VIEW silver.filtered_samples.metadata_table AS ( SELECT * FROM bronze.samples.table WHERE sample_status ='pass' )```I have tried moving these a repo but I ...
- 3770 Views
- 1 replies
- 2 kudos
- 2 kudos
@Alexis Lucattini​ You're right, the best approach would be to use notebooks with a single %sql magic block.
- 2 kudos
- 3480 Views
- 1 replies
- 1 kudos
Resolved! Databricks query history api with filter_by warehouse_id
Hi, Im trying to pull query history filtered by warehouse id but my url is not working. Do you have an example on how the url will looks like?I tried this --> https://**.cloud.databricks.com/api/2.0/sql/history/queries?filter_by={"warehouse_id":"193b...
- 3480 Views
- 1 replies
- 1 kudos
- 1 kudos
Oh, looks like I need to add this raw data. { "filter_by": { "warehouse_ids": "193b15a590ed23d2" }}
- 1 kudos
- 1787 Views
- 0 replies
- 0 kudos
Spark DataFrame apply Databricks geospatial indexing functions
I have a spark DataFrame with `h3` hex ids and I am trying to obtain the polygon geometries. from pyspark.sql import SparkSession from pyspark.sql.functions import col, expr from pyspark.databricks.sql.functions import * from mosaic import enable_m...
- 1787 Views
- 0 replies
- 0 kudos
- 15208 Views
- 2 replies
- 6 kudos
METASTORE_DOWN: Cannot connect to metastore
I am trying to view databases and tables, default as well user created but it looks like the cluster created is not able to connect. I am using databricks default hive metastore. Viewing cluster logs provide following ventMETASTORE_DOWN Metastore is...
- 15208 Views
- 2 replies
- 6 kudos
- 6 kudos
@rami​ :If the metastore is down, it means that the Databricks cluster is not able to connect to the metastore. Here are a few things you can try to resolve the issue:Check if the Hive metastore is up and running. You can try to connect to the metast...
- 6 kudos
- 4721 Views
- 2 replies
- 2 kudos
Resolved! Error 95 when importing one Notebook into another
When I follow the instructions Modularize your code using files I get the following error:I am on azure, use DBRT 12.2 LTS, use ADLS as storage, I am happy to provide more details if needed. My research suggest that the reason is that the dfbs fuse...
- 4721 Views
- 2 replies
- 2 kudos
- 2 kudos
import works for .py files..%run is for notebooks.is lib a .py file or a notebook?
- 2 kudos
- 5621 Views
- 3 replies
- 4 kudos
How do I define & run jobs that execute scripts that are copied inside a custom DataBricks container?
Hi all, we are building custom Databricks containers (https://docs.databricks.com/clusters/custom-containers.html). During the container build process we install dependencies and also python source code scripts. We now want to run some of these scrip...
- 5621 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Thijs van den Berg​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 4 kudos
- 5656 Views
- 2 replies
- 1 kudos
Resolved! Is it possible to write a pyspark dataframe to a custom log table in Log Analytics workspace?
I have a pyspark dataframe that contains information about the tables that I have on sql database (creation date, number of rows, etc)Sample data: { "Day":"2023-04-28", "Environment":"dev", "DatabaseName":"default", "TableName":"discount"...
- 5656 Views
- 2 replies
- 1 kudos
- 1 kudos
@Bruno Simoes​ :Yes, it is possible to write a PySpark DataFrame to a custom log table in Log Analytics workspace using the Azure Log Analytics Workspace API.Here's a high-level overview of the steps you can follow:Create an Azure Log Analytics Works...
- 1 kudos
- 2367 Views
- 1 replies
- 1 kudos
Connecting to Azure SQL database using R in Databricks and Azure Key Vaults
I'm trying to connect to an Azure SQL database using R in Databricks. I want to read the credentials stored in Azure secret key vaults rather than hard coding in R code. I've seen some examples of it being done in Scala, however i'm after a R solutio...
- 2367 Views
- 1 replies
- 1 kudos
- 1 kudos
Did you find a solution for this @Long Pham​ ?? I am having the same issue
- 1 kudos
- 6832 Views
- 2 replies
- 4 kudos
Resolved! Delta Table Merge statement is not accepting broadcast hint
I have a statement like this with pyspark:target_tbl.alias("target")\ .merge(stage_df.hint("broadcast").alias("source"), merge_join_expr)\ .whenMatchedUpdateAll()\ .whenNotMatchedInsertAll()\ .w...
- 6832 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Nikolay Chalkanov​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
