- 378 Views
- 1 replies
- 1 kudos
Access the data from cross-cloud.
Hi All ,We have a use case where we need to connect AWS Databricks to a GCP storage bucket to access the data. In Databricks We're trying to use external locations and storage credentials, but it seems like AWS Databricks only supports AWS storage b...
- 378 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Phani1 ,You can use delta sharing. In that way you can create share that will allow you to access data stored in GCS and it's govern by UC permissions model.What is Delta Sharing? | Databricks on AWSYou can also use legacy approach, but it doesn'...
- 1 kudos
- 5176 Views
- 4 replies
- 0 kudos
Encountering Error UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE
As of this morning we started receiving the following error message on a Databricks job with a single Pyspark Notebook task. The job has not had any code changes in 2 months. The cluster configuration has also not changed. The last successful run of ...
- 5176 Views
- 4 replies
- 0 kudos
- 0 kudos
i am also facing the same issue :- [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Unity Credential Scope id not found in thread locals.Issue occurs:-when we try to list files using dbutils.fs.lsand also this occurs at times when we ...
- 0 kudos
- 821 Views
- 1 replies
- 2 kudos
Resolved! Azure Databricks quota restrictions on compute in Azure for students subscription
Hi All,Regrading creating clusters in Databricks I'm getting quota error have tried to increase quotas in the region where the resource is hosted still unable to increase the limit, is there any workaround or could you help select the right cluster ...
- 821 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @svm_varma ,You can try to create Standard_DS3_v2 cluster. It has 4 cores and your current subscription limit for given region is 6 cores. The one you're trying to create needs 8 cores and hence you're getting quota exceeded exception.You can also...
- 2 kudos
- 31165 Views
- 6 replies
- 10 kudos
Resolved! Table write command stuck "Filtering files for query."
Hello all,Background:I am having an issue today with databricks using pyspark-sql and writing a delta table. The dataframe is made by doing an inner join between two tables and that is the table which I am trying to write to a delta table. The table ...
- 31165 Views
- 6 replies
- 10 kudos
- 10 kudos
Even if I vacuum and optimize, it keeps getting stuck.cluster type is r6gd.xlarge min:4, max:6driver type is r6gd.2xlarge
- 10 kudos
- 738 Views
- 9 replies
- 0 kudos
databricks job taking longer time to load 2.3 gb data from bolb to ssms table
df_CorpBond= spark.read.format("parquet").option("header", "true").load(f"/mnt/{container_name}/raw_data/dsl.corporate.parquet") df_CorpBond.repartition(20).write\ .format("jdbc")\ .option("url", url_connector)\ .option("dbtable", "MarkIt...
- 738 Views
- 9 replies
- 0 kudos
- 0 kudos
Hi @aayrm5 this is my dag digramfile size is 3.5 gb and in future we need to load 14gb as well
- 0 kudos
- 405 Views
- 0 replies
- 0 kudos
Dataiku and Databricks a promising future to democratize AI.
Democratize Data and AI is the prime focus of organization in 2025. With this some of the products that are going to fly with their offerings are Dataiku and Databricks.Please check my article on this:https://medium.com/@infinitylearnings1201/databri...
- 405 Views
- 0 replies
- 0 kudos
- 1864 Views
- 1 replies
- 1 kudos
Apache Spark SQL query to get organization hierarchy
I'm currently diving deep into Spark SQL and its capabilities, and I'm facing an interesting challenge. I'm eager to learn how to write CTE recursive queries in Spark SQL, but after thorough research, it seems that Spark doesn't natively support recu...
- 1864 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @singhanuj2803, It is correct that Spark SQL does not natively support recursive Common Table Expressions (CTEs). However, there are some workarounds and alternative methods you can use to achieve similar results. Using DataFrame API with Loops:...
- 1 kudos
- 375 Views
- 1 replies
- 1 kudos
How to run stored procedure in Azure Database for PostgreSQL using Azure Databricks Notebook
We have Stored Procedure available in Azure Database for PostgreSQL and we want to call or run or execute the postgreSQL stored procedures in Azure Databricks through NotebookWe are attempting to run PostgreSQL stored procedures, through Azure Databr...
- 375 Views
- 1 replies
- 1 kudos
- 1 kudos
To execute a PostgreSQL stored procedure from an Azure Databricks notebook, you need to follow these steps: Required Libraries:You need to install the psycopg2 library, which is a PostgreSQL adapter for Python. This can be done using the %pip install...
- 1 kudos
- 1113 Views
- 1 replies
- 0 kudos
Resolved! How to execute SQL stored procedure in Azure Database for SQL Server using Azure Databricks Notebook
We have Stored Procedure available in Azure Database for SQL Server and we want to call or run or execute the postgreSQL stored procedures in Azure Databricks through NotebookWe are attempting to run SQL stored procedures, through Azure Databricks no...
- 1113 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @singhanuj2803, To execute a SQL stored procedure in Azure Databricks, you can follow these steps: Required Libraries:You need to install the pyodbc library to connect to Azure SQL Database using ODBC. You can install it using the following comman...
- 0 kudos
- 1675 Views
- 3 replies
- 3 kudos
Resolved! Deleting records from Delta table that are not in relational table
I have a Delta table that I keep in sync with a relational (SQL Server) table. The inserts and updates are easy but checking for records to delete is prohibitively slow. I am querying the relational table for all primary key values and any primary ke...
- 1675 Views
- 3 replies
- 3 kudos
- 3 kudos
Let's understand the complexity behind this code when executed on delta table along with Spark.pks = spark.read.format("jdbc").option("query": "SELECT pk FROM sql_table_name").load() delta_table = spark.read.table(delta_table_name) r = target_table.f...
- 3 kudos
- 1696 Views
- 9 replies
- 4 kudos
Resolved! Magic Commands (%sql) Not Working with Databricks Extension for VS Code
Hi Community,I’ve encountered an issue with the Databricks Extension for VS Code that seems to contradict the documentation. According to the Databricks documentation, the extension supports magic commands like %sql when used with Databricks Connect:...
- 1696 Views
- 9 replies
- 4 kudos
- 4 kudos
In reality, there is nothing to do with grpc_wait_for_shutdown_with_timeout. Although we haven't yet implemented a solution, we have an open issue for it, but it shouldn't stop SQL magic from working.Or is the "Interactive" tab where you encounter th...
- 4 kudos
- 1664 Views
- 7 replies
- 2 kudos
Databricks Job cluster for continuous run
Hi AllI am having situation where I wanted to run job as continuous trigger by using job cluster, cluster terminating and re-creating in every run within continuous trigger.I just wanted two know if we have any option where I can use same job cluster...
- 1664 Views
- 7 replies
- 2 kudos
- 2 kudos
@Ajay-Pandey cant we achieve the similar functionalities with the help of cluster Pools , why don't you try cluster pools.
- 2 kudos
- 3653 Views
- 13 replies
- 10 kudos
Suddenly can't find the option to uplaod files into Databricks Community Edition
Hi everyone,I am suddenly unable to find the option to upload my files into Databricks Community Edition today. Please find the same in the screenshot attached. Is anyone else also facing this issue?
- 3653 Views
- 13 replies
- 10 kudos
- 10 kudos
GChandra, did it get removed again? I just signed up for Community free edition, and I don't see an option to Upload files. Please advise.
- 10 kudos
- 1225 Views
- 3 replies
- 1 kudos
Deleting Records from DLT Bronze and Silver Tables
I have a pipeline that generates two DLT streaming tables: a Bronze table and a Silver table. I need to delete specific records from both tables. I've read an article (https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-de...
- 1225 Views
- 3 replies
- 1 kudos
- 1 kudos
Remove records using the DELETE operation in both Bronze & Silver tables.After doing each delete step, you can Optimize the table which rewrites the parquet files for that table behind the scenes to improve the data layout (Read more about optimize h...
- 1 kudos
- 1371 Views
- 2 replies
- 0 kudos
Volumes unzip files
I have this shell unzip that I use to unzip files %shsudo apt-get updatesudo apt-get install -y p7zip-full But when it comes to new workspace, I get error sudo: a terminal is required to read the password; either use the -S option to read from standa...
- 1371 Views
- 2 replies
- 0 kudos
- 0 kudos
First, you can read the ZIP file in a binary format [ spark.read.format("binaryFile") ], then use the zipfile Python package to unzip and extract all the files from the zipped file and store them in a Volume.
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
