- 11186 Views
- 8 replies
- 3 kudos
How to stop continuous running streaming job over weekend
I have an continuous running streaming Job, I would like to stop this over weekend and start again on Monday. Here is my streaming job code.(spark.readStream.format("delta").load(input_path).writeStream.option("checkpointLocation", input_checkpoint_p...
- 11186 Views
- 8 replies
- 3 kudos
- 3 kudos
@sanjay Any luck on that, I am also looking for the solution for the same issue
- 3 kudos
- 2115 Views
- 0 replies
- 0 kudos
Connect a SQL warehouse to an Azure Blob container
Hi all,I was wondering if it was possible to configure a single SQL warehouse to a specific Blob container?I know you can add configuration properties to the Workspace from the Admin settings, but it appears that those configurations will be shared b...
- 2115 Views
- 0 replies
- 0 kudos
- 66300 Views
- 14 replies
- 9 kudos
How to import one databricks python notebook into another?
I have a python notebook A in Azure Databricks having import statement as below: import xyz, datetime,... I have another notebook xyz being imported in notebook A as shown in above code. When I run notebook A, it throws the following error: ImportEr...
- 66300 Views
- 14 replies
- 9 kudos
- 9 kudos
Create a repository containing an __init__.py fileAdd your library as .py file(s). Let's imagine that our library is composed by multiple sub-folders consolidated in "my_folder", one of sub-folders is named as "math_library" and contains my_awesome_l...
- 9 kudos
- 8081 Views
- 9 replies
- 2 kudos
- 8081 Views
- 9 replies
- 2 kudos
- 2 kudos
Hi @Amardeep Singh​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 2 kudos
- 9431 Views
- 14 replies
- 0 kudos
Create a Hive db in Azure Databricks with the LOCATION parameter set to an ADLS GEN2 account fails
Hello,I am trying to create a database in Azure Databricks using the abfss Location in the create database statement and it throws an exception.%sqlCREATE DATABASE IF NOT EXISTS test COMMENT "Database for Test Area" LOCATION "abfss://test@storagetemp...
- 9431 Views
- 14 replies
- 0 kudos
- 0 kudos
ok.Can you check this link?Because I suspect something is wrong, and so dbrx uses the account key as a default (fs.azure.account.key)
- 0 kudos
- 2538 Views
- 2 replies
- 1 kudos
regexp_extract Error when put { in pattern
HI guys,I tried to use the regex_extract but when I put { in pattern I received errordisplay( df_containerlog.withColumn('reg', regexp_extract(col('LogEntry'), "((^.*))({'data':)", 1)) ) have any ideaThank you
- 2538 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello William,You have to escape the { as it is a special char in case of regex."((^.*))(\{'data':)"
- 1 kudos
- 3772 Views
- 2 replies
- 1 kudos
Resolved! How can I specify a custom CRAN mirror to be used permanently by default when installing packages within R Notebooks?
When installing Notebook-scoped R libraries I don't want to manually specify the custom CRAN mirror each time like this:install.packages("diffdf", repos="my_custom_cran_url'')Instead I want to take the custom CRAN mirror URL by default so that I don'...
- 3772 Views
- 2 replies
- 1 kudos
- 1 kudos
Got solution on Stack Overflow for this problem: https://stackoverflow.com/a/76777228/18082636
- 1 kudos
- 12966 Views
- 3 replies
- 3 kudos
Resolved! Error creating external location in Unity Catalog
HelloWhen I try to create an external location I get this error:Failed to access cloud storage: [AbfsRestOperationException] HTTP Error -1CustomTokenProvider getAccessToken threw com.databricks.api.base.DatabricksServiceException : INTERNAL_ERROR: Un...
- 12966 Views
- 3 replies
- 3 kudos
- 3 kudos
I think I sould have something missconfigured, the way I solved the problem was to re-create the workspace and start from scratch, it was a small one for testing proposes.
- 3 kudos
- 2675 Views
- 2 replies
- 6 kudos
_sqldf bugs in GCP workspaces?
Utilizing GCP instances using the 12.2DBR ML runtime. Prior to ~7:10CT last night, _sqldf commands in notebooks to reference the previously executed %sql cell would work locally as well as in scheduled Workflow Job runs. Now it appears that the code ...
- 2675 Views
- 2 replies
- 6 kudos
- 6 kudos
It looks like Azure was having the same issue, it might just be all 12.2 Photon clusters.https://community.databricks.com/t5/data-engineering/sqldf-bugs-in-gcp-workspaces/td-p/38578 That post says it was been fixed, but last I checked it was still fa...
- 6 kudos
- 5679 Views
- 2 replies
- 1 kudos
Resolved! How to solve - gRPC message exceeds maximum size 4194304 pubsub using databricks
I am getting below error while streaming the data from pubsub using databricks DLT pipelines If anyone can help to increase the gRPC message size will help alot.
- 5679 Views
- 2 replies
- 1 kudos
- 1996 Views
- 2 replies
- 0 kudos
Getting error while mounting storage account to databricks notebook
Please help to reolve this issue i implemented all possible steps, credentials are correct and i am not using free trial account, i was watching tutorial to mount i used same code and everything is done as said in tutorial but iam getting this weird ...
- 1996 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, I am not so sure. But very likely you have tried with any cluster with Access mode as Shared.Can you try with a single node cluster, if the problem still exists, please share your problem once again here.please tag me in your question again if ne...
- 0 kudos
- 14884 Views
- 5 replies
- 6 kudos
Snowflake connection to Databricks error
When I am trying to read snowflake table from my databricks notebook, it is giving the error as:df1.read.format("snowflake") \.options(**options) \.option("query", "select * from abc") \.save()Getting below errorjava.sql.SQLException: No suitable dri...
- 14884 Views
- 5 replies
- 6 kudos
- 12845 Views
- 3 replies
- 3 kudos
Resolved! query takes too long to write into delta table.
hello, am running into in issue while trying to write the data into a delta table, the query is a join between 3 tables and it takes 5 minutes to fetch the data but 3hours to write the data into the table, the select has 700 records. here are the app...
- 12845 Views
- 3 replies
- 3 kudos
- 3 kudos
it turned out that the issue was not in the writing side, even when i was getting the results in under 5min, the issue was in the cross join in my query i resolved the issue by doing the same cross joins via dataframes got the results computed and wr...
- 3 kudos
- 5389 Views
- 4 replies
- 3 kudos
Resolved! Issue while re-starting Streaming DLT pipeline - PUBSUB
I am not able to restart our streaming DLT pipeline as it's giving "__tmp_path_dir" number format error, I am using PUB-SUB as a source for streaming.Any solutions will help - #pusub #databricks
- 5389 Views
- 4 replies
- 3 kudos
- 39000 Views
- 9 replies
- 5 kudos
Resolved! SQL Declare Variable equivalent in databricks
Hello. What would be the equivalent of the below in databricks?DECLARE @LastChangeDate as dateSET @LastChangeDate = GetDate()I already tried the below and worked. However I need to know how set a sql variable dynamicallySET da.dbname = test;SELECT "$...
- 39000 Views
- 9 replies
- 5 kudos
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
