- 2160 Views
- 1 replies
- 0 kudos
Delta Sharing - Who provides the server?
I would like to understand who provides the server when using Delta sharing? If a customer exposes their delta table through Delta sharing, is it the customer who needs to setup a cluster or server to process the incoming requests?
- 2160 Views
- 1 replies
- 0 kudos
- 0 kudos
The producer does need a cluster to set up Delta Sharing. However, once the handoff happens no cluster is needed, the data will be delivered via storage services.
- 0 kudos
- 3725 Views
- 1 replies
- 2 kudos
Resolved! Is there a Python API for vacuum with dry run?
I have the below sql command where i am doing a dry run with vacuum. ​%sql VACUUM <table_name> RETAIN 500 HOURS DRY RUN;wanted to check if there is a way to achieve this in python api?​I​ tried the below. But, not sure if there is a parameter that we...
- 3725 Views
- 1 replies
- 2 kudos
- 2 kudos
Equivalent of sql command "VACUUM <table_name> RETAIN 500 HOURS DRY RUN;" in python is spark.sql("VACUUM <table_name> RETAIN 500 HOURS DRY RUN;")
- 2 kudos
- 15980 Views
- 3 replies
- 3 kudos
Resolved! Data Tab is not showing any databases and tables even though cluster is running (Community edition)
Hi, I have a cluster running: But I dont see anything in Data Tab: As you can see it tells about some error, but error appeared after I deleted clusters which were terminated. Before it said something that cluster is not running, dont remember exactl...
- 15980 Views
- 3 replies
- 3 kudos
- 3 kudos
@Retko Okter​ You need to enable DBFS File Browser from the Admin Settings hope this helps.
- 3 kudos
- 1444 Views
- 0 replies
- 0 kudos
Can the databricks_mount timeout be changed. ?
I am using terrafom to do databricks workspace configuration and while mounting 6 buckets if duration of mount is bigger than 20 min I get timeout. Is it possible to change the timeout ? thanksHoratiu
- 1444 Views
- 0 replies
- 0 kudos
- 1401 Views
- 0 replies
- 0 kudos
Does Spark Structured streaming query require write-ahead logs for at least once delivery with ForeachBatch output sink?
Is it sufficient to use the checkpoint directory with write-ahead logs?BTW. I use Kafka connector to read data from EventHub
- 1401 Views
- 0 replies
- 0 kudos
- 3991 Views
- 3 replies
- 3 kudos
Unable to run jobs with git notebooks
So, in this case our jobs are deployed from our development workspace to our isolated testing workspace via an automated Azure DevOps pipeline. As such, they are created (and thus run as) a service account user.Recently we made the switch to using gi...
- 3991 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Chris Block​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 3 kudos
- 2202 Views
- 0 replies
- 0 kudos
i am working on the parquet file level column encryption and decryption on user specific permission
i am able to encrypt and decrypt the daat in multiple ways and able to save the encrypted parquet file, but i want to decrypt the data if the user has specific permission otherwise he will get the encrypted data,.is there any permanent solution to de...
- 2202 Views
- 0 replies
- 0 kudos
- 3996 Views
- 2 replies
- 2 kudos
how to get dbutils in Runtime 13
We're using the following method (generated by using dbx) to access dbutils, e.g. to retrieve parameters from secret scopes: @staticmethod def _get_dbutils(spark: SparkSession) -> "dbutils": try: from pyspark.dbutils import...
- 3996 Views
- 2 replies
- 2 kudos
- 2 kudos
We have something similar in our code. This worked using runtime 13 until last week. Also the Machine Learning DBR doesn't work either.
- 2 kudos
- 12196 Views
- 4 replies
- 3 kudos
Resolved! Autoloader works on compute cluster, but does not work within a task in workflows
I feel like I am going crazy with this. I have tested a data pipeline on my standard compute cluster. I am loading new files as batch from a Google Cloud Storage bucket. Autoloader works exactly as expected from my notebook on my compute cluster. The...
- 12196 Views
- 4 replies
- 3 kudos
- 3 kudos
I found the issue. I describe the solution in the following SO post. https://stackoverflow.com/questions/76287095/databricks-autoloader-works-on-compute-cluster-but-does-not-work-within-a-task/76313794#76313794
- 3 kudos
- 1817 Views
- 1 replies
- 0 kudos
Function in databricks
Im having a hard time to convert below function from SSMS to databricks function. Any help would be appreciated! CREATE FUNCTION [dbo].[MaxOf5Values] (@D1 [int],@D2 [int],@D3 [int],@D4 [int],@D5 [int]) RETURNS int AS BEGIN DECLARE @Result int ...
- 1817 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Givi Salu​ ,​Please refer to this link that will help you convert this function.
- 0 kudos
- 6633 Views
- 1 replies
- 0 kudos
Execute COPY INTO query with source as temp view
We are having the files that needs to be loaded into the delta tableNow we want to perform some transformation on the files and load that into the tableWhat we didCreate a Spark DF from that fileApply transformation on the DFCreate a temp view from t...
- 6633 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sravan Kumar Mohanraj​ ,Yes, you can use copy query in this case your temp_view will be source.For more info, please visit these links.
- 0 kudos
- 4887 Views
- 2 replies
- 3 kudos
Resolved! API Authentication
I am trying to run some API calls to the account console. I tried with every syntax possible encoded and decoded to get the call successfully but it returns a "user not authenticated" error. But when I tried with the Account admin it worked. I need t...
- 4887 Views
- 2 replies
- 3 kudos
- 1211 Views
- 0 replies
- 1 kudos
�� Welcome to Chennai User Group Community! �� We are thrilled to have you join our vibrant and enthusiastic community of users! Whether...
Welcome to Chennai User Group Community! We are thrilled to have you join our vibrant and enthusiastic community of users! Whether you are a seasoned expert or a newcomer, this group is the perfect place for you to connect, learn, and grow alongside...
- 1211 Views
- 0 replies
- 1 kudos
- 5804 Views
- 3 replies
- 4 kudos
Resolved! how to create a scala jar using db notebook and save it in a file path inside databricks
I have scala function as below, i am unable to understand how to write a scala jar with the same, please find below code i have used Enforcing Column-Level Encryption - Databrick %scala import com.macasaet.fernet.{Key, StringValidator, Token}import o...
- 5804 Views
- 3 replies
- 4 kudos
- 4 kudos
I had to finally create the jar using teh intellij and sbt iconfiguration on the same env. and then installed the jar in the cluster it worked
- 4 kudos
- 8577 Views
- 5 replies
- 5 kudos
- 8577 Views
- 5 replies
- 5 kudos
- 5 kudos
If you are streaming to delta, not much, the micro batch will fail and in next time the stream will pick up from last successful write (due to ACID). I don't know about other formats, what happens if the stream is aborted in mid micro batch.
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
