- 6586 Views
- 4 replies
- 0 kudos
Parallel execution of SQL cell in Databricks Notebooks
Hi Team,Please provide guidance on enabling SQL cells parallel execution in a notebook containing multiple SQL cells. Currently, when we execute notebook and all the SQL cells they run sequentially. I would appreciate assistance on how to execute th...
- 6586 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Phani1 Yes you can achieve this scenario with the help of Databricks Workflow jobs where you can create task and dependencies for each other.
- 0 kudos
- 3744 Views
- 5 replies
- 0 kudos
why the userIdentity is anonymous?
Do you know why the userIdentity is anonymous in AWS Cloudtail's logs even though I have specified an instance profile?
- 3744 Views
- 5 replies
- 0 kudos
- 0 kudos
If you're using AssumeRole to switch roles, make sure that the assumed role session is being used correctly. The Security Token Service (STS) is responsible for issuing temporary security credentials when assuming roles. Ensure that your EC2 instance...
- 0 kudos
- 10124 Views
- 3 replies
- 2 kudos
My cluster can't access any tables in data catalogs
My goal is to have table access control in the data science and engineering workspace. So I enabled access control to my cluster using this config "spark.databricks.acl.dfAclsEnabled": "true" and my cluster is shown as Table ACLs enabled now (shield ...
- 10124 Views
- 3 replies
- 2 kudos
- 2 kudos
Here is my use case: https://community.databricks.com/t5/data-engineering/structured-streaming-using-delta-as-source-and-delta-as-sink-and/td-p/67825And I get this error: "py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.Datase...
- 2 kudos
- 1115 Views
- 1 replies
- 0 kudos
Structured Streaming using Delta as Source and Delta as Sink and Delta tables are under unity catalo
Hello Everyone,Here is my use case.1. My source table (bronze delta table) is under unity catalog and is a transaction (Insert/Update) table.2. My target table (silver delta table) is also under unity catalog.3. On daily basis I need to ingest the in...
- 1115 Views
- 1 replies
- 0 kudos
- 0 kudos
I came across this article : readStream() is not whitelisted error when running a query - Databricksit states the solution as " You should use a cluster that does not have table access control enabled for streaming queries."However, the source and ta...
- 0 kudos
- 8987 Views
- 9 replies
- 3 kudos
DLT managed by non-existent pipeline
I am building out a new DLT pipeline and have since had to rebuild it from scratch. Having deleted the old pipeline and constructed a new one I now get this error:Table 'X' is already managed by pipeline 'Y'. As I only have the one pipeline how would...
- 8987 Views
- 9 replies
- 3 kudos
- 3 kudos
rename your function from @Dlt.table, for exemple:@Dlt.table( comment="exemple", table_properties={"exemple": "exemple"}, partition_cols=["a", "b", "c"])def modify_this_name():
- 3 kudos
- 1202 Views
- 0 replies
- 0 kudos
Ingestion time clustering -Initial load
As per info available ingestion time clustering makes use of time of the time a file is written or ingested in databricks. In a use case where there is new delta table and an etl which runs in timely fashion(say daily) inserting records, am able to ...
- 1202 Views
- 0 replies
- 0 kudos
- 978 Views
- 1 replies
- 0 kudos
URGENT - Error while logging in
I am facing an login issue since yesterday, it says the wrong email address or password but I am entering the right one.Even if I click on forgot password I don't get any link to reset it on my respective email. Please help!
- 978 Views
- 1 replies
- 0 kudos
- 0 kudos
It's a login attempt in the community version!
- 0 kudos
- 3170 Views
- 5 replies
- 0 kudos
Resolved! Temporary views no longer working for Share Compute
If I do this%sqlcreate or replace temporary view myviewasselect * from silver.<schema>.<table>;SHOW VIEWS;select * from myview;It works. But if I do the same on a Shared Compute it fails with[TABLE_OR_VIEW_NOT_FOUND] The table or view `myview` cannot...
- 3170 Views
- 5 replies
- 0 kudos
- 12267 Views
- 6 replies
- 2 kudos
EDU discount for university research project
I need databricks for a university research project. Is there any possibility of EDU discounts on DBU? So far I was unable to reach out to Databricks sales. Can you connect me with someone from DB?
- 12267 Views
- 6 replies
- 2 kudos
- 6227 Views
- 8 replies
- 0 kudos
Databricks Asset Bundle Workflow Redeployment Issue
Hello All,In my Databricks workflows, I have three tasks configured, with the final task set to run only if the condition "ALL_DONE" is met. During the first deployment, I observed that the dependency "ALL_DONE" was correctly assigned to the last tas...
- 6227 Views
- 8 replies
- 0 kudos
- 0 kudos
@Sikki thanks for confirming. For Azure DevOps please check the version of Databricks CLI installed.
- 0 kudos
- 1962 Views
- 2 replies
- 0 kudos
DownLoad CSV files from Delta Lake
We have around 1800 tables in Parq format (Delta Lake). These 1800 tables are very big, we have all these 1800 tables are converted into tables. But we have a requirement that, we need to download in CSV. (from PowerBI / any other reporting tool). Cu...
- 1962 Views
- 2 replies
- 0 kudos
- 0 kudos
Dear Kaniz,Thank you. one doubt 1) converting tables data into CSV and saving again one more of storage layer.IS there any way on fly we can convert these tables into CSV's. and export into PowerBI? and again i see in powerBI has limitations around >...
- 0 kudos
- 7660 Views
- 2 replies
- 1 kudos
The star inside WHERE
The star (*) can be used inside the WHERE clause in #Databricks as of runtime version 15.
- 7660 Views
- 2 replies
- 1 kudos
- 4364 Views
- 3 replies
- 1 kudos
Filter not using partition
I have the following code:spark.sparkContext.setCheckpointDir("dbfs:/mnt/lifestrategy-blob/checkpoints") result_df.repartitionByRange(200, "IdStation") result_df_checked = result_df.checkpoint(eager=True) unique_stations = result_df.select("IdStation...
- 4364 Views
- 3 replies
- 1 kudos
- 1 kudos
it seems like there is a filter being apply according to this. Filter (isnotnull(IdStation#2678) AND (IdStation#2678 = 1119844)) I would like to share the following notebook that covers in detail this topic, in case you would like to check it out h...
- 1 kudos
- 7103 Views
- 1 replies
- 0 kudos
RocksDB results in empty stream/stream joins dataframe
Since we enable RocksDB in our spark.conf the stream to stream joins/unions results in empty dataframe, does anyone else have the same experience? it is on AWSspark.conf.set("spark.sql.streaming.stateStore.providerClass","com.databricks.sql.streaming...
- 7103 Views
- 1 replies
- 0 kudos
- 0 kudos
Did you also update the checkpoint? You might need to use a new checkpoint after you enable the RocksDB state store.
- 0 kudos
- 3527 Views
- 5 replies
- 2 kudos
Trying to run databricks academy labs, but execution fails due to method to clearcache not whilelist
Hi there,Im trying to run DE 2.1 - Querying Files Directly on my workspace with a default cluster configuration for found below,but I cannot seem to run this file (or any other labs) as it gives me this error message Resetting the learning environme...
- 3527 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Retired_mod and databricks team,Did you already found some other solution for this? Thanks,Bram
- 2 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
