- 1203 Views
- 1 replies
- 0 kudos
Structured Streaming using Delta as Source and Delta as Sink and Delta tables are under unity catalo
Hello Everyone,Here is my use case.1. My source table (bronze delta table) is under unity catalog and is a transaction (Insert/Update) table.2. My target table (silver delta table) is also under unity catalog.3. On daily basis I need to ingest the in...
- 1203 Views
- 1 replies
- 0 kudos
- 0 kudos
I came across this article : readStream() is not whitelisted error when running a query - Databricksit states the solution as " You should use a cluster that does not have table access control enabled for streaming queries."However, the source and ta...
- 0 kudos
- 9391 Views
- 9 replies
- 3 kudos
DLT managed by non-existent pipeline
I am building out a new DLT pipeline and have since had to rebuild it from scratch. Having deleted the old pipeline and constructed a new one I now get this error:Table 'X' is already managed by pipeline 'Y'. As I only have the one pipeline how would...
- 9391 Views
- 9 replies
- 3 kudos
- 3 kudos
rename your function from @Dlt.table, for exemple:@Dlt.table( comment="exemple", table_properties={"exemple": "exemple"}, partition_cols=["a", "b", "c"])def modify_this_name():
- 3 kudos
- 1266 Views
- 0 replies
- 0 kudos
Ingestion time clustering -Initial load
As per info available ingestion time clustering makes use of time of the time a file is written or ingested in databricks. In a use case where there is new delta table and an etl which runs in timely fashion(say daily) inserting records, am able to ...
- 1266 Views
- 0 replies
- 0 kudos
- 1039 Views
- 1 replies
- 0 kudos
URGENT - Error while logging in
I am facing an login issue since yesterday, it says the wrong email address or password but I am entering the right one.Even if I click on forgot password I don't get any link to reset it on my respective email. Please help!
- 1039 Views
- 1 replies
- 0 kudos
- 0 kudos
It's a login attempt in the community version!
- 0 kudos
- 3453 Views
- 5 replies
- 0 kudos
Resolved! Temporary views no longer working for Share Compute
If I do this%sqlcreate or replace temporary view myviewasselect * from silver.<schema>.<table>;SHOW VIEWS;select * from myview;It works. But if I do the same on a Shared Compute it fails with[TABLE_OR_VIEW_NOT_FOUND] The table or view `myview` cannot...
- 3453 Views
- 5 replies
- 0 kudos
- 12483 Views
- 6 replies
- 2 kudos
EDU discount for university research project
I need databricks for a university research project. Is there any possibility of EDU discounts on DBU? So far I was unable to reach out to Databricks sales. Can you connect me with someone from DB?
- 12483 Views
- 6 replies
- 2 kudos
- 6794 Views
- 8 replies
- 0 kudos
Databricks Asset Bundle Workflow Redeployment Issue
Hello All,In my Databricks workflows, I have three tasks configured, with the final task set to run only if the condition "ALL_DONE" is met. During the first deployment, I observed that the dependency "ALL_DONE" was correctly assigned to the last tas...
- 6794 Views
- 8 replies
- 0 kudos
- 0 kudos
@Sikki thanks for confirming. For Azure DevOps please check the version of Databricks CLI installed.
- 0 kudos
- 2112 Views
- 2 replies
- 0 kudos
DownLoad CSV files from Delta Lake
We have around 1800 tables in Parq format (Delta Lake). These 1800 tables are very big, we have all these 1800 tables are converted into tables. But we have a requirement that, we need to download in CSV. (from PowerBI / any other reporting tool). Cu...
- 2112 Views
- 2 replies
- 0 kudos
- 0 kudos
Dear Kaniz,Thank you. one doubt 1) converting tables data into CSV and saving again one more of storage layer.IS there any way on fly we can convert these tables into CSV's. and export into PowerBI? and again i see in powerBI has limitations around >...
- 0 kudos
- 7776 Views
- 2 replies
- 1 kudos
The star inside WHERE
The star (*) can be used inside the WHERE clause in #Databricks as of runtime version 15.
- 7776 Views
- 2 replies
- 1 kudos
- 4632 Views
- 3 replies
- 1 kudos
Filter not using partition
I have the following code:spark.sparkContext.setCheckpointDir("dbfs:/mnt/lifestrategy-blob/checkpoints") result_df.repartitionByRange(200, "IdStation") result_df_checked = result_df.checkpoint(eager=True) unique_stations = result_df.select("IdStation...
- 4632 Views
- 3 replies
- 1 kudos
- 1 kudos
it seems like there is a filter being apply according to this. Filter (isnotnull(IdStation#2678) AND (IdStation#2678 = 1119844)) I would like to share the following notebook that covers in detail this topic, in case you would like to check it out h...
- 1 kudos
- 7195 Views
- 1 replies
- 0 kudos
RocksDB results in empty stream/stream joins dataframe
Since we enable RocksDB in our spark.conf the stream to stream joins/unions results in empty dataframe, does anyone else have the same experience? it is on AWSspark.conf.set("spark.sql.streaming.stateStore.providerClass","com.databricks.sql.streaming...
- 7195 Views
- 1 replies
- 0 kudos
- 0 kudos
Did you also update the checkpoint? You might need to use a new checkpoint after you enable the RocksDB state store.
- 0 kudos
- 3812 Views
- 5 replies
- 2 kudos
Trying to run databricks academy labs, but execution fails due to method to clearcache not whilelist
Hi there,Im trying to run DE 2.1 - Querying Files Directly on my workspace with a default cluster configuration for found below,but I cannot seem to run this file (or any other labs) as it gives me this error message Resetting the learning environme...
- 3812 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Retired_mod and databricks team,Did you already found some other solution for this? Thanks,Bram
- 2 kudos
- 1233 Views
- 0 replies
- 0 kudos
Chaining window aggregations in SQL
In my SQL data transformation pipeline, I'm doing chained/cascading window aggregations: for example, I want to do average over the last 5 minutes, then compute average over the past day on top of the 5 minute average, so that my aggregations are mor...
- 1233 Views
- 0 replies
- 0 kudos
- 1363 Views
- 0 replies
- 0 kudos
users are deleted/ unsynced from azure AD to databricks
In azure AD, it's shows users are synced to Databricks. But in Databricks, it's showing users is not a part of the group. The user is not part of only one group , he is part of remaining groups. All the syncing works fine till yesterday. I don't now ...
- 1363 Views
- 0 replies
- 0 kudos
- 1917 Views
- 2 replies
- 0 kudos
Delta Live table getting error of garbage collection after running few days
Hi, i am using delta live table in continuous mode for a real time streaming data pipeline. After running the pipeline like 2-3 days i am getting this garbage collection error:Driver/10.15.0.73 paused the JVM process 68 seconds during the past 120 se...
- 1917 Views
- 2 replies
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 518 | |
| 349 | |
| 287 |
