- 1194 Views
- 0 replies
- 0 kudos
- 1194 Views
- 0 replies
- 0 kudos
Forum Posts
- 1561 Views
- 1 replies
- 0 kudos
Is this trigger supported -- trigger(Trigger.Continuous("1 second"))
Does delta file format support the continous trigger streaming as a sink?.trigger(Trigger.Continuous("1 second"))Can't find a document around itIn the spark documentation, I could see that below mentioned sinks are supported:Sinks:Kafka sink: All opt...
- 1561 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Nope This is not supported with delta file format
- 0 kudos
- 1598 Views
- 1 replies
- 1 kudos
Resolved! If multiple streams as jar is running and one fail will it fails others also
Hi Team, IHAC who is running multiple stream in the jar based job, one of the streams got terminated, but other streams are processing without termination.Is this know behaviour in case of jar based streaming application? Any insight please? (edited)
- 1598 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Failure in any of the active streaming queries causes the active run to fail and terminate all the other streaming queries. You do not need to use streamingQuery.awaitTermination()or spark.streams.awaitAnyTermination() at the end of your notebook. ...
- 1 kudos
- 1672 Views
- 1 replies
- 0 kudos
Resolved! new input sources to a streaming job while retaining the old checkpoint ?
Hello Team, is there any way to add new input sources to a streaming job while retaining the old checkpoint ? I have seen - Changes in the number or type (i.e. different source) of input sources: This is not allowed . Want to find out if there is any...
- 1672 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
No you cannot do this, but often you can start a new checkpoint that starts from the same offsets the old one left off at: https://docs.databricks.com/delta/delta-streaming.html#specify-initial-position
- 0 kudos
- 1314 Views
- 1 replies
- 0 kudos
For auto-loader, do we pause fetching the messages from SQS
Hi Team, For auto-loader, do we pause fetching the messages from SQS. Apple restarted their streaming workload and found no new messages are fetched or deleted. Taking the difference of the latest sequence processed and the latest sequence available ...
- 1314 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Yes, this is something that is fixed in DBR 8.3
- 0 kudos
- 2086 Views
- 0 replies
- 1 kudos
Hi streaming team, do we have any good integration for Azure Service Bus?
Hi streaming team, do we have any good integration for Azure Service Bus?
- 2086 Views
- 0 replies
- 1 kudos
- 1322 Views
- 1 replies
- 0 kudos
How to provide access to user bases on the login credentials databricks
Hi Team I am trying to do security audit and its become tough to manage so many credentials and IAM role we have in databricks Different clusters, Is it possible that I simplify it , like a user who has type of access in s3 bucket get same type of...
- 1322 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
This is a great question and Databricks is working continuously working on management of security , to make user experience better and simple.The use case you are trying to solve will be easily solved using high concurrency cluster and checkin...
- 0 kudos
- 1347 Views
- 1 replies
- 1 kudos
- 1347 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Access control: Rich suite of access control all the way down to the storage layer. Databricks can take advantage of its cloud backbone by utilizing state-of-the-art AWS security services right in the platform. Federate your existing AWS data access ...
- 1 kudos
- 1779 Views
- 1 replies
- 0 kudos
spark is reading data from source even I am persisting the data
hI allI am reading data and I am caching the data and then I am performing Action Count to get the data in memory, but still, in dag I found out that every time it reads data from SOURCE.
- 1779 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
It looks like the the spark memory is not sufficient to cache all the data so it read always from source
- 0 kudos
- 4025 Views
- 1 replies
- 0 kudos
Resolved! Change in spark code if I migrate from spark 2.4 to 3.0
I am thinking of migrating the spark 2.4 to 3.0, what should I know to take care of changes thet I need to look at while migrating
- 4025 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
I see there are many changes that need to take care of if you have used coding in spark 2.4 the changes are in Data set API StatementBuiltin UDF and functionsMore you can get from spark Documentation https://spark.apache.org/docs/latest/sql-migrat...
- 0 kudos
- 1792 Views
- 1 replies
- 0 kudos
What type of aws instance and how many are used for an L sized Databricks SQL(SQLA) cluster ?
What type of aws instance and how many are used for an L sized Databricks SQL(SQLA) cluster with Photon enabled
- 1792 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
L size is 16 workers of i3.8xlarge
- 0 kudos
- 2205 Views
- 1 replies
- 0 kudos
Refreshing external tables
After I vacuum the tables, do i need to update the manifest table and parquet table to refresh my external tables for integrations to work?
- 2205 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Manifest files need to be re-created when partitions are added or altered. Since a VACUUM only deletes all historical versions, you shouldn't need to create an updated manifest file unless you are also running an OPTIMIZE.
- 0 kudos
- 4239 Views
- 1 replies
- 1 kudos
- 4239 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
G1GC can solve problems in some cases where garbage collection is a bottleneck. checkout https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html
- 1 kudos
- 2510 Views
- 1 replies
- 0 kudos
- 2510 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
As of this comment, SQL analytics still requires a few additional enablement steps. You will need to ask your Databricks account team to help turn this on in your workspace.
- 0 kudos
- 2498 Views
- 1 replies
- 0 kudos
- 2498 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
You could potentially do this through a Global Init Script - https://docs.databricks.com/clusters/init-scripts.html
- 0 kudos
| User | Count |
|---|---|
| 1632 | |
| 791 | |
| 513 | |
| 349 | |
| 287 |
