cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

User16826994223
by Honored Contributor III
  • 1633 Views
  • 1 replies
  • 0 kudos
  • 1633 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

ou must explicitly enable the change data feed option using one of the following methods:New table: Set the table property  delta.enableChangeDataFeed = true in the CREATE TABLE command.CREATE TABLE student (id INT, name STRING, age INT) TBLPROPERTIE...

  • 0 kudos
User16826994223
by Honored Contributor III
  • 1266 Views
  • 1 replies
  • 1 kudos

Resolved! prerequisite for SCIM provisioning

Hi Team Want to know what is the prerequisite for provisioning SCIM provisisoning in Azure

  • 1266 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 1 kudos

Your Azure Databricks account must have the Azure Databricks Premium Plan.Your Azure Active Directory account must be a Premium edition account.You must be a global administrator for the Azure Active Directory account.

  • 1 kudos
User16826994223
by Honored Contributor III
  • 1031 Views
  • 1 replies
  • 0 kudos
  • 1031 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

Design to honor API and other limits of the platform.• Max API calls/ hr = 1500• Jobs per hour per workspace = 1000• Maximum concurrent Notebooks per cluster = 145

  • 0 kudos
User16826994223
by Honored Contributor III
  • 1324 Views
  • 1 replies
  • 1 kudos

Resolved! Can we convert non NPIP to NPIP works space

Hi I have a cluster which I created as NON NPIP initially on Azure , But I want to change it To NPIP , I am searching on portal to change but I am not seeing any optopn to change, is it possible or do I need to create a Complete different workspa...

  • 1324 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 1 kudos

if you deployed a workspace with NPIP unchecked, you can't convert this to a NPIP workspace (unless something has changed that I was not aware of since a month ago). You'll have to redeploy the workspace with NPIP. NPIP should be a default for all de...

  • 1 kudos
User16826994223
by Honored Contributor III
  • 1096 Views
  • 1 replies
  • 0 kudos

Is this trigger supported -- trigger(Trigger.Continuous("1 second"))

Does delta file format support the continous trigger streaming as a sink?.trigger(Trigger.Continuous("1 second"))Can't find a document around itIn the spark documentation, I could see that below mentioned sinks are supported:Sinks:Kafka sink: All opt...

  • 1096 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

Nope This is not supported with delta file format

  • 0 kudos
User16826994223
by Honored Contributor III
  • 1030 Views
  • 1 replies
  • 1 kudos

Resolved! If multiple streams as jar is running and one fail will it fails others also

Hi Team, IHAC who is running multiple stream in the jar based job, one of the streams got terminated, but other streams are processing without termination.Is this know behaviour in case of jar based streaming application? Any insight please? (edited)

  • 1030 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 1 kudos

Failure in any of the active streaming queries causes the active run to fail and terminate all the other streaming queries. You do not need to use  streamingQuery.awaitTermination()or spark.streams.awaitAnyTermination() at the end of your notebook. ...

  • 1 kudos
User16826994223
by Honored Contributor III
  • 1129 Views
  • 1 replies
  • 0 kudos

Resolved! new input sources to a streaming job while retaining the old checkpoint ?

Hello Team, is there any way to add new input sources to a streaming job while retaining the old checkpoint ? I have seen - Changes in the number or type (i.e. different source) of input sources: This is not allowed . Want to find out if there is any...

  • 1129 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

No you cannot do this, but often you can start a new checkpoint that starts from the same offsets the old one left off at: https://docs.databricks.com/delta/delta-streaming.html#specify-initial-position

  • 0 kudos
User16826994223
by Honored Contributor III
  • 920 Views
  • 1 replies
  • 0 kudos

For auto-loader, do we pause fetching the messages from SQS

Hi Team, For auto-loader, do we pause fetching the messages from SQS. Apple restarted their streaming workload and found no new messages are fetched or deleted. Taking the difference of the latest sequence processed and the latest sequence available ...

  • 920 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

Yes, this is something that is fixed in DBR 8.3

  • 0 kudos
User16826994223
by Honored Contributor III
  • 981 Views
  • 1 replies
  • 0 kudos

How to provide access to user bases on the login credentials databricks

Hi Team I am trying to do security audit and its become tough to manage so many credentials and IAM role we have in databricks Different clusters, Is it possible that I simplify it , like a user who has type of access in s3 bucket get same type of...

  • 981 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

This is a great question and Databricks is working continuously working on management of security , to make user experience better and simple.The use case you are trying to solve will be easily solved using high concurrency cluster and checkin...

  • 0 kudos
User16826994223
by Honored Contributor III
  • 862 Views
  • 1 replies
  • 1 kudos
  • 862 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 1 kudos

Access control: Rich suite of access control all the way down to the storage layer. Databricks can take advantage of its cloud backbone by utilizing state-of-the-art AWS security services right in the platform. Federate your existing AWS data access ...

  • 1 kudos
User16826994223
by Honored Contributor III
  • 1103 Views
  • 1 replies
  • 0 kudos

spark is reading data from source even I am persisting the data

hI allI am reading data and I am caching the data and then I am performing Action Count to get the data in memory, but still, in dag I found out that every time it reads data from SOURCE.

  • 1103 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

It looks like the the spark memory is not sufficient to cache all the data so it read always from source

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels