cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

pSdatabricks
by New Contributor II
  • 1977 Views
  • 3 replies
  • 0 kudos

Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise

I am trying to evaluate options for Monitoring and Alerting tools like New Relic, Datadog, Grafana with Databricks on Azure . No one supports when reached out to them. I would like to hear from the databricks team on the recommended tool / framework ...

  • 1977 Views
  • 3 replies
  • 0 kudos
Latest Reply
Sruthivika
New Contributor II
  • 0 kudos

I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, makin...

  • 0 kudos
2 More Replies
SamarthJain
by New Contributor II
  • 2572 Views
  • 4 replies
  • 2 kudos

Hi All,I'm facing an issue with my Spark Streaming Job. It gets stuck in the "Stream Initializing" phase for more than 3 hours.Need your...

Hi All,I'm facing an issue with my Spark Streaming Job. It gets stuck in the "Stream Initializing" phase for more than 3 hours.Need your help here to understand what happens internally at the "Stream Initializing" phase of the Spark Streaming job tha...

  • 2572 Views
  • 4 replies
  • 2 kudos
Latest Reply
MohsenJ
New Contributor III
  • 2 kudos

I'm facing the same issue when I try to run this example Create a monitor using the API | Databricks on AWS (Inference Lakehouse Monitor regression example notebook). any idea? 

  • 2 kudos
3 More Replies
Bas1
by New Contributor III
  • 5552 Views
  • 17 replies
  • 20 kudos

Resolved! network security for DBFS storage account

In Azure Databricks the DBFS storage account is open to all networks. Changing that to use a private endpoint or minimizing access to selected networks is not allowed.Is there any way to add network security to this storage account? Alternatively, is...

  • 5552 Views
  • 17 replies
  • 20 kudos
Latest Reply
Odee79
New Contributor II
  • 20 kudos

How can we secure the storage account in the managed resource group which holds the DBFS with restricted network access, since access from all networks is blocked by our Azure storage account policy?

  • 20 kudos
16 More Replies
verargulla
by New Contributor III
  • 6180 Views
  • 3 replies
  • 4 kudos

Azure Databricks: Error Creating Cluster

We have provisioned a new workspace in Azure using our own VNet. Upon creating the first cluster, I encounter this error:Control Plane Request Failure: Failed to get instance bootstrap steps from the Databricks Control Plane. Please check that instan...

  • 6180 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Ver Argulla​ Hope everything is going great.Does @Kaniz Fatma​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 4 kudos
2 More Replies
AlexWeh
by New Contributor II
  • 1725 Views
  • 1 replies
  • 2 kudos

Universal Azure Credential Passthrough

At the moment, Azure Databricks has the feature to use AzureAD login for the workspace and create single user clusters with Azure Data Lake Storage credential passthrough. But this can only be used for Data Lake Storage.Is there already a way, or are...

  • 1725 Views
  • 1 replies
  • 2 kudos
Latest Reply
polivbr
New Contributor II
  • 2 kudos

I have exactly the same issue. I have the need to call a protected API within a notebook but have no access to the current user's access token. I've had to resort to nasty workarounds involving installing and running the Azure CLI from within the not...

  • 2 kudos
Kazer
by New Contributor III
  • 2935 Views
  • 2 replies
  • 1 kudos

Resolved! com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption.

Hi. I am trying to read from our Microsoft SQL Server from Azure Databricks via spark.read.jdbc() as described here: Query databases using JDBC - Azure Databricks | Microsoft Learn. The SQL Server is on an Azure VM in a virtual network peered with th...

  • 2935 Views
  • 2 replies
  • 1 kudos
Latest Reply
databricks26
New Contributor II
  • 1 kudos

Hi @Kazer ,Even if I use a new table name, I get the same error. Do you have any suggestions?Thanks,

  • 1 kudos
1 More Replies
mudholkar
by New Contributor III
  • 1373 Views
  • 1 replies
  • 6 kudos

I am getting an SSLError: HTTPSConnectionPool while making a call to https restapis from azure databricks I have tried to set a verify=false parameter in the call too.

response = requests.request("POST", url, verify=False, headers=headers, data=payload)   SSLError: HTTPSConnectionPool(host='dcs.adobedc.net', port=443): Max retries exceeded with url: /collection/d99e6dfcffb0b5aeaec2cf76cd3bc2b9e9c414b0c74a528d13dd39...

  • 1373 Views
  • 1 replies
  • 6 kudos
Latest Reply
JFG
New Contributor II
  • 6 kudos

Any luck with this?

  • 6 kudos
Anotech
by New Contributor II
  • 3110 Views
  • 2 replies
  • 1 kudos

How can I fix this error. ExecutionError: An error occurred while calling o392.mount: java.lang.NullPointerException

Hello, I'm trying to mount my Databricks to my Azure gen 2 data lake to read in data from the container, but I get an error when executing this line of code: dbutils.fs.mount( source = "abfss://resumes@choisysresume.dfs.core.windows.net/", mount_poin...

  • 3110 Views
  • 2 replies
  • 1 kudos
Latest Reply
WernerS
New Contributor III
  • 1 kudos

checked it with my mount script and that is exactly the same except that I do not put a '/' after dfs.core.windows.netYou might wanna try that.Also, is Unity enabled?  Because Unity does not allow mounts.

  • 1 kudos
1 More Replies
learnerbricks
by New Contributor II
  • 3219 Views
  • 4 replies
  • 0 kudos

Unable to save file in DBFS

I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :" No such file or directory: 'No such file or directory: '/dbfs...

  • 3219 Views
  • 4 replies
  • 0 kudos
Latest Reply
pardosa
New Contributor II
  • 0 kudos

Hi,after some exercise you need to aware folder create in dbutils.fs.mkdirs("/dbfs/tmp/myfolder") it's created in /dbfs/dbfs/tmp/myfolderif you want to access path to_csv("/dbfs/tmp/myfolder/mytest.csv") you should created with this script dbutils.fs...

  • 0 kudos
3 More Replies
Akshith_Rajesh
by New Contributor III
  • 7200 Views
  • 4 replies
  • 5 kudos

Resolved! Call a Stored Procedure in Azure Synapse with input and output Params

driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager connection = driver_manager.getConnection(mssql_url, mssql_user, mssql_pass) connection.prepareCall("EXEC sys.sp_tables").execute() connection.close()The above code works fine but however...

  • 7200 Views
  • 4 replies
  • 5 kudos
Latest Reply
sivaram_bh
New Contributor II
  • 5 kudos

statement="EXEC procedurename  imputparametre , ? "driver_manager = spark._sc._gateway.jvm.java.sql.DriverManagercon = driver_manager.getConnection(jdbcUrl, username, pwd)exec_statement = con.prepareCall(statement)exec_statement.registerOutParameter(...

  • 5 kudos
3 More Replies
DJey
by New Contributor III
  • 4025 Views
  • 5 replies
  • 2 kudos

Resolved! MergeSchema Not Working

Hi All, I have a scenario where my Exisiting Delta Table looks like below:Now I have an incremental data with an additional column i.e. owner:Dataframe Name --> scdDFBelow is the code snippet to merge Incremental Dataframe to targetTable, but the new...

image image image image
  • 4025 Views
  • 5 replies
  • 2 kudos
Latest Reply
DJey
New Contributor III
  • 2 kudos

@Vidula Khanna​  Enabling the below property resolved my issue:spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled",True) Thanks v much!

  • 2 kudos
4 More Replies
clapton79
by New Contributor II
  • 4872 Views
  • 5 replies
  • 5 kudos

Resolved! on-behalf-of token creation (for SPN)

I am trying to create an on-behalf-token for and SPN on my Azure Databricks Premium instance. The response is a FEATURE_DISABLED error message ("On-behalf-of token creation for service principals is not enabled for this workspace"). How do I turn on ...

  • 4872 Views
  • 5 replies
  • 5 kudos
Latest Reply
alexott
Valued Contributor II
  • 5 kudos

There is no On-behalf-of token on Azure - just generate an AAD token for the Service Principal and use it to create PAT (make sure that SP has permission to use PATs). The easiest way of doing it is to use the new Databricks CLI that supports unified...

  • 5 kudos
4 More Replies
lstk
by New Contributor
  • 1252 Views
  • 2 replies
  • 1 kudos

Resolved! Job ID value out of range - Azure Logic App Connector

Hello everybody,i tried to build a Logic App Custom Connector following this one explanation. (https://medium.com/@poojaanilshinde/create-azure-logic-apps-custom-connector-for-azure-databricks-e51f4524ab27)Now i run in the following Problem and wante...

image.png
  • 1252 Views
  • 2 replies
  • 1 kudos
Latest Reply
stefnhuy
New Contributor III
  • 1 kudos

Hey Lukas,I can totally relate to the frustration of encountering those confounding errors when building custom connectors in Azure Logic Apps. The "Job ID value out of range" issue can be quite perplexing, but fear not, for there's a solution on the...

  • 1 kudos
1 More Replies
UmaMahesh1
by Honored Contributor III
  • 3194 Views
  • 8 replies
  • 17 kudos

Spark Structured Streaming : Data write is too slow into adls.

I'm a bit new to spark structured streaming stuff so do ask all the relevant questions if I missed any.I have a notebook which consumes the events from a kafka topic and writes those records into adls. The topic is json serialized so I'm just writing...

  • 3194 Views
  • 8 replies
  • 17 kudos
Latest Reply
Kaniz
Community Manager
  • 17 kudos

Hi @UmaMahesh1 ,  • Spark Structured Streaming interacts with Kafka in a certain way, leading to the observed behaviour.• The parameter maxOffsetsPerTrigger in Spark Structured Streaming determines the maximum rate of data read from Kafka.• However, ...

  • 17 kudos
7 More Replies
FerArribas
by Contributor
  • 1755 Views
  • 3 replies
  • 0 kudos

Resolved! Azure Databricks - Difference between protecting the WEB UI with IP Access list or disabling public access?

Hi, Thoroughly investigating the best security practices for accessing the Databricks WEB UI. I have doubts about the difference between protecting the WEB UI with (1) IP Access list (https://learn.microsoft.com/en-us/azure/databricks/security/networ...

  • 1755 Views
  • 3 replies
  • 0 kudos
Latest Reply
Rik
New Contributor III
  • 0 kudos

"In short, would it be the same to configure only the IP of the private endpoint in the IP access list vs disable public access?"The access list doesn't apply to private IPs, only to public IP (internet). Relevant part from the docs:"If you use Priva...

  • 0 kudos
2 More Replies
Labels