cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Hal
by New Contributor II
  • 291 Views
  • 1 replies
  • 3 kudos

Connecting Power BI on Azure to Databricks on AWS?

Can someone share with me the proper way to connect Power BI running on Azure to Databricks running on AWS?

  • 291 Views
  • 1 replies
  • 3 kudos
Latest Reply
bhanadi
New Contributor II
  • 3 kudos

Have the same question. Do we have to take care of any specific tasks to make it work. Anyone who implemented it?

  • 3 kudos
Bas1
by New Contributor III
  • 4535 Views
  • 17 replies
  • 20 kudos

Resolved! network security for DBFS storage account

In Azure Databricks the DBFS storage account is open to all networks. Changing that to use a private endpoint or minimizing access to selected networks is not allowed.Is there any way to add network security to this storage account? Alternatively, is...

  • 4535 Views
  • 17 replies
  • 20 kudos
Latest Reply
Odee79
New Contributor II
  • 20 kudos

How can we secure the storage account in the managed resource group which holds the DBFS with restricted network access, since access from all networks is blocked by our Azure storage account policy?

  • 20 kudos
16 More Replies
Baldrez
by New Contributor II
  • 1940 Views
  • 4 replies
  • 5 kudos

Resolved! REST API for Stream Monitoring

Hi, everyone. I just recently started using Databricks on Azure so my question is probably very basic but I am really stuck right now.I need to capture some streaming metrics (number of input rows and their time) so I tried using the Spark Rest Api ...

  • 1940 Views
  • 4 replies
  • 5 kudos
Latest Reply
jose_gonzalez
Moderator
  • 5 kudos

hi @Roberto Baldrez​ ,if you think that @Gaurav Rupnar​ solved your question, then please select it as best response to it can be moved to the top of the topic and it will help more users in the future.Thank you

  • 5 kudos
3 More Replies
jwilliam
by Contributor
  • 1446 Views
  • 3 replies
  • 2 kudos

Resolved! How to mount Azure Blob Storage with OAuth2?

We already know that we can mount Azure Data Lake Gen2 with OAuth2 using this:configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", ...

  • 1446 Views
  • 3 replies
  • 2 kudos
Latest Reply
mathijs-fish
New Contributor III
  • 2 kudos

Is there any update on this feature request? OAuth still seems not to be working with Azure Blob Storage... Configuration works fine for ADLS gen 2, but for Azure Blob Storage still only SAS and Account key seems to be working.

  • 2 kudos
2 More Replies
joao_vnb
by New Contributor III
  • 14913 Views
  • 7 replies
  • 11 kudos

Resolved! Automate the Databricks workflow deployment

Hi everyone,Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?

  • 14913 Views
  • 7 replies
  • 11 kudos
Latest Reply
asingamaneni
New Contributor II
  • 11 kudos

Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflowYou can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic w...

  • 11 kudos
6 More Replies
ChrisS
by New Contributor III
  • 1269 Views
  • 7 replies
  • 8 kudos

How to get data scraped from the web into your data storage

I learning data bricks for the first time following the book that is copywrited in 2020 so I imagine it might be a little outdated at this point. What I am trying to do is move data from an online source (in this specific case using shell script but ...

  • 1269 Views
  • 7 replies
  • 8 kudos
Latest Reply
CharlesReily
New Contributor III
  • 8 kudos

In Databricks, you can install external libraries by going to the Clusters tab, selecting your cluster, and then adding the Maven coordinates for Deequ. This represents the best b2b data enrichment services in Databricks.In your notebook or script, y...

  • 8 kudos
6 More Replies
MartinH
by New Contributor II
  • 1647 Views
  • 6 replies
  • 3 kudos

Azure Data Factory and Photon

Hello, we have Databricks Python workbooks accessing Delta tables. These workbooks are scheduled/invoked by Azure Data Factory. How can I enable Photon on the linked services that are used to call Databricks?If I specify new job cluster, there does n...

  • 1647 Views
  • 6 replies
  • 3 kudos
Latest Reply
CharlesReily
New Contributor III
  • 3 kudos

When you create a cluster on Databricks, you can enable Photon by selecting the "Photon" option in the cluster configuration settings. This is typically done when creating a new cluster, and you would find the option in the advanced cluster configura...

  • 3 kudos
5 More Replies
pgruetter
by Contributor
  • 2167 Views
  • 7 replies
  • 1 kudos

Run Task as Service Principal with Code in Azure DevOps Repo

Hi allI have a task of type Notebook, source is Git (Azure DevOps). This task runs fine with my user, but if I change the Owner to a service principal, I get the following error:Run result unavailable: run failed with error message Failed to checkout...

  • 2167 Views
  • 7 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@pgruetter​ :To enable a service principal to access a specific Azure DevOps repository, you need to grant it the necessary permissions at both the organization and repository levels.Here are the steps to grant the service principal the necessary per...

  • 1 kudos
6 More Replies
AlexWeh
by New Contributor II
  • 1098 Views
  • 1 replies
  • 2 kudos

Universal Azure Credential Passthrough

At the moment, Azure Databricks has the feature to use AzureAD login for the workspace and create single user clusters with Azure Data Lake Storage credential passthrough. But this can only be used for Data Lake Storage.Is there already a way, or are...

  • 1098 Views
  • 1 replies
  • 2 kudos
Latest Reply
polivbr
New Contributor II
  • 2 kudos

I have exactly the same issue. I have the need to call a protected API within a notebook but have no access to the current user's access token. I've had to resort to nasty workarounds involving installing and running the Azure CLI from within the not...

  • 2 kudos
prapot
by New Contributor II
  • 4400 Views
  • 2 replies
  • 2 kudos

Resolved! How to write a Spark DataFrame to CSV file with our .CRC in Azure Databricks?

val spark:SparkSession = SparkSession.builder() .master("local[3]") .appName("SparkByExamples.com") .getOrCreate()//Spark Read CSV Fileval df = spark.read.option("header",true).csv("address.csv")//Write DataFrame to address directorydf.write...

  • 4400 Views
  • 2 replies
  • 2 kudos
Latest Reply
Nw2this
New Contributor II
  • 2 kudos

Will your csv have the name prefix 'part-' or can you name it whatever you like?

  • 2 kudos
1 More Replies
Kazer
by New Contributor III
  • 2231 Views
  • 2 replies
  • 1 kudos

Resolved! com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption.

Hi. I am trying to read from our Microsoft SQL Server from Azure Databricks via spark.read.jdbc() as described here: Query databases using JDBC - Azure Databricks | Microsoft Learn. The SQL Server is on an Azure VM in a virtual network peered with th...

  • 2231 Views
  • 2 replies
  • 1 kudos
Latest Reply
databricks26
New Contributor II
  • 1 kudos

Hi @Kazer ,Even if I use a new table name, I get the same error. Do you have any suggestions?Thanks,

  • 1 kudos
1 More Replies
mudholkar
by New Contributor III
  • 1145 Views
  • 1 replies
  • 6 kudos

I am getting an SSLError: HTTPSConnectionPool while making a call to https restapis from azure databricks I have tried to set a verify=false parameter in the call too.

response = requests.request("POST", url, verify=False, headers=headers, data=payload)   SSLError: HTTPSConnectionPool(host='dcs.adobedc.net', port=443): Max retries exceeded with url: /collection/d99e6dfcffb0b5aeaec2cf76cd3bc2b9e9c414b0c74a528d13dd39...

  • 1145 Views
  • 1 replies
  • 6 kudos
Latest Reply
JFG
New Contributor II
  • 6 kudos

Any luck with this?

  • 6 kudos
Constantine
by Contributor III
  • 1970 Views
  • 4 replies
  • 1 kudos

Resolved! How to use Databricks Query History API (REST API)

I have setup authentication using this page https://docs.databricks.com/sql/api/authentication.html and run curl -n -X GET https://<databricks-instance>.cloud.databricks.com/api/2.0/sql/history/queriesTo get history of all sql endpoint queries, but I...

  • 1970 Views
  • 4 replies
  • 1 kudos
Latest Reply
MorpheusGoGo
New Contributor II
  • 1 kudos

Are you sure this works?payload = { "filter_by": {    }, "max_results": 1} Returns 1 result. payload = { "filter_by": {      "query_start_time_range":{       "start_time_ms" :1640995200000,        "end_time_ms" : 1641081599000   } }, "max_results": 1...

  • 1 kudos
3 More Replies
Anotech
by New Contributor II
  • 2480 Views
  • 2 replies
  • 1 kudos

How can I fix this error. ExecutionError: An error occurred while calling o392.mount: java.lang.NullPointerException

Hello, I'm trying to mount my Databricks to my Azure gen 2 data lake to read in data from the container, but I get an error when executing this line of code: dbutils.fs.mount( source = "abfss://resumes@choisysresume.dfs.core.windows.net/", mount_poin...

  • 2480 Views
  • 2 replies
  • 1 kudos
Latest Reply
WernerS
New Contributor III
  • 1 kudos

checked it with my mount script and that is exactly the same except that I do not put a '/' after dfs.core.windows.netYou might wanna try that.Also, is Unity enabled?  Because Unity does not allow mounts.

  • 1 kudos
1 More Replies
Akshith_Rajesh
by New Contributor III
  • 6762 Views
  • 4 replies
  • 5 kudos

Resolved! Call a Stored Procedure in Azure Synapse with input and output Params

driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager connection = driver_manager.getConnection(mssql_url, mssql_user, mssql_pass) connection.prepareCall("EXEC sys.sp_tables").execute() connection.close()The above code works fine but however...

  • 6762 Views
  • 4 replies
  • 5 kudos
Latest Reply
sivaram_bh
New Contributor II
  • 5 kudos

statement="EXEC procedurename  imputparametre , ? "driver_manager = spark._sc._gateway.jvm.java.sql.DriverManagercon = driver_manager.getConnection(jdbcUrl, username, pwd)exec_statement = con.prepareCall(statement)exec_statement.registerOutParameter(...

  • 5 kudos
3 More Replies
Labels