- 17117 Views
- 6 replies
- 2 kudos
Resolved! MergeSchema Not Working
Hi All, I have a scenario where my Exisiting Delta Table looks like below:Now I have an incremental data with an additional column i.e. owner:Dataframe Name --> scdDFBelow is the code snippet to merge Incremental Dataframe to targetTable, but the new...
- 17117 Views
- 6 replies
- 2 kudos
- 2 kudos
In Databricks Runtime 15.2 and above, you can specify schema evolution in a merge statement using SQL or Delta table APIs:MERGE WITH SCHEMA EVOLUTION INTO targetUSING sourceON source.key = target.keyWHEN MATCHED THENUPDATE SET *WHEN NOT MATCHED THENI...
- 2 kudos
- 16203 Views
- 12 replies
- 10 kudos
Connecting SQL Server (on-premise) to Databricks via jdbc:sqlserver
Is it possible to connect to SQL Server on-premise (Not Azure) from Databricks?I tried to ping my virtualbox VM (with Windows Server 2022) from within Databricks and the request timed out.%sh ping 122.138.0.14This is what my connection might look l...
- 16203 Views
- 12 replies
- 10 kudos
- 10 kudos
I tried to connect to localhost sql server through databricks community edition, but it failed. I have created an IP rule on port 1433 allowed inbound connection from all public network, but still didn't connect. I tried locally using python its work...
- 10 kudos
- 7197 Views
- 9 replies
- 3 kudos
Resolved! MLFlow Serve Logging
When using Azure Databricks and serving a model, we have received requests to capture additional logging. In some instances, they would like to capture input and output or even some of the steps from a pipeline. Is there any way we can extend the lo...
- 7197 Views
- 9 replies
- 3 kudos
- 3 kudos
Another word from a Databricks employee:"""You can use the custom model approach but configuring it is painful. Plus you have ended every loggable model in the custom model. Another less intrusive solution would be to have a proxy server do the loggi...
- 3 kudos
- 2886 Views
- 2 replies
- 6 kudos
Getting error popup in databricks
when i migrated to new databricks workspace, I am getting error popup message continuously and also indentation what I changed it is getting changed to other value every with new login .
- 2886 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi I am facing this issue session expired pop-ups frequently every 3 minutes while I start working on databricks.
- 6 kudos
- 1316 Views
- 1 replies
- 0 kudos
Security Threats in Databricks for File Upload
Dear community, we are using the Azure Databricks service and wondering if uploading a file to the DBFS (or to a storage accessed directly from a notebook in Databricks) could be a potential security threat. Imagine you upload some files with 'malici...
- 1316 Views
- 1 replies
- 0 kudos
- 0 kudos
Uploading a file to the Databricks File System (DBFS) or accessing storage directly from a notebook in Azure Databricks could pose potential security risks if not managed properly. Here are some considerations:Sensitive Data Exposure: Uploading sensi...
- 0 kudos
- 13074 Views
- 5 replies
- 6 kudos
Resolved! Call a Stored Procedure in Azure Synapse with input and output Params
driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager connection = driver_manager.getConnection(mssql_url, mssql_user, mssql_pass) connection.prepareCall("EXEC sys.sp_tables").execute() connection.close()The above code works fine but however...
- 13074 Views
- 5 replies
- 6 kudos
- 6 kudos
This blog helped me with the output of the stored procedure: https://medium.com/@judy3.yang/how-to-run-sql-procedure-in-databricks-notebook-e28023555565
- 6 kudos
- 4769 Views
- 3 replies
- 0 kudos
Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise
I am trying to evaluate options for Monitoring and Alerting tools like New Relic, Datadog, Grafana with Databricks on Azure . No one supports when reached out to them. I would like to hear from the databricks team on the recommended tool / framework ...
- 4769 Views
- 3 replies
- 0 kudos
- 0 kudos
I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, makin...
- 0 kudos
- 7285 Views
- 4 replies
- 2 kudos
Hi All,I'm facing an issue with my Spark Streaming Job. It gets stuck in the "Stream Initializing" phase for more than 3 hours.Need your...
Hi All,I'm facing an issue with my Spark Streaming Job. It gets stuck in the "Stream Initializing" phase for more than 3 hours.Need your help here to understand what happens internally at the "Stream Initializing" phase of the Spark Streaming job tha...
- 7285 Views
- 4 replies
- 2 kudos
- 2 kudos
I'm facing the same issue when I try to run this example Create a monitor using the API | Databricks on AWS (Inference Lakehouse Monitor regression example notebook). any idea?
- 2 kudos
- 15165 Views
- 16 replies
- 20 kudos
Resolved! network security for DBFS storage account
In Azure Databricks the DBFS storage account is open to all networks. Changing that to use a private endpoint or minimizing access to selected networks is not allowed.Is there any way to add network security to this storage account? Alternatively, is...
- 15165 Views
- 16 replies
- 20 kudos
- 20 kudos
How can we secure the storage account in the managed resource group which holds the DBFS with restricted network access, since access from all networks is blocked by our Azure storage account policy?
- 20 kudos
- 13973 Views
- 1 replies
- 2 kudos
Universal Azure Credential Passthrough
At the moment, Azure Databricks has the feature to use AzureAD login for the workspace and create single user clusters with Azure Data Lake Storage credential passthrough. But this can only be used for Data Lake Storage.Is there already a way, or are...
- 13973 Views
- 1 replies
- 2 kudos
- 2 kudos
I have exactly the same issue. I have the need to call a protected API within a notebook but have no access to the current user's access token. I've had to resort to nasty workarounds involving installing and running the Azure CLI from within the not...
- 2 kudos
- 8085 Views
- 2 replies
- 1 kudos
Resolved! com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption.
Hi. I am trying to read from our Microsoft SQL Server from Azure Databricks via spark.read.jdbc() as described here: Query databases using JDBC - Azure Databricks | Microsoft Learn. The SQL Server is on an Azure VM in a virtual network peered with th...
- 8085 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kazer ,Even if I use a new table name, I get the same error. Do you have any suggestions?Thanks,
- 1 kudos
- 3014 Views
- 1 replies
- 6 kudos
I am getting an SSLError: HTTPSConnectionPool while making a call to https restapis from azure databricks I have tried to set a verify=false parameter in the call too.
response = requests.request("POST", url, verify=False, headers=headers, data=payload) SSLError: HTTPSConnectionPool(host='dcs.adobedc.net', port=443): Max retries exceeded with url: /collection/d99e6dfcffb0b5aeaec2cf76cd3bc2b9e9c414b0c74a528d13dd39...
- 3014 Views
- 1 replies
- 6 kudos
- 6 kudos
Any luck with this?
- 6 kudos
- 9522 Views
- 2 replies
- 1 kudos
How can I fix this error. ExecutionError: An error occurred while calling o392.mount: java.lang.NullPointerException
Hello, I'm trying to mount my Databricks to my Azure gen 2 data lake to read in data from the container, but I get an error when executing this line of code: dbutils.fs.mount( source = "abfss://resumes@choisysresume.dfs.core.windows.net/", mount_poin...
- 9522 Views
- 2 replies
- 1 kudos
- 1 kudos
checked it with my mount script and that is exactly the same except that I do not put a '/' after dfs.core.windows.netYou might wanna try that.Also, is Unity enabled? Because Unity does not allow mounts.
- 1 kudos
- 6984 Views
- 4 replies
- 0 kudos
Unable to save file in DBFS
I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :" No such file or directory: 'No such file or directory: '/dbfs...
- 6984 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi,after some exercise you need to aware folder create in dbutils.fs.mkdirs("/dbfs/tmp/myfolder") it's created in /dbfs/dbfs/tmp/myfolderif you want to access path to_csv("/dbfs/tmp/myfolder/mytest.csv") you should created with this script dbutils.fs...
- 0 kudos
- 13156 Views
- 5 replies
- 7 kudos
Resolved! on-behalf-of token creation (for SPN)
I am trying to create an on-behalf-token for and SPN on my Azure Databricks Premium instance. The response is a FEATURE_DISABLED error message ("On-behalf-of token creation for service principals is not enabled for this workspace"). How do I turn on ...
- 13156 Views
- 5 replies
- 7 kudos
- 7 kudos
There is no On-behalf-of token on Azure - just generate an AAD token for the Service Principal and use it to create PAT (make sure that SP has permission to use PATs). The easiest way of doing it is to use the new Databricks CLI that supports unified...
- 7 kudos