by
Phani1
• Valued Contributor
- 36 Views
- 1 replies
- 0 kudos
Hi Team,Is there a particular reason why we should avoid using UDF and instead convert to DataFrame code?Are there any restrictions or limitations (in terms of performance or governance) when using UDFs in Databricks? Regards,Janga
- 36 Views
- 1 replies
- 0 kudos
Latest Reply
Hello some of the things you need to take in consideration is that:UDFs might introduce significant processing bottlenecks into code execution. Databricks uses a number of different optimizers automatically for code written with included Apache Spark...
by
ande
• New Contributor
- 50 Views
- 1 replies
- 0 kudos
I am trying to pull in data to my Databricks workspace via an external SFTP server. I am using Azure for my compute. To access the SFTP server they need to whitelist my IP address. My IP address in Azure Databricks seems to be constantly changing fro...
- 50 Views
- 1 replies
- 0 kudos
Latest Reply
Azure Databricks, like many cloud services, does not provide static IP addresses for outbound connections. This is because the compute resources are dynamically allocated and can change over time.
One potential workaround could be to use a Virtual N...
- 70 Views
- 1 replies
- 0 kudos
I have set up a volume in unity catalog in the format catalog/schema/volume, and granted all permissions to all users on the catalog, schema and volume.From the notebook I can see the /Volumes directory in the root of the file system but am unable to...
- 70 Views
- 1 replies
- 0 kudos
Latest Reply
Ensure that you have the necessary privileges on the catalog, schema, and volume, to access a volume, you must have the USE CATALOG privilege on the Volume’s parent catalog and the USE SCHEMA privilege on its parent schema.
If you're trying to cr...
- 83 Views
- 1 replies
- 0 kudos
I am getting a connect timed out error when attempting to access a sql server. I can successfully ping the server from Databricks. I have used the jdbc connection and the sqlserver included driver and both result in the same error. I have also attemp...
- 83 Views
- 1 replies
- 0 kudos
Latest Reply
Can you run the following command in a notebook using the same cluster you are using to connect:%sh
nc -vz <hostname> <port>
This test will confirm us if we are able to communicate with the SQL server by using the port you are defining to connect. If...
- 29 Views
- 0 replies
- 0 kudos
I was exploring on unity catalog option on Databricks premium workspace.I understood that i need to create storage account credentials and external connection in workspace.Later, i can access the cloud data using 'abfss://storage_account_details' .I ...
- 29 Views
- 0 replies
- 0 kudos
- 41 Views
- 0 replies
- 0 kudos
Fixit Cream is a topical medication used to relieve pain in the joints. It is not surprising that the medication is meant to treat the symptoms and indicators of degenerative conditions including osteochondrosis and . It is helpful to reduce inflamm...
- 41 Views
- 0 replies
- 0 kudos
- 42 Views
- 0 replies
- 0 kudos
There are multiple tables in the config/metadata table. These tables need to bevalidated for DQ rules.1.Natural Key / Business Key /Primary Key cannot be null orblank.2.Natural Key/Primary Key cannot be duplicate.3.Join columns missing values4.Busine...
- 42 Views
- 0 replies
- 0 kudos
- 91 Views
- 1 replies
- 0 kudos
Hello All,In my Databricks workflows, I have three tasks configured, with the final task set to run only if the condition "ALL_DONE" is met. During the first deployment, I observed that the dependency "ALL_DONE" was correctly assigned to the last tas...
- 91 Views
- 1 replies
- 0 kudos
Latest Reply
Hi, I've attached two snapshots, one from the initial deployment and another from the second redeployment, where no changes were made to the code.
- 17705 Views
- 2 replies
- 3 kudos
The difference between Global and Temp is how the lifetime of the view is tied to the application:http://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.createOrReplaceTempView.html?highlight=createorreplacetempview#pyspar...
- 17705 Views
- 2 replies
- 3 kudos
Latest Reply
Correct A Temp View is scoped to the SparkSession and dropped when that session closes. Each notebook runs in its own SparkSession. The Global Temp View is scoped to the cluster and dropped when the cluster re-starts or you drop it.
---------------...
1 More Replies
- 58 Views
- 0 replies
- 0 kudos
I have a notebook in Azure Databricks that does some transformations on a bronze tier table and inserts the transformed data into a silver tier table. This notebook is used to do an initial load of the data from our existing system into our new datal...
- 58 Views
- 0 replies
- 0 kudos
by
sp1
• New Contributor II
- 8323 Views
- 7 replies
- 4 kudos
I want to pass yesterday date (In the example 20230115*.csv) in the csv file. Don't know how to create parameter and use it here.CREATE OR REPLACE TEMPORARY VIEW abc_delivery_logUSING CSVOPTIONS ( header="true", delimiter=",", inferSchema="true", pat...
- 8323 Views
- 7 replies
- 4 kudos
Latest Reply
@Kaniz @sp1 @Chaitanya_Raju @daniel_sahal Hi Everyone,I need the same scenario on SQL code, because my DBR cluster not allowed me to run python codeError: Unsupported cell during execution. SQL warehouses only support executing SQL cells.I appreciate...
6 More Replies
- 37 Views
- 0 replies
- 0 kudos
Hi, I am loading data from a kinesis data stream using DLT. CREATE STREAMING TABLE Consumers_kinesis_2
(
...,
unbase64(data) String,
...
)
AS
SELECT * FROM STREAM read_kinesis (...) Is it possible to directly cast, unbase64, and/or transform the resu...
- 37 Views
- 0 replies
- 0 kudos
- 730 Views
- 2 replies
- 1 kudos
Hi,I am having an issue of loading source data into a delta table/ unity catalog. The error we are recieving is the following:grpc_message:"[DELTA_EXCEED_CHAR_VARCHAR_LIMIT] Exceeds char/varchar type length limitation. Failed check: (isnull(\'metric_...
- 730 Views
- 2 replies
- 1 kudos
Latest Reply
Hey @Paul92S Looking at the error message it looks like column "metric_name" is the culprit here:Understanding the Error:Character Limit Violation: The error indicates that values in the metric_name column are exceeding the maximum length allowed fo...
1 More Replies
- 36 Views
- 0 replies
- 0 kudos
Hello,I have a Job A that runs a Job B, and Job A defines a globalTempView and I would like to somehow access it in the child job. Is that in anyway possible ? Can the same cluster be used for both jobs ? If it is not possible, does someone know of a...
- 36 Views
- 0 replies
- 0 kudos
- 32 Views
- 0 replies
- 0 kudos
Today, the entire private limited company registration process and other regulatory filings are paperless; documents are filed electronically through the MCA website and is processed at the Central Registration Centre (CRC). The Online Private Limite...
- 32 Views
- 0 replies
- 0 kudos
