- 3323 Views
- 4 replies
- 3 kudos
I feel like I am going crazy with this. I have tested a data pipeline on my standard compute cluster. I am loading new files as batch from a Google Cloud Storage bucket. Autoloader works exactly as expected from my notebook on my compute cluster. The...
- 3323 Views
- 4 replies
- 3 kudos
Latest Reply
I found the issue. I describe the solution in the following SO post. https://stackoverflow.com/questions/76287095/databricks-autoloader-works-on-compute-cluster-but-does-not-work-within-a-task/76313794#76313794
3 More Replies
by
g96g
• New Contributor III
- 561 Views
- 1 replies
- 0 kudos
Im having a hard time to convert below function from SSMS to databricks function. Any help would be appreciated! CREATE FUNCTION [dbo].[MaxOf5Values] (@D1 [int],@D2 [int],@D3 [int],@D4 [int],@D5 [int]) RETURNS int
AS
BEGIN
DECLARE @Result int
...
- 561 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Givi Salu​ ,​Please refer to this link that will help you convert this function.
- 829 Views
- 1 replies
- 0 kudos
We are having the files that needs to be loaded into the delta tableNow we want to perform some transformation on the files and load that into the tableWhat we didCreate a Spark DF from that fileApply transformation on the DFCreate a temp view from t...
- 829 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Sravan Kumar Mohanraj​ ,Yes, you can use copy query in this case your temp_view will be source.For more info, please visit these links.
- 1192 Views
- 2 replies
- 3 kudos
I am trying to run some API calls to the account console. I tried with every syntax possible encoded and decoded to get the call successfully but it returns a "user not authenticated" error. But when I tried with the Account admin it worked. I need t...
- 1192 Views
- 2 replies
- 3 kudos
Latest Reply
Hi Venkat, that sounds like a good idea. Thanks
1 More Replies
- 300 Views
- 0 replies
- 1 kudos
Welcome to Chennai User Group Community! We are thrilled to have you join our vibrant and enthusiastic community of users! Whether you are a seasoned expert or a newcomer, this group is the perfect place for you to connect, learn, and grow alongside...
- 300 Views
- 0 replies
- 1 kudos
- 1615 Views
- 3 replies
- 4 kudos
I have scala function as below, i am unable to understand how to write a scala jar with the same, please find below code i have used Enforcing Column-Level Encryption - Databrick %scala import com.macasaet.fernet.{Key, StringValidator, Token}import o...
- 1615 Views
- 3 replies
- 4 kudos
Latest Reply
I had to finally create the jar using teh intellij and sbt iconfiguration on the same env. and then installed the jar in the cluster it worked
2 More Replies
- 1181 Views
- 2 replies
- 0 kudos
My Code:-- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT Key_ID, Distributor_ID, Customer_ID, Customer_Name, ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source table...
- 1181 Views
- 2 replies
- 0 kudos
Latest Reply
Hi @Prashant Joshi​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
1 More Replies
- 1756 Views
- 6 replies
- 6 kudos
https://www.databricks.com/notebooks/delta-lake-cdf.htmlI am trying to understand the above article. Could someone explain to be the below questions?a) From SELECT * FROM table_changes('gold_consensus_eps', 2)why is consensus_eps values of 2.1 and 2....
- 1756 Views
- 6 replies
- 6 kudos
Latest Reply
Hi @THIAM HUAT TAN​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
5 More Replies
- 1020 Views
- 3 replies
- 1 kudos
Hi there, is it possible to change the sender of an Alert to appear as a company email (abcd@companyname.com) instead of Databricks and the noreply@databricks.com?We're in the process of rebuilding some alerts in Databricks as we're transitioning fro...
- 1020 Views
- 3 replies
- 1 kudos
Latest Reply
Hi @Cooper Chung​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
2 More Replies
- 1048 Views
- 3 replies
- 2 kudos
Is there a way to perform Z-ordering using python? With sql you you should be able to use:%sql
OPTIMIZE df
ZORDER BY (column)however I get the error "Table or view 'df' not found in database 'default''" and since I'm not really using sql, I would lik...
- 1048 Views
- 3 replies
- 2 kudos
Latest Reply
Hi @Lukas Goldschmied​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
2 More Replies
- 5761 Views
- 7 replies
- 1 kudos
Please explain with some use cases which show the difference between DLT and dbt.
- 5761 Views
- 7 replies
- 1 kudos
Latest Reply
Hi @Prachi Sankhala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
6 More Replies
- 2189 Views
- 3 replies
- 0 kudos
Hi,I set up an alert and apparently it worked because status changes according to the Trigger condition. However I don't received any notification on my email, or Teams channel. Any idea what is happening?Thanks
- 2189 Views
- 3 replies
- 0 kudos
Latest Reply
Hi @uriel chombo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
2 More Replies
- 1937 Views
- 4 replies
- 3 kudos
We run a dbx job that uses a jdbc connection to three (SQL Server) on-prem databases. Very occasionally a connection will go to the wrong server.In the job log we see the error "com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '...
- 1937 Views
- 4 replies
- 3 kudos
Latest Reply
Hi @Michel Smits​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
3 More Replies
- 6265 Views
- 3 replies
- 0 kudos
Should we create separate workspaces for Dev/Test/Prod ? Or should we have 1 workspace and create separate folders for Dev/Test/Prod?
- 6265 Views
- 3 replies
- 0 kudos
Latest Reply
as per my previous experience, its always good to have different workspaces for different envs, its easy to maintain and helps better with CICD pipeline as well, because lot of organizations provide deployment access to Developers in Dev env but not ...
2 More Replies
