- 2192 Views
- 1 replies
- 1 kudos
how to execute the CHANGE_TRACKING snowflake query in databricks
please check the attached image which needs to resolve. anyone have come across this kind of issues
- 2192 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @sherbin w Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 13080 Views
- 3 replies
- 4 kudos
to_utc_timestamp is subtracting a different time delta and from_utc_timestamp is not adding the same delta.
My session timezone is Australia/Sydneyif i run the below query my expectation is first column and third column should show the same value. But it is not working as expected for 1753-01-01 00:00:00 timestamp.spark.conf.set("spark.sql.session.timeZone...
- 13080 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Rahul Lalwani (Customer),In Interactive cluster spark.sql.datetime.java8API.enabled is disabled when we enable spark.sql.datetime.java8API.enabled to true , we can see crt values for 1753-01-01 as well.The reason for enabling the above config ...
- 4 kudos
- 7603 Views
- 2 replies
- 1 kudos
Error "Invalid configuration value detected for fs.azure.account.key" when listing files stored in an Azure Storage account using "dbutils.fs.ls"
I get the following error when getting a list of files stored in an Azure Storage account using "dbutils.fs.ls" command in Databrciks.Failure to initialize configuration for storage account AAAAA.dfs.core.windows.net: Invalid configuration value dete...
- 7603 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Mohammad Saber Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 1015 Views
- 1 replies
- 0 kudos
Unable to configure compute of cloned job
If I clone an existing job without making any changes, I am able to reconfigure the compute successfully. If I remove or add a spark environment variable to test modifications, such as using secrets for example, and I confirm the changes to the job, ...
- 1015 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Marvin Ginns Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 4838 Views
- 3 replies
- 4 kudos
Can you use autoloader with a fixed width file?
I have a collection fixed width files that I would like to ingest monthly with autoloader but I can't seem to find an example. I can read the files into Dataframes using a python function to map the index and length of each field with no issues but ...
- 4838 Views
- 3 replies
- 4 kudos
- 4 kudos
I found a way to get what I needed and I can apply this to any fixed width file. Will share for anyone trying to do the same thing. I accomplished this in a Python notebook and will explain the code:Import the libraries needed and define a schema.i...
- 4 kudos
- 1682 Views
- 1 replies
- 0 kudos
Read the file from datalake in databricks -No such file directory error
0I have a problem with reading the file from ADLS gen 2.I have dont the mounting properly as after executing dbutils.fs.ls('/mnt/bronze') I can see the file path.the way how I did the mounting: # dbutils.fs.mount( # source = "abfss://"+container_r...
- 1682 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Givi Salu Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 3863 Views
- 1 replies
- 2 kudos
Pyspark Merge parquet and delta file
Is it possible to use merge command when source file is parquet and destination file is delta? Or both files must delta files? Currently, I'm using this code and I transform parquet into delta and it works. But I want to avoid of this tranformation.T...
- 3863 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Ales ventus We haven't heard from you since the last response from @Kaniz Fatma , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others...
- 2 kudos
- 1170 Views
- 0 replies
- 0 kudos
HI ML Practitioners, I want to ask you all how are you productionizing your ML workloads? Are you using ML Flow?Whats your take on ML Flow Recipies?...
HI ML Practitioners, I want to ask you all how are you productionizing your ML workloads? Are you using ML Flow?Whats your take on ML Flow Recipies? Lets get the conversation started.MLflow Recipes (previously known as MLflow Pipelines) is a framewo...
- 1170 Views
- 0 replies
- 0 kudos
- 2242 Views
- 1 replies
- 2 kudos
Connecting to Synapse database using AzureCliCredential token in Spark
I want to connect to my Azure Synapse database using Spark. I can do this in pyodbc no problem but that is not what I want.Here is how I get my credentialscredential = AzureCliCredential() databaseToken = credential.get_token('https://database.window...
- 2242 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Patrick Grover We haven't heard from you since the last response from @Kaniz Fatma , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ot...
- 2 kudos
- 2579 Views
- 2 replies
- 3 kudos
View rocksdb statestore
I used rocksdb statestore for streaming and use collect_set to achieve de-dup logic.Is there any way I can convert rocksdb key value iterator to normal string as I need to validate how internally it is stored as I might need to store 50k distinct val...
- 2579 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @somanath Sankaran Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 3 kudos
- 2308 Views
- 2 replies
- 2 kudos
Bug with display function with structs?
It appears to me that there's a deceptive bug when using the databricks display function and viewing struct data. For whatever reason, multiple spaces are cut down to only one:from pyspark.sql.functions import struct, col df = spark.createDataFrame...
- 2308 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Patrick Mascari Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 20060 Views
- 14 replies
- 5 kudos
Resolved! How do you get data from Azure Data Lake Gen 2 Mounted or Imported and Exported from Databricks?
The example that data bricks gives is not helpful and does not tell me exactly what I need to do. I am new to this and not sure what I need to do in azure to get this done. I just need to be able to pull data and write data to the data containers. Be...
- 20060 Views
- 14 replies
- 5 kudos
- 5 kudos
Hi @Chris Sarrico Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 5 kudos
- 2686 Views
- 1 replies
- 6 kudos
docs.databricks.com
New unified Databricks navigationDatabricks plans to enable the new navigation experience (Public Preview) by default for all users. You’ll be able to opt out by clicking Disable new UI in the sidebar.The goal of the new experience is to reduce click...
- 2686 Views
- 1 replies
- 6 kudos
- 6 kudos
Hi @Priyadarshini G Thank you for providing accurate and valuable information.Best Regards
- 6 kudos
- 3581 Views
- 4 replies
- 3 kudos
dbx execute cluster issue
I am facing issue with cluster not being able to run the updated code using dbx execute command. Any changes I make to the code is not reflected in the execution until I restart the cluster. I am using a photon enable cluster with Standard_D4s_v5 as ...
- 3581 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Tarique Anwar Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 3 kudos
- 2683 Views
- 3 replies
- 3 kudos
Can someone help my academy account switching from partner to customer?
Hi, Today I accidentally created my account in partner-academy.I created my academy account in partner-academy with my company email, then I realized I can't create account with same email in customer-academy which is correct place I should have crea...
- 2683 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Haekyung Won Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 3 kudos
