- 9435 Views
- 9 replies
- 5 kudos
Hi, I am facing a problem that I hope to get some help to understand. I have created a function that is supposed to check if the input data already exist in a saved delta table and if not, it should create some calculations and append the new data to...
- 9435 Views
- 9 replies
- 5 kudos
Latest Reply
Hi,im also having similar issue ..does creating temp view and reading it again after saving to a table works?? /
8 More Replies
- 1279 Views
- 1 replies
- 2 kudos
Hi, I am facing an issue where one of my jobs taking so long since certain time, previously its only needs less than 1 hour to run a batch job that load json data and do a truncate and load to a delta table, but since june 2nd, it become so long that...
- 1279 Views
- 1 replies
- 2 kudos
Latest Reply
Hi @krisna math Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
by
az38
• New Contributor II
- 7418 Views
- 2 replies
- 3 kudos
Hi, community!How do you think what is the best way to load from Azure ADLS (actually, filesystem doesn't matter) into df onli files modified after some point in time?Is there any function like input_file_name() but for last_modified to use it in a w...
- 7418 Views
- 2 replies
- 3 kudos
Latest Reply
_metadata will provide file modification timestamp. I tried on dbfs but not sure for ADLS.https://docs.databricks.com/ingestion/file-metadata-column.html
1 More Replies
- 2038 Views
- 3 replies
- 1 kudos
I am using DLT to load csv in ADLS, below is my sql query in notebook:CREATE OR REFRESH STREAMING LIVE TABLE test_account_raw
AS SELECT * FROM cloud_files(
"abfss://my_container@my_storageaccount.dfs.core.windows.net/test_csv/",
"csv",
map("h...
- 2038 Views
- 3 replies
- 1 kudos
Latest Reply
thank you every one, the problem is resolved, problem is gone when I have workspace admin access.
2 More Replies
- 4129 Views
- 3 replies
- 3 kudos
let us say I already have the data 'TotalData'write.csv(TotalData,file='/tmp/TotalData.csv',row.names = FALSE)I do not see any error from abovewhen I list files below:%fs ls /tmpI do not see any files written there. Why?
- 4129 Views
- 3 replies
- 3 kudos
Latest Reply
Hi Thiam,Thank you for reaching out to us. In this case it seems that you have written a file to the OS /tmp and tried to fetch the same folder in DBFS.Written >> /tmp/TotalData.csvReading >> /dbfs/tmp/TotalData.csvPlease try to execute write.csv wit...
2 More Replies
- 2224 Views
- 3 replies
- 3 kudos
Hello,I have created my table in Databricks, at this point everything is perfect i got the same value than in my CSV. for my column "Exposure" I have :0 0,00
1 0,00
2 0,00
3 0,00
4 0,00
...But when I load my fi...
- 2224 Views
- 3 replies
- 3 kudos
Latest Reply
Hi @Anis Ben Salem ,How do you read your CSV file? do you use Pandas or Pyspark APIs? also, how do you created your table?could you share more details on the code you are trying to run?
2 More Replies
