- 3012 Views
- 1 replies
- 22 kudos
How to process files from the internet in databricks? "spark.sparkContext.addFile" download file to HDFS directory. "SparkFiles.get&quo...
How to process files from the internet in databricks?"spark.sparkContext.addFile" download file to HDFS directory. "SparkFiles.get" return the path and the name. However, as Databricks use the DBFS file system, we need to add the "file:///" prefix to...
- 3012 Views
- 1 replies
- 22 kudos
- 22 kudos
@Hubert Dudek Do you know if addFile should work with abfss:// path? Trying to add a file from azure data lake with external location in unity catalog.
- 22 kudos
- 18902 Views
- 9 replies
- 7 kudos
DLT and Modularity (best practices?)
I have [very] recently started using DLT for the first time. One of the challenges I have run into is how to include other "modules" within my pipelines. I missed the documentation where magic commands (with the exception of %pip) are ignored and was...
- 18902 Views
- 9 replies
- 7 kudos
- 7 kudos
I like the approach @Arvind Ravish shared since you can't currently use %run in DLT pipelines. However, it took a little testing to be clear on how exactly to make it work. First, ensure in the Admin Console that the repos feature is configured as f...
- 7 kudos
- 1732 Views
- 1 replies
- 1 kudos
how to add the title excelsheet with python
i want to write title with some combination of rows in pandas df, and write into excel sheet. i tried some method but i could see styler object is not subscriptable
- 1732 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Mohammed sadamusean ,Can you please share the sample input and sample expected output, so that we can try on our end and can let you know.Happy Learning!!
- 1 kudos
- 9778 Views
- 5 replies
- 0 kudos
Screenshot 2023-01-09 094039
Hey guys,Databricks academy login is not working. I have been trying for the past 1 hour and still doesn't work. It seems to be with the Databricks https certificate being expired but not sure. I'm attaching an image with the error. Any help with thi...
- 9778 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Andre Paiva ,Can you please try now I can able to load both customer and partner academy websites, I think the Academy team has fixed the issue. Happy Learning!!
- 0 kudos
- 4057 Views
- 4 replies
- 3 kudos
Resolved! How to add a current date as suffix while using copy?
how to add a current date after filename suffix while copy from the dbutils like report20221223.xlsxdbutils.fs.cp('dbfs://temp/balancing/report.xlsx','abfss://con@adls/provsn/result/report.xlsx',True)i need to add the current date in the file like ...
- 4057 Views
- 4 replies
- 3 kudos
- 3 kudos
@Mohammed sadamusean hope the below code might help you, from datetime import datetime date_value = datetime.now().strftime("%Y%m%d") src = 'dbfs:/FileStore/Test/File.csv' trgt = f'dbfs:/FileStore/Test/File_{date_value}.csv' dbutils.fs.cp(src,t...
- 3 kudos
- 7904 Views
- 6 replies
- 0 kudos
Resolved! Monaco editor - Toggle line comment not working
I recently tried the new editor, and usual shortcuts like CTRL + / to comment is ineffective. Is this a known issue? It's working fine with the classic editor, so I am switching back to it in the meantime, but it would be great to use this new additi...
- 7904 Views
- 6 replies
- 0 kudos
- 1738 Views
- 1 replies
- 19 kudos
�� Deltalake Vs Datalake in Databricks ��Delta Lake Databricks Delta Lake is an open-source storage layer that sits on top of existing d...
Deltalake Vs Datalake in Databricks Delta Lake DatabricksDelta Lake is an open-source storage layer that sits on top of existing data lake storage, such as Azure Data Lake Store or Amazon S3. It provides a more robust and scalable alternative to tra...
- 1738 Views
- 1 replies
- 19 kudos
- 7488 Views
- 2 replies
- 0 kudos
Can't parallelize model training with sc.parallelize, even tough I can run the same code without parallelizing
I'm training a NeuralProphet for a time series forecasting problem. I'm trying to parallelize my training, but this error is appearingThe folder lightning_logs has a hparams.yaml but it's empty. Is this related to permissions on the cluster? Thanks i...
- 7488 Views
- 2 replies
- 0 kudos
- 9508 Views
- 3 replies
- 5 kudos
Resolved! Azure Databricks unable to connect to private DNS KeyVault in createScope, showing "DNS invalid"
I have an Azure KeyVault with private endpoint created in the same Vnet as Azure Databricks. While trying to add it as a scope using the private DNS Zone ie <KVname>.privatelink.vaultcore.azure.netgetting error "DNS is invalid and cannot be reached....
- 9508 Views
- 3 replies
- 5 kudos
- 5 kudos
I got it working by creating the KV backed scope via UI. I used the the dns without the private part: <KVName>.vault.azure.netThe private dns will resolve it to the right IP.You do have to check the "Allow trusted Microsoft services to bypass this fi...
- 5 kudos
- 2979 Views
- 1 replies
- 20 kudos
⏩ Understanding Unity Catalog in Databricks ⏮ In Databricks, the Unity Catalog is a data catalog that allows you to store, access, and manage data wit...
Understanding Unity Catalog in Databricks In Databricks, the Unity Catalog is a data catalog that allows you to store, access, and manage data within your Databricks workspace. It provides a unified interface for working with data across different s...
- 2979 Views
- 1 replies
- 20 kudos
- 2918 Views
- 1 replies
- 1 kudos
DB Connect failing
For some reason, my dbconnect failed and I haven't been able to resolve the issue. I am connecting to an enterprise server. I was getting the following errors which (I believe) are now resolved. I defined the PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHO...
- 2918 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, Could you please confirm if you are providing the right shard ID/URL? Also, please let us know which guide has been followed to configure the connectivity. Also, please provide the screenshot of the error after entering the shard URL.
- 1 kudos
- 1861 Views
- 2 replies
- 0 kudos
How do I find the documentation for a Databricks "platform release"?
My DB portal says the platform version is v3.86, and provides a link to all the releases. But none of those releases state the "platform version". And I can't find "v3.86" by searching in the Databricks docs.So, how does one find the documentation fo...
- 1861 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @Samuel Yang ,Here you will find all detailshttps://docs.databricks.com/release-notes/index.htmlThanksAviral Bhardwaj
- 0 kudos
- 4064 Views
- 3 replies
- 4 kudos
Link a visio diagram in a markdown cell
Is there a way to have databricks pull a diagram directly from visio? I've tried to use the embed links from visio but the image won't render. I'm trying to get around loading the image to DBFS as there may be updates to the image that I want it to g...
- 4064 Views
- 3 replies
- 4 kudos
- 3166 Views
- 1 replies
- 1 kudos
Resolved! CloudFormation failure while creating workspace
The cloud formation script is fails to create workspace due to following error. The solution to this error is not available in the standard troubleshooting guide. Following is the error message from cloud watch logs. It seems like a bug in the api, b...
- 3166 Views
- 1 replies
- 1 kudos
- 1 kudos
I found a solution to this by creating a new role following these articles -https://docs.databricks.com/administration-guide/account-api/iam-role.htmlhttps://docs.databricks.com/administration-guide/account-api/aws-storage.html
- 1 kudos
- 3102 Views
- 7 replies
- 0 kudos
Hi Databricks Team, I passed the associate data engineer exam day before but still haven't received on accredible or on db academy. My registere...
Hi Databricks Team, I passed the associate data engineer exam day before but still haven't received on accredible or on db academy. My registered email id for exam is sharath.koushik@gmail.com. Could you please help ?
- 3102 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @Sharath K Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
