- 7675 Views
- 2 replies
- 0 kudos
Can't parallelize model training with sc.parallelize, even tough I can run the same code without parallelizing
I'm training a NeuralProphet for a time series forecasting problem. I'm trying to parallelize my training, but this error is appearingThe folder lightning_logs has a hparams.yaml but it's empty. Is this related to permissions on the cluster? Thanks i...
- 7675 Views
- 2 replies
- 0 kudos
- 9676 Views
- 3 replies
- 5 kudos
Resolved! Azure Databricks unable to connect to private DNS KeyVault in createScope, showing "DNS invalid"
I have an Azure KeyVault with private endpoint created in the same Vnet as Azure Databricks. While trying to add it as a scope using the private DNS Zone ie <KVname>.privatelink.vaultcore.azure.netgetting error "DNS is invalid and cannot be reached....
- 9676 Views
- 3 replies
- 5 kudos
- 5 kudos
I got it working by creating the KV backed scope via UI. I used the the dns without the private part: <KVName>.vault.azure.netThe private dns will resolve it to the right IP.You do have to check the "Allow trusted Microsoft services to bypass this fi...
- 5 kudos
- 3068 Views
- 1 replies
- 20 kudos
⏩ Understanding Unity Catalog in Databricks ⏮ In Databricks, the Unity Catalog is a data catalog that allows you to store, access, and manage data wit...
Understanding Unity Catalog in Databricks In Databricks, the Unity Catalog is a data catalog that allows you to store, access, and manage data within your Databricks workspace. It provides a unified interface for working with data across different s...
- 3068 Views
- 1 replies
- 20 kudos
- 3006 Views
- 1 replies
- 1 kudos
DB Connect failing
For some reason, my dbconnect failed and I haven't been able to resolve the issue. I am connecting to an enterprise server. I was getting the following errors which (I believe) are now resolved. I defined the PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHO...
- 3006 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, Could you please confirm if you are providing the right shard ID/URL? Also, please let us know which guide has been followed to configure the connectivity. Also, please provide the screenshot of the error after entering the shard URL.
- 1 kudos
- 1945 Views
- 2 replies
- 0 kudos
How do I find the documentation for a Databricks "platform release"?
My DB portal says the platform version is v3.86, and provides a link to all the releases. But none of those releases state the "platform version". And I can't find "v3.86" by searching in the Databricks docs.So, how does one find the documentation fo...
- 1945 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @Samuel Yang ,Here you will find all detailshttps://docs.databricks.com/release-notes/index.htmlThanksAviral Bhardwaj
- 0 kudos
- 4236 Views
- 3 replies
- 4 kudos
Link a visio diagram in a markdown cell
Is there a way to have databricks pull a diagram directly from visio? I've tried to use the embed links from visio but the image won't render. I'm trying to get around loading the image to DBFS as there may be updates to the image that I want it to g...
- 4236 Views
- 3 replies
- 4 kudos
- 3392 Views
- 1 replies
- 1 kudos
Resolved! CloudFormation failure while creating workspace
The cloud formation script is fails to create workspace due to following error. The solution to this error is not available in the standard troubleshooting guide. Following is the error message from cloud watch logs. It seems like a bug in the api, b...
- 3392 Views
- 1 replies
- 1 kudos
- 1 kudos
I found a solution to this by creating a new role following these articles -https://docs.databricks.com/administration-guide/account-api/iam-role.htmlhttps://docs.databricks.com/administration-guide/account-api/aws-storage.html
- 1 kudos
- 3239 Views
- 7 replies
- 0 kudos
Hi Databricks Team, I passed the associate data engineer exam day before but still haven't received on accredible or on db academy. My registere...
Hi Databricks Team, I passed the associate data engineer exam day before but still haven't received on accredible or on db academy. My registered email id for exam is sharath.koushik@gmail.com. Could you please help ?
- 3239 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @Sharath K Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 5830 Views
- 3 replies
- 2 kudos
How to remove extra ENTER line in csv UTF-16 while reading
Dear Friends,I have a csv and it looks like this‡‡Id‡‡,‡‡Version‡‡,‡‡Questionnaire‡‡,‡‡Date‡‡‡‡123456‡‡,‡‡Version2‡‡,‡‡All questions have been answered accurately and the guidance in the questionnaire was understood and followed‡‡,‡‡2010-12-16 00:01:...
- 5830 Views
- 3 replies
- 2 kudos
- 2 kudos
This is working fine, from pyspark.sql.functions import regexp_replace path="dbfs:/FileStore/df/test.csv" dff = spark.read.option("header", "true").option("inferSchema", "true").option('multiline', 'true').option('encoding', 'UTF-8').option("delimi...
- 2 kudos
- 6648 Views
- 6 replies
- 8 kudos
Resolved! How can we move the excel file from adls to share point
I have one excel file in the adls, i want to move that file into sharepoint, but i tried this method in data factory, but in sink sharepoint is not available, is there any possible way to do this?
- 6648 Views
- 6 replies
- 8 kudos
- 8 kudos
If you want to upload raw Excel file (not Sharepoint List) then IMO it would be easier to simply mount the Sharepoint Site as a network drive (https://support.microsoft.com/en-us/office/map-a-network-drive-to-a-sharepoint-library-751148de-f579-42f9-b...
- 8 kudos
- 2626 Views
- 4 replies
- 3 kudos
- 2626 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @A DB Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 8104 Views
- 5 replies
- 4 kudos
"Detected schema change" error while reading from delta table in streaming after applying "ALTER COLUMN DROP NOT NULL" to more than one columns.
Hi!I have a delta table and a process that reading a stream from this table.I need to drop the NOT NULL constraint from some of the columns of this table.The first drop command does not affect the reading stream.But the second command results in erro...
- 8104 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @Anatoly Tikhonov Hope everything is going great.Does @Kaniz Fatma response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 4 kudos
- 4916 Views
- 7 replies
- 4 kudos
Certificate not received -
Hi ThereI successfully passed the exam on 27th Oct 2022 - but I haven't received the certificate yet.
- 4916 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @Gaurav Jhamb Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 4 kudos
- 4373 Views
- 6 replies
- 0 kudos
I've successfully passed Databricks Data Engineer Associate Certified exam but still have not received the certificate. Could you help on it pleas...
I've successfully passed Databricks Data Engineer Associate Certified exam but still have not received the certificate. Could you help on it please.
- 4373 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Venkata Sai Anuroop Samudrala Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to he...
- 0 kudos
- 2835 Views
- 2 replies
- 1 kudos
Configure DLT Pipeline to use existing running cluster
How can I configure a DLT pipeline to use an existing running cluster? I don't see where in the settings to set the pipeline to use an existing cluster. Instead it wants to always standup a new cluster.
- 2835 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Justin Stuparitz Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 1 kudos
| User | Count |
|---|---|
| 1626 | |
| 790 | |
| 506 | |
| 349 | |
| 287 |
