Hi guys We want to move from databricks workspace to databricks repos. Now when we create a new folder under repos and connect to our azure dev ops repository all our dbutils.widget parameters are lost. How to avoid this? We want to fully depend on a...
Hi @Kevin Peter M. Marti​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
Hi All,We want to read/write data to Databricks SQL using powerapps. I have been looking for documentation around accessing databases in databricks SQL via rest api.Appreciate your help on this.
Along withh several other issues I'm encountering, I am finding pandas dataframe to_sql being very slowI am writing to an Azure SQL database and performance is woeful. This is a test database and it has S3 100DTU and one user, me as it's configuratio...
Hi @Peter McLarty​ Does @Debayan Mukherjee​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
I'm working on a task where I transform a dataset and re-save it to an S3 bucket. This involves joining the dataset to two others, dropping fields from the initial dataset which overlapped with fields from the other two, hashing certain fields with p...
Now, I can't login to https://community.cloud.databricks.com/login.html with the correct username and password, later I click the button to reset my password and I receive the email for modifying password, I have modified password, But I still can't ...
REST API Documentation is out of date since the release of Delta Live TablesWhen using the `2.0/clusters/list` endpoint in an environment with running clusters provisioned by DLTs, the clusters will be returned with a `cluster_source` value of `PIPEL...
Hi @Sam Steere​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
Hi! I'm using the code from here to execute a query on Databricks, it goes flawlessly, can follow it from the Spark UI, etc. The problem here is at the moment it seems the download of the result (spark is idle, there is a green check in the query his...
Hi @Alejandro Martinez​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...
I have 150k small csv files (~50Mb) stored in S3 which I want to load into a delta table.All CSV files are stored in the following structure in S3:bucket/folder/name_00000000_00000100.csvbucket/folder/name_00000100_00000200.csvThis is the code I use ...
Hi @Jan R​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
Issue: Spark structured streaming applicationAfter adding the listener jar file in the cluster init script, the listener is working (From what I see in the stdout/log4j logs)But when I try to hit the 'Content-Type: application/json' http://host:port/...
Hi @swetha kadiyala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
Hi @Nidhi kawale​ Thank you for reaching out!Let us look into this for you, and we will get back to you with an update.Kindly, share your email id at community@databricks.com.
I am trying to install the systemml package from Maven, I ignored the librarieslog4j:log4j, com:sun.jdmk, com:sun.jmx, javax:jmsBut when I run one command of systemml, then spark/databricks can no longer select from tables, effectively breaking somet...
We're trying to use DBR 11.2 & 10.4LTS in Shared mode on a customer managed vpc. But we're running into following issues Is this issue related to our customer managed VPC setup or is it specific to DBR 11.2.Same issue also seen in DBR 11.1 and 10.4 L...
Hello,I found some benchmarks between GPU and CPU Spark-based systems that are not performed in the same hardware. Is this faire since a powerful CPU server could eventually outperforms a GPU server?For example,Here, the performance comparison is don...
Right now the feature seems to be public available. It is possible to schedule jobs with ADLS passthough enabled and do not have to provide service principal credentials.However I ask myself how that works behind the scenses. When working interactive...
Hi @Lindsay Olson​ ,I have attended the virtual instructor-led training on 23-08-2022 (https://www.databricks.com/p/webinar/databricks-lakehouse-fundamentals-learning-plan). As per the conditions mentioned, I have completed all of the steps for getti...
