- 3641 Views
- 2 replies
- 1 kudos
- 3641 Views
- 2 replies
- 1 kudos
- 1 kudos
a bit old, but I just faced the same issue, specifying a custom EncryptionMaterialsProvider (as described in the previous post) did the trick for me but I did had to also specify my kms endpoint, just because my region:"fs.s3.cse.kms.endpoint" -> "km...
- 1 kudos
- 3667 Views
- 3 replies
- 1 kudos
Passing Parameter from SSRS to Databricks user defined function
I am trying to pass parameter from SSRS to User Defined Function in Databricks which in turn will return table that will be shown as output in report.I tried below calling function from SSRS, but it looks like parameter value is not passed. I have di...
- 3667 Views
- 3 replies
- 1 kudos
- 15500 Views
- 2 replies
- 0 kudos
Resolved! SQL Merge Statement not working
Hi I am trying to use the SQL Merge statement on databricksMERGE INTO targetUSING sourceON source.key = target.keyWHEN MATCHED UPDATE SET *WHEN NOT MATCHED INSERT *WHEN NOT MATCHED BY SOURCE DELETEThis is failing with the error [PARSE_SYNTAX_ERROR...
- 15500 Views
- 2 replies
- 0 kudos
- 0 kudos
I was missing the THEN before UPDATE, INSERT and DELETE. This keyword is missing from the documentation on Databricks https://learn.microsoft.com/en-us/azure/databricks/delta/mergeIt now works. Thanks
- 0 kudos
- 8263 Views
- 3 replies
- 2 kudos
AWS Databricks Pyspark - Unable to connect to Azure MySQL - Shows "SSL Connection is required"
Even after specifying SSL options, unable to connect to MySQL. What could have gone wrong? Could anyone experience similar issues? df_target_master = spark.read.format("jdbc")\.option("driver", "com.mysql.jdbc.Driver")\.option("url", host_url)\.optio...
- 8263 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey,Here the solution: The correct option for ssl is "useSSL" and not just "ssl".This code below could works:df_target_master = spark.read.format("jdbc")\.option("driver", "com.mysql.jdbc.Driver")\.option("url", host_url)\.option("dbtable", supply_ma...
- 2 kudos
- 2664 Views
- 1 replies
- 1 kudos
Error while running spark.catalog.listDatabases()
I am running steps mentioned in https://github.com/databrickslabs/splunk-integration/blob/master/notebooks/source/push_to_splunk.pyWhen I am running spark.catalog.listDatabases()getting error py4j.security.Py4JSecurityException: Method public java.l...
- 2664 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Purnima Bhatia​ , I faced a similar error for a different command when I was using a wrong type of cluster access mode. You can try to create a different cluster with different access mode and check. I might be wrong but try and check this.
- 1 kudos
- 4519 Views
- 3 replies
- 1 kudos
Saving Number field as String in Databricks
Do we see any risk of saving a Number field as String? Will we use any functionality/feature if we save as String ? Will it have any impact on performance ?
- 4519 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Manju Chugani​. Yes. In Short, it is not really recommended to save the columns as string if all the values are expected to be numbers.Here are some of them Storage Space: Storing numbers as strings can take up more storage space than storing the...
- 1 kudos
- 16123 Views
- 10 replies
- 7 kudos
Resolved! Can Terraform be used to set configurations in Admin / workspace settings?
I am posting this on behalf of my customer. They are currently working on the deployment & config of their workspace on AWS via Terraform.Is it possible to set some configs in the Admin/workspace settings via TF? According to the Terraform module, it...
- 16123 Views
- 10 replies
- 7 kudos
- 7 kudos
Hi, can I get a full list of the latest configurable supported workspace_conf on tf, I can't find the list on tf registry site.
- 7 kudos
- 3981 Views
- 3 replies
- 0 kudos
Cluster Configuration for ML Model Training
Hi!I am training a Random Forest (pyspark.ml.classification.RandomForestClassifier) on Databricks with 1,000,000 training examples and 25 features. I employ a cluster with one driver (16 GB Memory, 4 Cores), 2-6 workers (32-96 GB Memory, 8-24 Cores),...
- 3981 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @John B​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can...
- 0 kudos
- 3076 Views
- 3 replies
- 3 kudos
training_set.load_df().toPandas() fails with the new pandas version (2.0.0)
pandas 2.0.0 was released on 4.3.2023 and was pushed to my cluster on the same day. The day after I tried using training_set.load_df().toPandas() and it failed. Reverting to pandas 1.5.3. fixed the problem.
- 3076 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Al IDI​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your q...
- 3 kudos
- 2490 Views
- 3 replies
- 0 kudos
Hi, I have completed lakehouse fundamentals accreditation, but did not receive the badge yet, only have the certificate of completion.
Hi, I have completed lakehouse fundamentals accreditation, but did not receive the badge yet, only have the certificate of completion.
- 2490 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @RD DO​ Apologies as we have an issue with our credentials app. We are working with the vendor to resolve it. We expect to be able to grant your badge soon.Thank you!
- 0 kudos
- 5680 Views
- 2 replies
- 0 kudos
TPC-DI Benchmark in Databricks (ETL)
Similar to TPC-DS and TPC-H benchmark, is it possible to execute TPC-DI benchmark in Databricks. What is the feasibility to execute TPC-DI benchmark, please let me know if there is any existing ETL Tool available in Databricks for TPC-DI. For referen...
- 5680 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, we recently made public a repo that makes it easy to run the TPC-DI benchmark end-to-end - in a variety of ways. The repo can be found here. Feel free to let me know if you have any issues running it!
- 0 kudos
- 11271 Views
- 4 replies
- 0 kudos
Update record in databricks sql table from C#.Net in visual studio 2022 using ODBC
I am trying to make a backend method call work which connects to database and updates the record in a table. The method call works perfectly and it is not throwing any error but at the same time it does not update any record in the table either.Note:...
- 11271 Views
- 4 replies
- 0 kudos
- 0 kudos
I found a temporary work around. Instead of passing the value through the parameters, I passed the value directly in query like below. I know it is not the ideal way but at this time this is working. If I do not pass the value as shown below, the log...
- 0 kudos
- 5885 Views
- 5 replies
- 1 kudos
parsing error in Databricks SQL endpoint
I have two tables EMPLOYEE & EMPLOYEE_ROLE. I'm trying to Update a column with a value from another column. I'm using SQL server join but i get an error - [parse_syntax_error] Syntax error at or near 'FROM' line 3. UPDATE CSET C.title = B.title FROM ...
- 5885 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Vijesh V​ Try to use merge into to perform cdc between tables :MERGE INTO target aUSING source bON {merge_condition}WHEN MATCHED THEN {matched_action}WHEN NOT MATCHED THEN {not_matched_action}
- 1 kudos
- 4719 Views
- 3 replies
- 2 kudos
- 4719 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Ravi Vishwakarma​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...
- 2 kudos
- 4794 Views
- 4 replies
- 3 kudos
[Databricks][JDBC](10400) Invalid type for data - column: 10, type: Array
Getting below error while mapping an Array Column to String[] entity. Please suggest if Databricks JDBC support entity mapping of Array Values [Worked the same code for below config - H2 DB version - 2.1.214 and org.hibernate.dialect.H2Dialect - ...
- 4794 Views
- 4 replies
- 3 kudos
- 3 kudos
Hello @Emmanuel Trindade​ @Praveen C​ This does not look like coming from Databricks end. Look at the error thread.javax.persistence.PersistenceException: org.hibernate.exception.DataException: Could not read entity state from ResultSet : EntityKey...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
