- 2101 Views
- 2 replies
- 0 kudos
- 2101 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @令辉 å”​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback will he...
- 0 kudos
- 3327 Views
- 2 replies
- 1 kudos
Cosmos DB Connector for 12.1 Cluster and above
Hi guys,​You know what`s the version suporte the databricks cluster version above 12.1 ?my cluster:error: Thank you
- 3327 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 1 kudos
- 3678 Views
- 1 replies
- 0 kudos
Resolved! TypeError on DataFrame via spark readStream transform invocation of UDF
Our use case is to "clean up" column names (remove spaces, etc) on ingestion of CSV data using the Delta Live Table capability. We desire to use the schema inference capability during ingestion so schema specification (up front) will not be happenin...
- 3678 Views
- 1 replies
- 0 kudos
- 0 kudos
The issue was erroneously believing the transform function needed UDF decorator. With the decorator removed the transform invokes (and works) as expected.
- 0 kudos
- 4491 Views
- 3 replies
- 0 kudos
Performance issue while loading bulk data into Post Gress DB from data bricks.
We are facing a performance issue while loading bulk data into Postgress DB from data bricks. We are using spark JDBC connections to move the data. However, the rate of transfer is very low which is causing performance bottleneck. is there any better...
- 4491 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @Janga Reddy​ @Daniel Sahal​ and @Vidula Khanna​ ,To enhance performance in general we need to design for more parallelism, in Spark JDBC context this controlled by the number of partitions for the data to be writtenThe example here shows how t...
- 0 kudos
- 5095 Views
- 2 replies
- 2 kudos
Resolved! I am new to data bricks. setting up Workspace for NON-prod environment Separate workspaces for DEV, QA or Just one work space for NON-prod ?
What i learned based on learning materials, documents, etc.. For data bricks it is a good practice to set up 1 non-prod workspace but separate clusters for Dev, QA, SIT, etc.Is it best practice to set up only 1 NON-PROD Workspace instead of separate ...
- 5095 Views
- 2 replies
- 2 kudos
- 6603 Views
- 4 replies
- 2 kudos
SAT - Security Analysis Tool implementation error
I want to implement SAT in my workspace account. I was able to execute the terraform that enable the necessary infra to work on that. When I try to execute the workflow "SAT Initializer Notebook (one-time)" it fails with the error:AnalysisException: ...
- 6603 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Arnold Souza​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 2 kudos
- 4974 Views
- 1 replies
- 7 kudos
SQL cells in databricks notebooks can now be run in parallel, which means faster query processing and analysis. This new feature is especially helpful...
SQL cells in databricks notebooks can now be run in parallel, which means faster query processing and analysis. This new feature is especially helpful for queries that take longer to run or analyze large datasets. With parallel processing, Databricks...
- 4974 Views
- 1 replies
- 7 kudos
- 7 kudos
Informative ​
- 7 kudos
- 17677 Views
- 1 replies
- 1 kudos
Resolved! MERGE to update a column of a table using Spark SQL
Coming from MS SQL background, I'm trying to write a query in Spark SQL that simply update a column value of table A (source table) by INNER JOINing a new table B with a filter.MS SQL query looks like this:UPDATE T SET T.OfferAmount = OSE.EndpointEve...
- 17677 Views
- 1 replies
- 1 kudos
- 1 kudos
Posting answer to my question: MERGE into TempOffer VIEW USING OfferSeq OSE ON VIEW.OfferId = OSE.OfferID AND OSE.OfferId = 1 WHEN MATCHED THEN UPDATE set VIEW.OfferAmount = OSE.EndpointEventAmountValue;
- 1 kudos
- 3999 Views
- 5 replies
- 2 kudos
Is there a stream / Kafka topic that we can connect to for monitoring all Databricks jobs/workflows (create/status update/fail/error/complete)?
Currently we are creating and monitoring jobs using the api. This results in a lot of polling of the API for job status. Is there a Kafka stream, we could listen to get jobs updates and significantly reduce the number of calls to the Databricks jobs...
- 3999 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Ryan Hager​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...
- 2 kudos
- 4410 Views
- 3 replies
- 3 kudos
Resolved! How do we set spark_version in cluster policies to select the latest GPU ML LTS version as defaultValue?
Currently, I use the below two different JSON snippets to choose either Standard or ML runtime. Similar to the below, what is the defaultValue for spark_version to select the latest GPU ML LTS runtime version? "spark_version": { "type": "regex", "p...
- 4410 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Ramana Kancharana​ ,As of right now these options are only available for non-GPU DBRs
- 3 kudos
- 6024 Views
- 1 replies
- 3 kudos
TimestampFormat issue
The databricks notebook failed yesterday due to timestamp format issue. error:"SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '2022-08-10 00:00:14.2760000' in the new parser. You can set spark.s...
- 6024 Views
- 1 replies
- 3 kudos
- 3 kudos
You must have solved this issue by now but for the sake of those that encounter this again, here's the solution that worked for me:spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
- 3 kudos
- 4596 Views
- 5 replies
- 0 kudos
Cannot find such info if Databricks supports nested jobs or tasks. For example, I have a 'job_a', which contains list of tasks, and another &#...
Cannot find such info if Databricks supports nested jobs or tasks. For example, I have a 'job_a', which contains list of tasks, and another 'job_b', also contains a list of tasks. Now I'd like to have a 'job_all' that will run both 'job_a' and 'job_b...
- 4596 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Yanan Zhang​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the response and select the one that best answers yo...
- 0 kudos
- 6597 Views
- 4 replies
- 2 kudos
Resolved! No Explicit Deny for User security configurations at the group level?
Currently when you add new users to the Databricks workspace they get added to a "Users" group that has full access to the workspace. There should be a way to use group security to explicitly deny access to those same settings. This setting should ov...
- 6597 Views
- 4 replies
- 2 kudos
- 2 kudos
@dean james​ I am not sure about your case why you want to deny access to the group once you create it. Anyhow, we can use deacticate/activate an user using "2.0/preview/scim/v2/Users/{id}" rest API endpoint. We can also deactivate users that have no...
- 2 kudos
- 6942 Views
- 4 replies
- 0 kudos
Resolved! Can merge() function be applied to dataframe?
if I have two dataframes df_target and df_source, can I do df_target.as("t).merge(df_source.as("s"), "s.id=t.id").whenMatched().updateAll().whenNotMatched.insertAll.execute(). when I tried the code above, I got the error "merge is not a member of the...
- 6942 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @andrew li​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 21596 Views
- 3 replies
- 3 kudos
Resolved! Does Spark SQL can perform UPDATE with INNER JOIN and LIKE with '%' + [column] + '%' ?
Hi All,I came from MS SQL and just started to learning more about Spark SQLHere is one part that I'm trying to perform. In MS SQL, it can be easily done, but it seems like it doesn't in SparkSo, I want to make a simple update to the record, if the co...
- 21596 Views
- 3 replies
- 3 kudos
- 3 kudos
@Hubert Dudek​ Hello, I'm having the same issue with using UPDATE in spark sql and came across your answer. When you say "replace source_table_reference with view" in MERGE, do you mean to replace "P" with "VIEW" that looks something as below:%sql ME...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
