- 8470 Views
- 2 replies
- 0 kudos
Resolved! WHEN NOT MATCHED BY SOURCE Syntax error at or near 'BY' (DBR 11.2 ML)
Hi, I have trouble with executing the given SQL Statement below.MERGE INTO warehouse.pdr_debit_card as TARGET USING (SELECT * FROM ( SELECT CIF, CARD_TYPE, ISSUE_DATE, MATURITY_DATE, BOO, DATA_DATE, row_number(...
- 8470 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Please refer: https://docs.databricks.com/sql/language-manual/delta-merge-into.html
- 0 kudos
- 1401 Views
- 0 replies
- 0 kudos
Delta Live Tables migration
How can I migrate a delta live tables workflow to another Databricks workspace?PS: Data source/sink will remain the same. I only want to migrate the DLT config.
- 1401 Views
- 0 replies
- 0 kudos
- 2844 Views
- 4 replies
- 0 kudos
What would be the best plan for independent course creator?
Hi folks! I want to use databrick community edition as the platform to teach online courses. As you may know, for community edition, you need to create a new cluster when the old one terminates. I found out however tables created from the old cluster...
- 2844 Views
- 4 replies
- 0 kudos
- 0 kudos
You can create a notebook for students which recreates everything, like doing the installation of tables etc., before every exercise.
- 0 kudos
- 2327 Views
- 1 replies
- 3 kudos
AutoML forecasting with monthly data?
ARIMA and FBProphet have the capability to forecast monthly data. When using AutoML (via the API or the UI) it seems like it is not possible to have a monthly freq (e.g. 'MS').Is there a way / workaround to make it work with monthly data or is it pla...
- 2327 Views
- 1 replies
- 3 kudos
- 3 kudos
It is possible to use AutoML to forecast monthly data, but it may require some additional steps or adjustments.One approach is to resample the monthly data to a lower frequency such as weekly or daily, and then use AutoML to forecast at that lower fr...
- 3 kudos
- 5775 Views
- 9 replies
- 11 kudos
Databricks start support to run selected text in a cell this will help us a lot during debugging of the code.In windows just select the line of code w...
Databricks start support to run selected text in a cell this will help us a lot during debugging of the code.In windows just select the line of code which you want to execute and press Ctrl+Shift+Enter
- 5775 Views
- 9 replies
- 11 kudos
- 1376 Views
- 0 replies
- 0 kudos
Model registry does not load adapted Normalization-Layer (Keras) correctly
Hello,I have a problem with the model registry. I'm not sure if I'm registering the model incorrectly or if it´s a bug.Here are some code snippets:import pandas as pd import mlflow import mlflow.keras import mlflow.tensorflow from mlflow.tracking.c...
- 1376 Views
- 0 replies
- 0 kudos
- 861 Views
- 0 replies
- 1 kudos
medium.com
During Shuffle operation Data is moving from memory to disk Why?Please find the detailed answer here if any question please comment and hit like and share if interested in upcoming articles.https://medium.com/@sharikrishna26/during-shuffle-operation-...
- 861 Views
- 0 replies
- 1 kudos
- 5005 Views
- 1 replies
- 1 kudos
PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was 1 and the length of input was 2.'.
I have pandas_udf, its working for 4 rows, but I tried with more than 4 rows getting below error.PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was...
- 5005 Views
- 1 replies
- 1 kudos
- 1 kudos
@Kaniz Fatma​ Can you please help me on pandas_udf ?Above scenario I have used regular expressions, for that we have our spark method, but I have other pandas_udf have same issue.
- 1 kudos
- 3620 Views
- 1 replies
- 0 kudos
Can you access a repo file in an init script?
I'd like to configure a cluster with python libraries as defined in a requirements file. I have a pip requirements.txt file in a private repo which I have integrated on Databricks (and I can access it through the UI and view it on Databricks). I upda...
- 3620 Views
- 1 replies
- 0 kudos
- 0 kudos
you can install in a cluster
- 0 kudos
- 1500 Views
- 1 replies
- 5 kudos
accessing secret from spark cluster.
passing spark configuration to access blob, adls from data factory while creating job clusterit's working fine, but when in the property we are accessing secret it's not workingspark.hadoop.fs.azure.account.auth.type.{{secrets/scope/key}}.dfs.core.wi...
- 1500 Views
- 1 replies
- 5 kudos
- 5 kudos
check here : https://docs.databricks.com/security/secrets/secrets.html
- 5 kudos
- 2111 Views
- 2 replies
- 0 kudos
adding Widget as a column and populating its value every-time in that column in a table.
hi , I want date for runtime from ADF as @utcnow() -- base paramater of notebook activity in ADF and take the data in ADB using widgets as runtime_date, further i want that column to be added in my table X with the populated value from the widget.Eve...
- 2111 Views
- 2 replies
- 0 kudos
- 0 kudos
you can use as current_timestamp() or now()refer link: https://docs.databricks.com/sql/language-manual/functions/current_timestamp.html
- 0 kudos
- 31235 Views
- 6 replies
- 7 kudos
Resolved! What does "Determining location of DBIO file fragments..." mean, and how do I speed it up?
Determining location of DBIO file fragments. This operation can take some time.What does this mean, and how do I prevent it from having to perform this apparently-expensive operation every time? This happens even when all the underlying tables are De...
- 31235 Views
- 6 replies
- 7 kudos
- 7 kudos
Determining location of DBIO file fragments" is a message that may be displayed during the boot process of a computer running the NetApp Data ONTAP operating system. This message indicates that the system is currently in the process of identifying an...
- 7 kudos
- 5614 Views
- 4 replies
- 6 kudos
How do I know how much of a query/job used Photon?
I'm trying to use the native execution engine, Photon. How can I tell if a query is using Photon or is falling back to the non-native Spark engine?
- 5614 Views
- 4 replies
- 6 kudos
- 6 kudos
Typo error in my second point of the previous post. Click the execution plan of your task[this is available under SQL/Dataframe tab in Spark UI]. It explains what operations run in the photon engine and what didn't execute by photon.
- 6 kudos
- 9754 Views
- 9 replies
- 2 kudos
text datatype not supported and data having huge data in text filed how to bring it over
Hello all,I have medical field data file and one of the field is the text field with huge data not the big problem is databrick does not support text data type so how can i bring the data over. i tried conversion, cast in various way but so far not ...
- 9754 Views
- 9 replies
- 2 kudos
- 2 kudos
Setting escapeQuotes to false has helped to bring huge text data in colomn.thanks
- 2 kudos
- 3580 Views
- 3 replies
- 1 kudos
Resolved! How to start my cluster
​I try to stop my_cluster from compute from admin role. BTW, using same account, I could not restart my_cluster. The information is as followings. How should I do?
- 3580 Views
- 3 replies
- 1 kudos
- 1 kudos
rit seems this is community edition so in CE this feature is disabled , delete this one and create new cluster
- 1 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 534 | |
| 349 | |
| 287 |
