- 4506 Views
- 5 replies
- 4 kudos
Resolved! Databricks Community version - Unable to clone a public git repository
Databricks Community version - Unable to clone a public git repository, as the 'Repository' tab that should appear below 'Workspace' tab on the portal does not appear and I am not aware of any alternate method. I have referred to some documents on th...
- 4506 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @Jay Kumar​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wi...
- 4 kudos
- 3264 Views
- 2 replies
- 1 kudos
Resolved! Is there a way to run Databricks query on vba?
Currently, I just manually copy paste ​a code from an excel sheet and paste it on a databricks notebook and run for results, then, copy paste the results to the same workbook. I'm sure there's a faster way to do it. The only solutions i can find is u...
- 3264 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Hanna Wade​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 1 kudos
- 3154 Views
- 6 replies
- 5 kudos
How to delete strings from the /FileStore/
We have just started working with databricks in one of my university modules, and the lecturers gave us a set of commands to practice saving data in the FileStore. One of the commands was the following:dbutils .fs.cp("/ databricks - datasets / weathh...
- 3154 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @Konrad Kawka​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest ...
- 5 kudos
- 1917 Views
- 1 replies
- 0 kudos
SET configuration in SQL DLT pipeline not working
I'm not able to get the SET command to work when using sql in DLT pipeline. I am copying the code from this documentation https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-sql-ref.html#sql-spec (relevant code below). When I ru...
- 1917 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Oliver Teng​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 0 kudos
- 3368 Views
- 7 replies
- 0 kudos
Resolved! Fetch new data from kinesis for every minute.
I want to fetch new data from kinesis source for every minute. I'm using "minFetchPeriod" option and specified 60s. But this doesn't seem to be working.Streaming query: spark \ .readStream \ .format("kinesis") \ .option("streamName", kinesis_stream_...
- 3368 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @Pranathi Girish​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedb...
- 0 kudos
- 4918 Views
- 1 replies
- 1 kudos
read cdm error: java.util.NoSuchElementException: None.get
Hi all, I'm trying to read cdm file and get the error in the image (I replaced the names in uppercase). Any ideas on how to solve it?Thank you!
- 4918 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @imma marra​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 1905 Views
- 2 replies
- 1 kudos
Resolved! Declaring parameters - SQL options
Hello. Following an older question SQL Declare Variable equivalent in databricks, we managed to find through the following article Converting Stored Procedures to Databricks | by Ryan Chynoweth | Dec, 2022 | Medium, a way to declaring more complicate...
- 1905 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ELENI GEORGOUSI​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 1 kudos
- 1686 Views
- 1 replies
- 2 kudos
Resolved! Ray as a cluster library instead of notebook-scoped library
This article rightly suggests to install `ray` with `%pip`, although it fails to mention that installing it as a cluster library won't work.The reason, I think, is that `setup_ray_cluster` will use `sys.executable` (ie `/local_disk0/.ephemeral_nfs/en...
- 1686 Views
- 1 replies
- 2 kudos
- 2 kudos
Ugly, but this seems to work for nowimport sys import os import shutil from ray.util.spark import setup_ray_cluster, shutdown_ray_cluster shutil.copy( "/local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/ray", os.path.dirname(sys.execu...
- 2 kudos
- 3401 Views
- 2 replies
- 4 kudos
Resolved! enabling the cell in Databricks.
Suppose if the query is long one and its commented due to some issues later I wanted to run that cell, is there any shortcut to enable the entire cell. The cell is with 800 lines of code and each line is commented with # symbol and I want to enable i...
- 3401 Views
- 2 replies
- 4 kudos
- 4 kudos
@KVNARK .​ Hi,If I understand correctly, you want to either comment or disable comment on your entire cell using a Shortcut.If that's the case, To do for the whole cell -> Do a Ctrl + A, then you can use Ctrl + / in Windows. It will add # to all your...
- 4 kudos
- 4110 Views
- 3 replies
- 1 kudos
Error while trying to implement Change Data Capture
Hi All,I'm new to databricks and learning towards taking up Associate Engineer Certification.While going through the section "Build Data Pipelines with Delta Live Tables". I'm trying to implement Change Data Capture, but it is erroring out when execu...
- 4110 Views
- 3 replies
- 1 kudos
- 1 kudos
Having had a quick look, I think your error is because you are trying to add SCD to a STREAMING LIVE table. I believe APPLY CHANGES INTO cannot be used on a streaming table.You can use a streaming table as a source though.Simply changing this line:C...
- 1 kudos
- 4285 Views
- 4 replies
- 3 kudos
Resolved! databricks-connect 11.3
Would there be databricks-connect for cluster version 11.3 ?If yes, when we should expect it?
- 4285 Views
- 4 replies
- 3 kudos
- 3 kudos
It looks like there are other issues. I saved the model generated with the code above in mlflowWhen I try to reload it with this code:import mlflow model = mlflow.spark.load_model('runs:/cb6ff62587a0404cabeadd47e4c9408a/model') It works in a notebook...
- 3 kudos
- 3272 Views
- 2 replies
- 1 kudos
Issue with inserting multiple rows in Delta table with identity column
Running DBR 11.3 / Azure DatabricksTable definition below:%sql CREATE OR REPLACE TABLE demo2 ( id BIGINT GENERATED BY DEFAULT AS IDENTITY, product_type STRING, sales BIGINT ) USING DELTA LOCATION '/folderlocation/' TBLPROPERTIES ( 'delta.column...
- 3272 Views
- 2 replies
- 1 kudos
- 1 kudos
Just updating it is possible this issue has now been addressed.As before working on Azure Databricks 11.3 DBRInserting into managed table:Also appears to be addressed for autoloader insertion into unmanaged table
- 1 kudos
- 1258 Views
- 1 replies
- 7 kudos
Starting from databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function t...
Starting from databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function takes an array column as input and returns a new row for each element in the array, offering new poss...
- 1258 Views
- 1 replies
- 7 kudos
- 7 kudos
It's very useful for SQL developers.
- 7 kudos
- 3634 Views
- 2 replies
- 0 kudos
Overwriting the existing table in Databricks; Mechanism and History?
Hi,Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table. (DF.write .format("delta") .mode("overwrite") .option("...
- 3634 Views
- 2 replies
- 0 kudos
- 0 kudos
the overwrite will add new files, keep the old ones and in a log keeps track of what is current data and what is old data.If the overwrite fails, you will get an error message in the spark program, and the data to be overwritten will still be the cur...
- 0 kudos
- 6913 Views
- 2 replies
- 0 kudos
Resolved! Convert date to integer
Hello. Is there a way in Databricks sql to convert a date to integer? In Db2, there is days function DAYS - IBM Documentation .For example '2023-03-01' is converted to 738580 value.Thank you in advance
- 6913 Views
- 2 replies
- 0 kudos
- 0 kudos
TRy this:CREATE OR REPLACE FUNCTION days(dt DATE) RETURN unix_date(dt) - unix_date(DATE'0001-01-01') + 1;SELECT current_date, days(current_date); 2023-03-09 738588I verified on Db2 for LUW and it matches up.
- 0 kudos