- 6782 Views
- 1 replies
- 1 kudos
read cdm error: java.util.NoSuchElementException: None.get
Hi all, I'm trying to read cdm file and get the error in the image (I replaced the names in uppercase). Any ideas on how to solve it?Thank you!
- 6782 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @imma marra​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 3609 Views
- 2 replies
- 1 kudos
Resolved! Declaring parameters - SQL options
Hello. Following an older question SQL Declare Variable equivalent in databricks, we managed to find through the following article Converting Stored Procedures to Databricks | by Ryan Chynoweth | Dec, 2022 | Medium, a way to declaring more complicate...
- 3609 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ELENI GEORGOUSI​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 1 kudos
- 3142 Views
- 1 replies
- 2 kudos
Resolved! Ray as a cluster library instead of notebook-scoped library
This article rightly suggests to install `ray` with `%pip`, although it fails to mention that installing it as a cluster library won't work.The reason, I think, is that `setup_ray_cluster` will use `sys.executable` (ie `/local_disk0/.ephemeral_nfs/en...
- 3142 Views
- 1 replies
- 2 kudos
- 2 kudos
Ugly, but this seems to work for nowimport sys import os import shutil from ray.util.spark import setup_ray_cluster, shutdown_ray_cluster shutil.copy( "/local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/ray", os.path.dirname(sys.execu...
- 2 kudos
- 7409 Views
- 2 replies
- 4 kudos
Resolved! enabling the cell in Databricks.
Suppose if the query is long one and its commented due to some issues later I wanted to run that cell, is there any shortcut to enable the entire cell. The cell is with 800 lines of code and each line is commented with # symbol and I want to enable i...
- 7409 Views
- 2 replies
- 4 kudos
- 4 kudos
@KVNARK .​ Hi,If I understand correctly, you want to either comment or disable comment on your entire cell using a Shortcut.If that's the case, To do for the whole cell -> Do a Ctrl + A, then you can use Ctrl + / in Windows. It will add # to all your...
- 4 kudos
- 7666 Views
- 3 replies
- 1 kudos
Error while trying to implement Change Data Capture
Hi All,I'm new to databricks and learning towards taking up Associate Engineer Certification.While going through the section "Build Data Pipelines with Delta Live Tables". I'm trying to implement Change Data Capture, but it is erroring out when execu...
- 7666 Views
- 3 replies
- 1 kudos
- 1 kudos
Having had a quick look, I think your error is because you are trying to add SCD to a STREAMING LIVE table. I believe APPLY CHANGES INTO cannot be used on a streaming table.You can use a streaming table as a source though.Simply changing this line:C...
- 1 kudos
- 6107 Views
- 2 replies
- 1 kudos
Issue with inserting multiple rows in Delta table with identity column
Running DBR 11.3 / Azure DatabricksTable definition below:%sql CREATE OR REPLACE TABLE demo2 ( id BIGINT GENERATED BY DEFAULT AS IDENTITY, product_type STRING, sales BIGINT ) USING DELTA LOCATION '/folderlocation/' TBLPROPERTIES ( 'delta.column...
- 6107 Views
- 2 replies
- 1 kudos
- 1 kudos
Just updating it is possible this issue has now been addressed.As before working on Azure Databricks 11.3 DBRInserting into managed table:Also appears to be addressed for autoloader insertion into unmanaged table
- 1 kudos
- 2493 Views
- 1 replies
- 7 kudos
Starting from databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function t...
Starting from databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function takes an array column as input and returns a new row for each element in the array, offering new poss...
- 2493 Views
- 1 replies
- 7 kudos
- 7 kudos
It's very useful for SQL developers.
- 7 kudos
- 6275 Views
- 2 replies
- 0 kudos
Overwriting the existing table in Databricks; Mechanism and History?
Hi,Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table. (DF.write .format("delta") .mode("overwrite") .option("...
- 6275 Views
- 2 replies
- 0 kudos
- 0 kudos
the overwrite will add new files, keep the old ones and in a log keeps track of what is current data and what is old data.If the overwrite fails, you will get an error message in the spark program, and the data to be overwritten will still be the cur...
- 0 kudos
- 11536 Views
- 2 replies
- 0 kudos
Resolved! Convert date to integer
Hello. Is there a way in Databricks sql to convert a date to integer? In Db2, there is days function DAYS - IBM Documentation .For example '2023-03-01' is converted to 738580 value.Thank you in advance
- 11536 Views
- 2 replies
- 0 kudos
- 0 kudos
TRy this:CREATE OR REPLACE FUNCTION days(dt DATE) RETURN unix_date(dt) - unix_date(DATE'0001-01-01') + 1;SELECT current_date, days(current_date); 2023-03-09 738588I verified on Db2 for LUW and it matches up.
- 0 kudos
- 13262 Views
- 5 replies
- 3 kudos
Resolved! Unable to import tkinter in notebook
Hello,I am unable to import tkinter (or Tkinter) into a python notebook.I also tried %pip install tkinter at the top of the notebook.Has anyone else been successful at this, or if it's impossible, why? Thank you
- 13262 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Will Griffin​ Can you confirm if this worked for you? I get a message `ERROR: No matching distribution found for python3-tk`.
- 3 kudos
- 5017 Views
- 4 replies
- 0 kudos
I added a secret scope to the databricks environment, and I get this error when trying to run either of the following:
Commands Attempted:dbutils.secrets.listScopes()dbutils.secrets.get(scope = "{InsertScope}", key = "{InsertKey}") Error: "shaded.v245.com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, ...
- 5017 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Haylee Gaddy​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 0 kudos
- 6155 Views
- 6 replies
- 0 kudos
GeneratedAlwaysAs' along with dataframe.write
Is it possible to use a calculated column (as like in the delta table using generatedAlwaysAs) definition while writing the data frame as a delta file like df.write.format("delta").Any options are there with the dataframe.write method to achieve this...
- 6155 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Thushar R​ ,This option is not a part of Dataframe write API as GeneratedAlwaysAs feature is only applicable to Delta format and df.write is a common API to handle writes for all formats. If you to achieve this programmatically, you can still use...
- 0 kudos
- 8632 Views
- 4 replies
- 7 kudos
Resolved! Delta Live Table Schema Comment
I predefined my schema for a Delta Live Table Autoload. This included comments for some attributes. When performing a standard readStream, my comments appear, but when in Delta Live Tables I get no comments. Is there anything I need to do get comment...
- 8632 Views
- 4 replies
- 7 kudos
- 7 kudos
You need to add your schema to dlt declaration:@dlt.table( name="test_bronze", comment = "test account data incrementally ingested from S3 Raw landing zone", table_properties={ "quality": "bronze" }, schema=schema)
- 7 kudos
- 36417 Views
- 4 replies
- 1 kudos
Resolved! How to get rid of "Command result size exceeds limit"
I am working on Databricks Notebook and trying to display a map using Floium and I keep getting this error > Command result size exceeds limit: Exceeded 20971520 bytes (current = 20973510)How can I get increase the memory limit?I already reduced the...
- 36417 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi, I have the same problem with keplergl, and the save to disk option, whilst helpful isn't super practical... So how does one plot large datasets in kepler?Any thought welcome
- 1 kudos
- 3485 Views
- 2 replies
- 3 kudos
Resolved! Data lineage graph now working
I created the below tables but when I click the lineage graph not able to see the upstream or downstream table .... the + sign goes away after a few sec but not able to click it ... anyone else having this issue?CREATE TABLE IF NOT EXISTS lineage_d...
- 3485 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Raj Sharma​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
