- 1465 Views
- 1 replies
- 0 kudos
SOAP API - Connection
HelloWe have a workflow in our team to perform usual monthly tasks to be ran on the first working day of the month.Each of the ~20 users will run a clone of this workflow most likely all around the same time but with different options. Because we don...
- 1465 Views
- 1 replies
- 0 kudos
- 0 kudos
maybe you can set a lock before call SOAP APIpython - Using a Lock with redis-py - Stack Overflow
- 0 kudos
- 1590 Views
- 1 replies
- 0 kudos
create an array sorted by a field
How do i create an array from a field while applying sorting?how do I do this in data brick since databricks does not support order by in array_agg? same is possible in Snowflake and(Array agg) or Redshift(listagg). SELECT ARRAY_AGG(O_ORDERKEY) WITH...
- 1590 Views
- 1 replies
- 0 kudos
- 0 kudos
%sql SELECT array_sort(array_agg(col) ,(left, right) -> CASE WHEN left < right THEN -1 WHEN left > right THEN 1 ELSE 0 END) arr_col FROM VALUES (3), (2), (1) AS tab(col); https://docs.databricks.com/en/sql/language-manual/functions/array_sort.h...
- 0 kudos
- 1591 Views
- 2 replies
- 0 kudos
Python Databricks how to run all cells in another notebook except the last cell
I have a Python Databricks notebook which I want to call/run another Databricks notebook using dbutils.notebook.run()... but I want to run all the cells in the "called" notebook except the last one.Is there a way to do a count of cells in the called ...
- 1591 Views
- 2 replies
- 0 kudos
- 0 kudos
In the alternative way, you can use dbutils.notebook.run to pass the parameters, and use dbutils.widgets.get in another notebook to get the parameter values,and determine the parameter values to decide whether to execute codes in the specified cellh...
- 0 kudos
- 2599 Views
- 1 replies
- 0 kudos
Resolved! Differences Between "TEMPORARY STREAMING TABLE" and "TEMPORARY STREAMING LIVE VIEW" in DLT
Hello Databricks community,I'm seeking clarification on the distinctions between the following two syntaxes:CREATE OR REFRESH TEMPORARY STREAMING TABLECREATE TEMPORARY STREAMING LIVE VIEWAs of my understanding, both of these methods do not store data...
- 2599 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Kai, The two syntaxes you're asking about, CREATE OR REFRESH TEMPORARY STREAMING TABLE and CREATE TEMPORARY STREAMING LIVE VIEW, are used in Delta Live Tables and have distinct purposes. CREATE OR REFRESH TEMPORARY STREAMING TABLE: This syntax i...
- 0 kudos
- 4262 Views
- 5 replies
- 0 kudos
Resolved! Databricks asset bundles job and pipeline
Hello, I'm looking to create a job which is linked to a delta live table. Given the job code like this: my_job_name: name: thejobname schedule: quartz_cron_expression: 56 30 12 * * ? timezone_id: UTC pause_stat...
- 4262 Views
- 5 replies
- 0 kudos
- 0 kudos
@RyHubb You can specify the variable of the ID and it will be materialized at deploy time. No need to do this yourself. An example is at https://github.com/databricks/bundle-examples/blob/24678f538415ab936e341a04fce207dce91093a8/default_python/...
- 0 kudos
- 15367 Views
- 1 replies
- 0 kudos
spark is case sensitive? Spark is not case sensitive by default. If you have same column name in different case (Name, name), if you try to select eit...
spark is case sensitive?Spark is not case sensitive by default. If you have same column name in different case (Name, name), if you try to select either "Name" or "name" column you will get column ambiguity error.There is a way to handle this issue b...
- 15367 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, even though i set the conf to be true, on writing to disk it had exceptions complaining it has duplicate columns.below is the error message org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data to save: branchavailablity....
- 0 kudos
- 5879 Views
- 7 replies
- 0 kudos
Resolved! How to load xml files with spark-xml ?
Hello,I cannot load xml files.First, I tried to install Maven library com.databricks:spark-xml_2.12:0.14.0 as it is told in documentation, but I could not find it. I only have HyukjinKwon:spark-xml:0.1.1-s_2.10, with this one I have this error: DRIVE...
- 5879 Views
- 7 replies
- 0 kudos
- 3945 Views
- 2 replies
- 1 kudos
DBT JOBS FAILING
HI ,we have dbt workflow jobs and its been failing randomly from last few days with below error. is there any known issue for this , any help on the root cause will be helpful.Encountered an error: Runtime Error Database Error __init__() got an une...
- 3945 Views
- 2 replies
- 1 kudos
- 1 kudos
setting dbt-databricks==1.7.3 solved this issue but now we randomly get the below error . it gets fixed after restrating the cluster sometimes. but is there any permanent solution for this ? from dbt.events import types_pb2 File "/databricks/python3/...
- 1 kudos
- 1253 Views
- 1 replies
- 0 kudos
Restore sql editor
any Options restore sql editors query after workspace was accidentally deleted and restored
- 1253 Views
- 1 replies
- 0 kudos
- 0 kudos
@Andyt If the workspace is accidentally deleted, there is not way to retrieve content from SQL editor.
- 0 kudos
- 1829 Views
- 1 replies
- 0 kudos
How to get Workflow status and error description programmatically ?
Hi,I want to take some basic info by running workflow and populate a table with those data. I want to add logic programmatically in a notebook and will run it by attaching it in a task of workflow.Information required to be populated in table:Job idJ...
- 1829 Views
- 1 replies
- 0 kudos
- 0 kudos
@WhistlePodu You can review the jobs API for getting the other fields like jobs status etc
- 0 kudos
- 886 Views
- 1 replies
- 1 kudos
data engineer course materials are throwing error
Your course material for data engineering associate program is throwing error.Please correct the below error: -This is from section 2.2 "ETL with Spark".
- 886 Views
- 1 replies
- 1 kudos
- 1 kudos
@Sangram Can you please confirm if that DBFS exists in the specified location?
- 1 kudos
- 822 Views
- 1 replies
- 0 kudos
Access Databricks Volume through CLI
Hi,I am able to connect to DBFS and transfer files there or download from there. But when I change the path to Volumes, it doesn't work. Even though I created the volume I still get this error message:Error: no such directory: /Volumes/bgem_dev/text_...
- 822 Views
- 1 replies
- 0 kudos
- 0 kudos
@pshuk If you are still facing the issue, can you please share the commands you are using ?May be you need to use the latest CLI version to use Volumes which is a new feature?
- 0 kudos
- 923 Views
- 1 replies
- 0 kudos
Streaming delta table - Performance with incremental refresh
Hi Team,We are hitting performance issues with Streaming live delta table specifically when evaluating large tables of more than 10million rows. What are the workarounds to handle these streaming live tables in an attempt to load these large tables. ...
- 923 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Fnazar When dealing with streaming data, you might end up with many small files, which can be inefficient. Use Delta Lake's OPTIMIZE command to compact files into larger ones and ZORDER to colocate related information in the same set of files. T...
- 0 kudos
- 6856 Views
- 1 replies
- 0 kudos
Databricks Connect V2 - OPENSSL_internal: CERTIFICATE_VERIFY_FAILED
I am getting an error when using Databricks V2 in when running anything relating to databricks-sql-connector/databricks.sql.connect(). Would anyone know how to resolve this issue?Sample Error Message Additional DetailsPython Version 3.11.4Sample Code...
- 6856 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you directly use Databricks connect and validate if it works from CLI?Also, confirm the databrics-connect version please
- 0 kudos
- 3511 Views
- 5 replies
- 2 kudos
Databricks Runtime compatibility error with latest version while reading from (ADLS) Dynamic 365 .
We are trying to establish ingestion from dynamic 365 >> ADLS >> Databricks, While reading information we need to use databricks runtime 6.4 to read the raw data from ADLS into Databricks. Latest databricks runtime couldn’t be used, Need your help to...
- 3511 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @nitinsingh1 Thank you for bringing up this topic, I'm also currently looking into how to ingest exported Dynamics 365 FO data (csv files with CDM) from ADLS into Databricks. Could you share how you achieved this? I'd be very curious to see your a...
- 2 kudos