- 2635 Views
- 2 replies
- 1 kudos
Creating All Purpose Cluster in Data Asset Bundles
There is no resource to create All Purpose Cluster, but I need it, so does it mean I should create it via Terraform or DBX and reference to it, which I dont prefer?
- 2635 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Ayushi_Suthar, Thanks for the quick reply! Where can I see these requests?https://ideas.databricks.com/ideas/DB-I-9451 ?
- 1 kudos
- 8012 Views
- 2 replies
- 1 kudos
Get Job Run Status
Is there a way to get a child Job Run status and show the result within the parent notebook execution?Here is the case: I have a master notebook and several child notebooks. As a result, I want to see which notebook failed: For example Notebook job s...
- 8012 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello, Are you also managing any return status while calling the notebook. Have a look the following reference URL : Run a Databricks notebook from another notebook | Databricks on AWS
- 1 kudos
- 3668 Views
- 2 replies
- 1 kudos
Resolved! How can I enable disk cache in this scenario/
I have a notebook where I read multiple tables from delta lake (let say schema is db) and after that I did some sort of transformation (image enclosed) using all these tables lwith transformations like join,filter etc. After transformation and writin...
- 3668 Views
- 2 replies
- 1 kudos
- 2317 Views
- 1 replies
- 1 kudos
Delta live tables running count output mode?
I have a DLT with a table that I want to contain the running aggregation (for the sake of simplicitly let's assume it's a count) for each value of some key column, using a session window. The input table goes back several years and to clean up aggreg...
- 2317 Views
- 1 replies
- 1 kudos
- 20714 Views
- 5 replies
- 2 kudos
Init script problems | workspace location
At this moment, I'm working on removing Legacy global and cluster-named init scripts due, it will be disabled for all workspaces on 01 Sept.At this moment, I'm facing a strange problem regarding moving init scripts from dbfs to the Workspace location...
- 20714 Views
- 5 replies
- 2 kudos
- 2 kudos
Using the new CLI v0.214, uploading ".sh" file works fine.`databricks workspace import --overwrite --format AUTO --file init_setup /init/user/job/init_setup`
- 2 kudos
- 8730 Views
- 9 replies
- 3 kudos
How to write to Salesforce from Databricks using the spark salesforce library
Hi, Im facing an issue while writing to Salesforce sandbox from Databricks. I have installed the "spark-salesforce_2.12-1.1.4" library and my code is as follows:-df_newLeads.write\ .format("com.springml.spark.salesforce")\ .option("username...
- 8730 Views
- 9 replies
- 3 kudos
- 3 kudos
I made a function that used the code below and returned url, connectionProperties, sfwriteurl ="https://login.salesforce.com/"dom = url.split('//')[1].split('.')[0]session_id, instance = SalesforceLogin(username=connectionProperties['name'], password...
- 3 kudos
- 2888 Views
- 2 replies
- 1 kudos
Migrate a workspace from one AWS account to another AWS account
Hi everyone,We have a Databricks workspace in an AWS account that we need to migrate to a new AWS account.The workspace has a lot of managed tables, workflows, saved queries, notebooks which need to be migrated, so looking for an efficient approach t...
- 2888 Views
- 2 replies
- 1 kudos
- 1 kudos
For a streamlined migration of your Databricks workspace from one AWS account to another, start by exporting notebook, workflow, and saved query configurations using Databricks REST API or CLI. Employ Deep Clone or Delta Sharing for managed table dat...
- 1 kudos
- 3765 Views
- 2 replies
- 2 kudos
Resolved! Variable referencing in EXECUTE IMMEDIATE
Hi all,As part of an on-going exercise to refactor existing T-SQL code into Databricks, we've stumbled into an issue that we can't seem to overcome through Spark SQL.Currently we use dynamic SQL to loop through a number of tables, where we use parame...
- 3765 Views
- 2 replies
- 2 kudos
- 2 kudos
DECLARE OR REPLACE varfield_names1 STRING; SET VAR varfield_names1 = 'field1 STRING'; DECLARE OR REPLACE varsqlstring1 STRING; SET VAR varsqlstring1 = 'CREATE TABLE table1 (PrimaryKey STRING, Table STRING, ' || varfield_names1 || ')'; EXECUTE IMMEDI...
- 2 kudos
- 5651 Views
- 2 replies
- 0 kudos
withColumnRenamed error on Unity Catalog 14.3 LTS
Hi - We are migrating to Unity Catalog 14.3 LTS and have seen a change in behavior using withColumnRenamed.There is an error COLUMN_ALREADY_EXISTS on the join key, even though the column being renamed is a different column. The joined DataFrame do...
- 5651 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @ksamborn I can think of 2 solutions:Rename the column in df_2 before joining: df_1_alias = df_1.alias("t1") df_2_alias = df_2.alias("t2") join_df = df_1_alias.join(df_2_alias, df_1_alias.key == df_2_alias.key) rename_df = join_df.withColumnRenam...
- 0 kudos
- 2431 Views
- 2 replies
- 0 kudos
Resolved! Unit tests in notebook not working
Hello, I'm trying to setup a notebook for tests or data quality checks. The name is not important.I basically read a table (the ETL output process - actual data).Then I read another table and do the calculation in the notebook (expected data)I'm stuc...
- 2431 Views
- 2 replies
- 0 kudos
- 0 kudos
thank you for the nutter. Tried it and it seems to answer my problematic.
- 0 kudos
- 2533 Views
- 0 replies
- 0 kudos
Unable to use SQL UDF
Hello, I want to create an sql udf as follows:%sqlCREATE or replace FUNCTION get_type(s STRING) RETURNS STRING LANGUAGE PYTHON AS $$ def get_type(table_name): from pyspark.sql.functions import col from pyspark.sql import SparkSession ...
- 2533 Views
- 0 replies
- 0 kudos
- 2577 Views
- 2 replies
- 1 kudos
Resolved! DLT cluster : can be manipulated?
Hi All I am using DLT pipeline to pull data from a ADLS gen2 which is mounted. The cluster that is getting fired has access mode is set to shared. I want to change it to single user. But the cluster being attached to DLT , I am not ablr to update and...
- 2577 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @kiko_roy Greetings! You can't use a single-user cluster to query tables from a Unity Catalog-enabled Delta Live Tables pipeline, including streaming tables and materialized views in Databricks SQL. To access these tables, you need to use a share...
- 1 kudos
- 1371 Views
- 1 replies
- 0 kudos
Resolved! Liquid Cluster - SHOW CREATE TABLE error
We've got this table with liquid clustering turned on at first, but then we switch off with below command.ALTER TABLE table_name CLUSTER BY NONE;Now, our downstream process that usually runs "SHOW CREATE TABLE" is hitting a snag. It's throwing this e...
- 1371 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ss6 , Hope you are doing well! We would like to inform you that currently, SHOW CREATE TABLE is not supported after running ALTER TABLE CLUSTER BY NONE. This is a known issue and our Engineering team is prioritizing a fix to retain the clusterin...
- 0 kudos
- 22063 Views
- 2 replies
- 2 kudos
Using Plotly Dash with Databricks
How to use Plotly Dash with DatabricksWe recommend checking out this article for the latest on building Dash Applications on top of the Databricks Lakehouse.Let us know in the comments if you use Plotly and if you're planning on adopting the latest i...
- 22063 Views
- 2 replies
- 2 kudos
- 2 kudos
Hey all. Just wanted to make sure everyone had some up-to-date intel regarding leveraging Plotly Dash with Databricks.Most Dash app integrations w Databricks today leverage the Databricks Python SQL Connector. More technical details are available v...
- 2 kudos
- 11587 Views
- 1 replies
- 0 kudos
Parallelizing processing of multiple spark dataframes
Hi all, I am trying to create a collection rd that contains a list of spark dataframes. I want to parallelize the cleaning process for each of these dataframes. Later on, I am sending each of these dataframes to another method. However, when I parall...
- 11587 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
