cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Erik
by Valued Contributor II
  • 4885 Views
  • 9 replies
  • 10 kudos

Resolved! How to use dbx for local development.

​Databricks connect is a program which allows you to run spark code locally, but the actual execution happens on a spark cluster. Noticeably, it allows you to debug and step through the code locally in your own IDE. Quite useful. But it is now beeing...

  • 4885 Views
  • 9 replies
  • 10 kudos
Latest Reply
FeliciaWilliam
Contributor
  • 10 kudos

I found answers to my questions here

  • 10 kudos
8 More Replies
Himanshu4
by New Contributor II
  • 957 Views
  • 4 replies
  • 2 kudos

Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API

Dear Databricks Community,I hope this message finds you well. I am currently working on automating cluster configuration updates in Databricks using the API. As part of this automation, I am looking to ensure that the Unity Catalog is enabled within ...

  • 957 Views
  • 4 replies
  • 2 kudos
Latest Reply
Himanshu4
New Contributor II
  • 2 kudos

Hi RaphaelCan we fetch job details from one workspace and create new job in new workspace with the same "job id" and configuration?

  • 2 kudos
3 More Replies
fury-kata
by New Contributor II
  • 543 Views
  • 2 replies
  • 0 kudos

ModuleNotFoundError when run with foreachBatch on serverless mode

I using Notebooks to do some transformations I install a new whl:  %pip install --force-reinstall /Workspace/<my_lib>.whl %restart_python  Then I  successfully import the installed lib  from my_lib.core import test  However when I run my code with fo...

  • 543 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @fury-kata,  Make sure that the path to your custom module is correctly added to the Python path (sys.path). You mentioned installing the .whl file, so ensure that the installation path is accessible from your Databricks notebook.Verify that th...

  • 0 kudos
1 More Replies
wilco
by New Contributor II
  • 1109 Views
  • 3 replies
  • 0 kudos

SQL Warehouse: Retrieving SQL ARRAY Type via JDBC driver

Hi all,we are currently running into the following issuewe are using serverless SQL warehousein a JAVA application we are using the latest Databricks JDBC driver (v2.6.36)we are querying the warehouse with a collect_list function, which should return...

  • 1109 Views
  • 3 replies
  • 0 kudos
Latest Reply
KTheJoker
Contributor II
  • 0 kudos

Hey Wilco, The answer is no, ODBC/JDBC don't support complex types so these need to be compressed into strings over the wire (usually in JSON representation) and rehydrated on the client side into a complex object.

  • 0 kudos
2 More Replies
source2sea
by Contributor
  • 2330 Views
  • 2 replies
  • 0 kudos

Resolved! ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)

ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)should one create it in the work space manually via UI? and how?would it get overwritten if work space is created via terraform?I use 10.4 LTS runtime.

  • 2330 Views
  • 2 replies
  • 0 kudos
Latest Reply
ashish2007g
New Contributor II
  • 0 kudos

I am experiencing significant delay on my streaming. I am using changefeed connector. Its processing streaming batch very frequently but experiencing sudden halt and shows no active stage for longer time. I observed below exception continuously promp...

  • 0 kudos
1 More Replies
kskistad
by New Contributor III
  • 3406 Views
  • 3 replies
  • 4 kudos

Resolved! Streaming Delta Live Tables

I'm a little confused about how streaming works with DLT. My first questions is what is the difference in behavior if you set the pipeline mode to "Continuous" but in your notebook you don't use the "streaming" prefix on table statements, and simila...

  • 3406 Views
  • 3 replies
  • 4 kudos
Latest Reply
Harsh141220
New Contributor II
  • 4 kudos

Is it possible to have custom upserts in streaming tables in a delta live tables pipeline?Use case: I am trying to maintain a valid session based on timestamp column and want to upsert to the target table.Tried going through the documentations but dl...

  • 4 kudos
2 More Replies
PearceR
by New Contributor III
  • 7481 Views
  • 3 replies
  • 1 kudos

Resolved! custom upsert for delta live tables apply_changes()

Hello community :).I am currently implementing some pipelines using DLT. They are working great for my medalion architecture for landed json in bronze -> silver (using apply_changes) then materialized gold views ontop.However, I am attempting to crea...

  • 7481 Views
  • 3 replies
  • 1 kudos
Latest Reply
Harsh141220
New Contributor II
  • 1 kudos

Is it possible to have custom upserts for streaming tables in delta live tables?Im getting the error:pyspark.errors.exceptions.captured.AnalysisException: `blusmart_poc.information_schema.sessions` is not a Delta table.

  • 1 kudos
2 More Replies
sreeyv
by New Contributor II
  • 403 Views
  • 2 replies
  • 0 kudos

Unable to execute update statement through Databricks Notebook

I am unable to execute update statements through Databricks Notebook, getting this error message "com.databricks.sql.transaction.tahoe.actions.InvalidProtocolVersionException: Delta protocol version is too new for this version of the Databricks Runti...

  • 403 Views
  • 2 replies
  • 0 kudos
Latest Reply
sreeyv
New Contributor II
  • 0 kudos

This is resolved, this happens when a Column in the table has a GENERATED BY DEFAULT AS IDENTITY defined. When you remove this column, it works fine

  • 0 kudos
1 More Replies
tariq
by New Contributor III
  • 1702 Views
  • 2 replies
  • 0 kudos

SqlContext in DBR 14.3

I have a Databricks workspace in GCP and I am using the cluster with the Runtime 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am trying to set the checkpoint directory location using the following command in a notebook:spark.sparkContext.set...

  • 1702 Views
  • 2 replies
  • 0 kudos
Latest Reply
sudipto84goswam
New Contributor II
  • 0 kudos

Is this fixed on 14.3 DBR. I am also facing the same when using in Azure Databricks. I am just using df.rdd.getNumPartitions() 

  • 0 kudos
1 More Replies
deepu
by New Contributor II
  • 797 Views
  • 1 replies
  • 1 kudos

performance issue with SIMBA ODBC using SSIS

i was trying to upload data into a table in hive_metastore using SSIS using SIMBA ODBC driver. The data set is huge (1.2 million records and 20 columns) , it is taking more than 40 mins to complete. is there an config change to improve the load time.

  • 797 Views
  • 1 replies
  • 1 kudos
Latest Reply
NandiniN
Honored Contributor
  • 1 kudos

Looks like a slow data upload into a table in hive_metastore using SSIS and the SIMBA ODBC driver. This could be due to a variety of factors, including the size of your dataset and the configuration of your system. One potential solution could be to ...

  • 1 kudos
Ramseths
by New Contributor
  • 459 Views
  • 1 replies
  • 0 kudos

Wrong Path Databricks Repos

In a Databricks environment, I have cloned a repository that I have in Azure DevOps Repos, the repository is inside the path:Workspace/Repos/<user_mail>/my_repo.Then when I create a Python script that I want to call in a notebook using an import: imp...

  • 459 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Honored Contributor
  • 0 kudos

Hi @Ramseths , If your notebook and script are in the same path, it would have picked the same relative path. Is your notebook located in /databricks/driver? Thanks!

  • 0 kudos
JonLaRose
by New Contributor III
  • 1511 Views
  • 3 replies
  • 0 kudos

Adding custom Jars to SQL Warehouses

Hi there,I want to add custom JARs to an SQL warehouse (Pro if that matters) like I can in an interactive cluster, yet I don't see a way.Is that a degraded functionality when transitioning to a SQL warehouse, or have I missed something? Thank you. 

  • 1511 Views
  • 3 replies
  • 0 kudos
Latest Reply
JunYang
New Contributor III
  • 0 kudos

ADD JAR is a SQL syntax for Databricks runtime, it does not work for DBSQL/warehouse. DBSQL would throw this error: [NOT_SUPPORTED_WITH_DB_SQL] LIST JAR(S) is not supported on a SQL warehouse. SQLSTATE: 0A000. This feature is not supported as of now....

  • 0 kudos
2 More Replies
pavel_merkle
by New Contributor II
  • 1293 Views
  • 3 replies
  • 0 kudos

Databrikcs SDK - create new job using JSON

Hello,I am trying to create a Job via Databricks SDK. As input, I use the JSON generated via Workflows UI (Worklflows->Jobs->View YAML/JSON->JSON API->Create) generating pavel_job.json. When trying to run SDK function jobs.create asdbk = WorkspaceCli...

  • 1293 Views
  • 3 replies
  • 0 kudos
Latest Reply
mhiltner
New Contributor III
  • 0 kudos

it does work for me, using directly on a notebook cell (havent tried in vscode).  This is my sample Json  { "name": "Agregacao_Gold", "email_notifications": {}, "webhook_notifications": {}, "timeout_seconds": 0, "max_concurrent_runs": 1, ...

  • 0 kudos
2 More Replies
leungi
by New Contributor III
  • 1221 Views
  • 6 replies
  • 1 kudos

Resolved! Unable to add column comment in Materialized View (MV)

The following doc suggests the ability to add column comments during MV creation via the `column list` parameter.Thus, the SQL code below is expected to generate a table where the columns `col_1` and `col_2` are commented; however, this is not the ca...

  • 1221 Views
  • 6 replies
  • 1 kudos
Latest Reply
raphaelblg
Honored Contributor
  • 1 kudos

@leungi you've shared the python language reference. This is the SQL Reference from where I've based my example.

  • 1 kudos
5 More Replies
Marcin_U
by New Contributor II
  • 216 Views
  • 1 replies
  • 0 kudos

Making transform on pyspark.sql.Column object outside DataFrame.withColumn method

Hello,I made some transform on pyspark.sql.Column object: file_path_splitted=f.split(df[filepath_col_name],'/') # return Column object file_name = file_path_splitted[f.size(file_path_splitted) - 1] # return Column object Next I used variable "file_na...

  • 216 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Honored Contributor
  • 0 kudos

Hello @Marcin_U , Thank you for reaching out. The transformation you apply within or outside the `withColumn` method will ultimately result in the same Spark plan. The answer is no, it's not possible to have rows mismatch if you're referring to the s...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels