cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

amitca71
by Contributor II
  • 6519 Views
  • 5 replies
  • 4 kudos

Resolved! exception when using java SQL client

Hi,I try to use java sql. i can see that the query on databricks is executed properly.However, on my client i get exception (see below).versions:jdk: jdk-20.0.1 (tryed also with version 16, same results)https://www.oracle.com/il-en/java/technologies/...

  • 6519 Views
  • 5 replies
  • 4 kudos
Latest Reply
xebia
New Contributor II
  • 4 kudos

I am using java 17 and getting the same error.

  • 4 kudos
4 More Replies
HaripriyaP
by New Contributor
  • 562 Views
  • 1 replies
  • 0 kudos

Multiple Notebooks Migration from one workspace to another without using Git.

Hi all!I need to migrate multiple notebooks from one workspace to another. Is there any way to do it without using Git?Since Manual Import and Export is difficult to do for multiple notebooks and folders, need an alternate solution.Please reply as so...

  • 562 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@HaripriyaP You can use databricks CLI to export and import notebooks from one workspace to another.CLI Documentation here:https://github.com/databricks/cli/blob/main/docs/commands.md#databricks-workspace-export---export-a-workspace-object

  • 0 kudos
HaripriyaP
by New Contributor
  • 936 Views
  • 1 replies
  • 0 kudos

Multiple Tables Migration from one workspace to another.

Hi all!I need to copy multiple tables from one workspace to another with metadata information. Is there any way to do it?Please reply as soon as possible.

  • 936 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Databricks Employee
  • 0 kudos

@HaripriyaP - Depends on your use case, Either of the below approach can be chosen. 1)  DELTA CLONE(DEEP CLONE) to clone them to the new workspace. 2) Have the same cluster policy/Instance profile of the old workspace to access them in the new worksp...

  • 0 kudos
bozhu
by Contributor
  • 2001 Views
  • 3 replies
  • 0 kudos

Delta Live Tables Materialised View Column Comment Error

While materialised view doc says MVs support columns comments, this does not seem like the case for MVs created by DLT. For example, when trying to add a comment to a MV created by DLT, it errors:Any ideas on when this will be fixed/supported?

bozhu_0-1692702233893.png
  • 2001 Views
  • 3 replies
  • 0 kudos
Latest Reply
bozhu
Contributor
  • 0 kudos

Just to close the loop here that it seems DLT generated MVs now support column comments.

  • 0 kudos
2 More Replies
toolhater
by New Contributor II
  • 695 Views
  • 1 replies
  • 0 kudos

Installing dlt causing error

I'm trying to use the example in big book of engineering 2nd edition-final.pdf and I had an issue with the statementimport dltSo I created another cell and installed it and I noticed I was getting this error:"dataclass_transform() got an unexpected k...

  • 695 Views
  • 1 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

could you get the full error stack trace please

  • 0 kudos
carlosancassani
by New Contributor III
  • 1858 Views
  • 1 replies
  • 0 kudos

Update DeltaTable on column type ArrayType(): add element to array

Hi all,I need to perform an Update on a Delta Table adding elements to a column of ArrayType(StringType()) which is initialized empty.Before UpdateCol_1 StringType()Col_2 StringType()Col_3 ArrayType()ValVal[ ]After UpdateCol_1 StringType()Col_2 Strin...

Data Engineering
deltatable
Update
  • 1858 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi all,I need to perform an Update on a Delta Table adding elements to a column of ArrayType(StringType()) which is initialized empty.Before UpdateCol_1 StringType()Col_2 StringType()Col_3 ArrayType()ValVal[ ]After UpdateCol_1 StringType()Col_2 Strin...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
yogu
by Honored Contributor III
  • 4079 Views
  • 8 replies
  • 73 kudos

Trying to claim reward points but its not reflecting my points

Hi Team,Can anyone help me why my reward point still showing 0 balance. My databricks community points is not reflecting on reward claim portal.i was login first time . Also I was wait for 3 business days but its still not reflecting

image
  • 4079 Views
  • 8 replies
  • 73 kudos
Latest Reply
Kaizen
Valued Contributor
  • 73 kudos

Can you also share the link for the reward points redemption?

  • 73 kudos
7 More Replies
cszczotka
by New Contributor III
  • 1147 Views
  • 0 replies
  • 0 kudos

Ephemeral storage how to create/mount.

Hi,I'm looking for information how to create/mount ephemeral storage to Databricks driver node in Azure Cloud.  Does anyone have any experience working with ephemeral storage?Thanks,

  • 1147 Views
  • 0 replies
  • 0 kudos
174817
by New Contributor III
  • 1415 Views
  • 2 replies
  • 3 kudos

DataBricks Rust client and/or OpenAPI spec

Hi,I'm looking for a DataBricks client for Rust.  I could only find these SDK implementations.Alternatively, I would be very happy with the OpenAPI spec.  Clearly one exists: the Go SDK implementation contains code to generate itself from such a spec...

Data Engineering
openapi
rust
sdk
unity
  • 1415 Views
  • 2 replies
  • 3 kudos
Latest Reply
feiyun0112
Honored Contributor
  • 3 kudos

Databricks REST API referenceThis reference contains information about the Databricks application programming interfaces (APIs). Each API reference page is presented primarily from a representational state transfer (REST) perspective. Databricks REST...

  • 3 kudos
1 More Replies
my_super_name
by New Contributor II
  • 1341 Views
  • 1 replies
  • 2 kudos

Auto Loader Schema Hint Behavior: Addressing Nested Field Errors

Hello,I'm using the auto loader to stream a table of data and have added schema hints to specify field values.I've observed that when my initial data file is missing fields specified in the schema hint,the auto loader correctly identifies this and ad...

  • 1341 Views
  • 1 replies
  • 2 kudos
Latest Reply
my_super_name
New Contributor II
  • 2 kudos

Hi @Retired_mod Thanks for your help!Your solution works for the initial issue,and I've implemented it first in my code.but it creates a other problem.When we explicitly define the struct hint as 'bbb STRUCT<ccc: INT>',it works until someone adds mor...

  • 2 kudos
RiyazAli
by Valued Contributor
  • 1167 Views
  • 1 replies
  • 0 kudos

Unable to create a record_id column via DLT - Autoloader

Hi Community,I'm trying to load data from the landing zone to the bronze layer via DLT- Autoloader, I want to add a column record_id to the bronze table while I fetch my data. I'm also using file arrival trigger in the workflow to update my table inc...

  • 1167 Views
  • 1 replies
  • 0 kudos
Latest Reply
RiyazAli
Valued Contributor
  • 0 kudos

Hey @Retired_mod  - could you or any body from the community team help me here, please? I've been stuck since quite some time now.

  • 0 kudos
Aidonis
by New Contributor III
  • 17101 Views
  • 3 replies
  • 2 kudos

Resolved! Flatten Deep Nested Struct

Hi All,I have a deeply nested spark dataframe struct something similar to below |-- id: integer (nullable = true) |-- lower: struct (nullable = true) | |-- field_a: integer (nullable = true) | |-- upper: struct (containsNull = true) | | ...

  • 17101 Views
  • 3 replies
  • 2 kudos
Latest Reply
Praveen-bpk21
New Contributor II
  • 2 kudos

@Aidonis You can try this as well:flatten-spark-dataframe · PyPIThis also allows for specific level of flattening.

  • 2 kudos
2 More Replies
SPres
by New Contributor
  • 1028 Views
  • 1 replies
  • 0 kudos

Passing Parameters from Azure Synapse

Hey Community!Just curious if anyone has tried using Azure Synapse for orchestration and passing parameters from Synapse to a Databricks Notebook. My team is testing out Databricks, and I'm replacing Synapse Notebooks with Databricks Notebooks, but I...

  • 1028 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 0 kudos

Hi @SPres You can definitely pass these parameters to databricks notebook also.Please refer below docs - Run a Databricks Notebook with the activity - Azure Data Factory | Microsoft Learn

  • 0 kudos
Chengzhu
by New Contributor
  • 684 Views
  • 0 replies
  • 0 kudos

Databricks Model Registry Notification

Hi community,Currently, I am training models on databricks cluster and use mlflow to log and register models. My goal is to send notification to me when a new version of registered model happens (if the new run achieves some model performance baselin...

Screenshot 2024-04-17 at 1.14.11 PM.png Screenshot 2024-04-17 at 1.13.14 PM.png
  • 684 Views
  • 0 replies
  • 0 kudos
dilkushpatel
by New Contributor II
  • 1752 Views
  • 2 replies
  • 0 kudos

Databricks connecting SQL Azure DW - Confused between Polybase and Copy Into

I see two articles on databricks documentationshttps://docs.databricks.com/en/archive/azure/synapse-polybase.html#language-pythonhttps://docs.databricks.com/en/connect/external-systems/synapse-analytics.html#service-principal Polybase one is legacy o...

Data Engineering
azure
Copy
help
Polybase
Synapse
  • 1752 Views
  • 2 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
I see two articles on databricks documentationshttps://docs.databricks.com/en/archive/azure/synapse-polybase.html#language-pythonhttps://docs.databricks.com/en/connect/external-systems/synapse-analytics.html#service-principal Polybase one is legacy o...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels