cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Snoonan
by Contributor
  • 2308 Views
  • 6 replies
  • 0 kudos

Resolved! Unity catalog issues

Hi all,I have recently enabled Unity catalog in my DBX workspace. I have created a new catalog with an external location on Azure data storage.I can create new schemas(databases) in the new catalog but I can't create a table. I get the below error wh...

  • 2308 Views
  • 6 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@Snoonan First of all, check the networking tab on the storage account to see if it's behind firewall. If it is, make sure that Databricks/Storage networking is properly configured (https://learn.microsoft.com/en-us/azure/databricks/security/network/...

  • 0 kudos
5 More Replies
Carlton
by New Contributor III
  • 312 Views
  • 1 replies
  • 0 kudos

Help Refactor T-SQL Code to Databricks SQL

Hello CommunityCan someone help refactor the following T-SQL Code to Databricks SQLCONVERT(DECIMAL(26, 8), ISNULL(xxx.xxxxxxx * ISNULL(RH.xxxxx, 1 / NULLIF(ST.xxxxxx, 0)), ST.xxxxx)) AS AmountWhen I attempt to execute the above code I get the followi...

  • 312 Views
  • 1 replies
  • 0 kudos
Latest Reply
AmanSehgal
Honored Contributor III
  • 0 kudos

You can use CAST instead.Eg:SELECT cast('2024' as int);

  • 0 kudos
jfvizoso
by New Contributor II
  • 5594 Views
  • 4 replies
  • 0 kudos

Can I pass parameters to a Delta Live Table pipeline at running time?

I need to execute a DLT pipeline from a Job, and I would like to know if there is any way of passing a parameter. I know you can have settings in the pipeline that you use in the DLT notebook, but it seems you can only assign values to them when crea...

  • 5594 Views
  • 4 replies
  • 0 kudos
Latest Reply
Mustafa_Kamal
New Contributor II
  • 0 kudos

Hi @jfvizoso ,I also have the same scenario, did you find any work around.Thanks in advance.

  • 0 kudos
3 More Replies
LorenRD
by Contributor
  • 6346 Views
  • 8 replies
  • 9 kudos
  • 6346 Views
  • 8 replies
  • 9 kudos
Latest Reply
miranda_luna_db
Contributor II
  • 9 kudos

Hi friends -  To confirm, with new lakeview dashboards you can share dashboards to users and groups in your organization without having to provide any workspace and/or compute access.  https://docs.databricks.com/en/dashboards/index.html#what-is-shar...

  • 9 kudos
7 More Replies
andre_rizzatti
by New Contributor II
  • 911 Views
  • 3 replies
  • 0 kudos

Ingest __databricks_internal catalog - PERMISSION DENIED

Good morning, I have a DLT process with CDC incremental load and I need to ingest the history as CDC transactions are only recent. To do this I need to ingest data in the __databricks_internal catalog. In my case, as I am full admin, I can do it, how...

image.png
  • 911 Views
  • 3 replies
  • 0 kudos
Latest Reply
andre_rizzatti
New Contributor II
  • 0 kudos

The tables do not have specific configuration, and the user who is receiving the error is in a group that has full permission in the INTERNAL catalog

  • 0 kudos
2 More Replies
amde99
by New Contributor
  • 516 Views
  • 2 replies
  • 0 kudos

How can I throw an exception when a .json.gz file has multiple roots?

I have a situation where source files in .json.gz sometimes arrive with invalid syntax containing multiple roots separated by empty braces []. How can I detect this and thrown an exception? Currently the code runs and picks up only record set 1, and ...

  • 516 Views
  • 2 replies
  • 0 kudos
Latest Reply
Lakshay
Esteemed Contributor
  • 0 kudos

Schema validation should help here.

  • 0 kudos
1 More Replies
amitca71
by Contributor II
  • 4158 Views
  • 5 replies
  • 4 kudos

Resolved! exception when using java SQL client

Hi,I try to use java sql. i can see that the query on databricks is executed properly.However, on my client i get exception (see below).versions:jdk: jdk-20.0.1 (tryed also with version 16, same results)https://www.oracle.com/il-en/java/technologies/...

  • 4158 Views
  • 5 replies
  • 4 kudos
Latest Reply
xebia
New Contributor II
  • 4 kudos

I am using java 17 and getting the same error.

  • 4 kudos
4 More Replies
drag7ter
by New Contributor III
  • 481 Views
  • 1 replies
  • 0 kudos

Configure Service Principle access to GiLab

I'm facing an issue while trying to run my job in db and my notebooks located in Git Lab. When I run job under my personal user_Id it works fine, because I added Git Lab token to my user_Id profile and job able to pull branch from repository. But whe...

  • 481 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @drag7ter, There might be a missing piece in the setup. Ensure that you’ve correctly entered the Git provider credentials (username and personal access token) for your Service Principle.Confirm that you’ve selected the correct Git provider (GitLab...

  • 0 kudos
cszczotka
by New Contributor III
  • 572 Views
  • 1 replies
  • 0 kudos

Ephemeral storage how to create/mount.

Hi,I'm looking for information how to create/mount ephemeral storage to Databricks driver node in Azure Cloud.  Does anyone have any experience working with ephemeral storage?Thanks,

  • 572 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @cszczotka,  Azure Databricks allows you to mount cloud object storage to the Databricks File System (DBFS) to simplify data access patterns for users who are unfamiliar with cloud concepts. Mounted data does not work with Unity Catalog, and Dat...

  • 0 kudos
mvmiller
by New Contributor III
  • 665 Views
  • 1 replies
  • 1 kudos

Workflow file arrival trigger - does it apply to overwritten files?

I am exploring the use of the "file arrival" trigger for a workflow for a use case I am working on.  I understand from the documentation that it checks every minute for new files in an external location, then initiates the workflow when it detects a ...

  • 665 Views
  • 1 replies
  • 1 kudos
Latest Reply
Rajani
Contributor
  • 1 kudos

Hi @mvmiller The  "file arrival" trigger for a workflow considers the name of the file,when the same name file was overwritten the workflow didnt triggerred.hope I answered your question! 

  • 1 kudos
dollyb
by New Contributor III
  • 479 Views
  • 1 replies
  • 0 kudos

Differences between Spark SQL and Databricks

Hello,I'm using a local Docker Spark 3.5 runtime to test my Databricks Connect code. However I've come across a couple of cases where my code would work in one environment, but not the other.Concrete example, I'm reading data from BigQuery via spark....

  • 479 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@dollyb That's because when you've added another dependency on Databricks, it doesn't really know which one it should use. By default it's using built-in com.google.cloud.spark.bigquery.BigQueryRelationProvider.What you can do is provide whole packag...

  • 0 kudos
HaripriyaP
by New Contributor
  • 231 Views
  • 1 replies
  • 0 kudos

Multiple Notebooks Migration from one workspace to another without using Git.

Hi all!I need to migrate multiple notebooks from one workspace to another. Is there any way to do it without using Git?Since Manual Import and Export is difficult to do for multiple notebooks and folders, need an alternate solution.Please reply as so...

  • 231 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@HaripriyaP You can use databricks CLI to export and import notebooks from one workspace to another.CLI Documentation here:https://github.com/databricks/cli/blob/main/docs/commands.md#databricks-workspace-export---export-a-workspace-object

  • 0 kudos
HaripriyaP
by New Contributor
  • 315 Views
  • 1 replies
  • 0 kudos

Multiple Tables Migration from one workspace to another.

Hi all!I need to copy multiple tables from one workspace to another with metadata information. Is there any way to do it?Please reply as soon as possible.

  • 315 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@HaripriyaP - Depends on your use case, Either of the below approach can be chosen. 1)  DELTA CLONE(DEEP CLONE) to clone them to the new workspace. 2) Have the same cluster policy/Instance profile of the old workspace to access them in the new worksp...

  • 0 kudos
bozhu
by Contributor
  • 1314 Views
  • 4 replies
  • 0 kudos

Delta Live Tables Materialised View Column Comment Error

While materialised view doc says MVs support columns comments, this does not seem like the case for MVs created by DLT. For example, when trying to add a comment to a MV created by DLT, it errors:Any ideas on when this will be fixed/supported?

bozhu_0-1692702233893.png
  • 1314 Views
  • 4 replies
  • 0 kudos
Latest Reply
bozhu
Contributor
  • 0 kudos

Just to close the loop here that it seems DLT generated MVs now support column comments.

  • 0 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors