cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

LearnDB1234
by New Contributor III
  • 1081 Views
  • 4 replies
  • 0 kudos

How to store SQL query output columns as variables to be used as parameters for API data call in DAT

I have a sql query which provides me with the below output :Select FirstName,LastName,Title From Default.Name Tony Gonzalez Mr Tom Brady Mr Patricia Carroll MissI would like to store FirstName, LastName & title column output...

  • 1081 Views
  • 4 replies
  • 0 kudos
Latest Reply
Brahmareddy
Esteemed Contributor
  • 0 kudos

Hi @LearnDB1234 ,Here is the approach - You can make your API call dynamic by first running your SQL query and storing the results in a DataFrame. Then, you can loop through each row in the DataFrame and extract the FirstName and LastName values, pas...

  • 0 kudos
3 More Replies
SOlivero
by New Contributor III
  • 616 Views
  • 1 replies
  • 0 kudos

Scheduling Jobs with Multiple Git Repos on a Single Job Cluster

Hi,I'm trying to create a scheduled job that runs notebooks from three different repos. However, since a job can only be associated with one repo, I've had to create three separate jobs and a master job that triggers them sequentially.This setup work...

  • 616 Views
  • 1 replies
  • 0 kudos
Latest Reply
Brahmareddy
Esteemed Contributor
  • 0 kudos

Hi @SOlivero ,Try configuring a shared all-purpose cluster and set each job to use this existing cluster rather than creating new job-specific clusters, ensuring the cluster stays warm and avoiding startup delays. Another option is to restructure you...

  • 0 kudos
Erik
by Valued Contributor III
  • 5233 Views
  • 6 replies
  • 4 kudos

Resolved! Powerbi databricks connector should import column description

I posted this idea in ideas.powerbi.com as well, but it is quite unclear to me whether the powerbi databricks connector is in fact made by MS or Databricks, so I post it here as well!It is possible to add comments/descriptions to databricks database ...

  • 5233 Views
  • 6 replies
  • 4 kudos
Latest Reply
capstone
New Contributor II
  • 4 kudos

You can use this C# script in Tabular Editor to achieve this. Basically, all the comments can be accessed via the 'information_schema' in Databricks. Import the relevant columns from the schema using this query select * from samples.information_schem...

  • 4 kudos
5 More Replies
NaeemS
by New Contributor III
  • 1723 Views
  • 2 replies
  • 0 kudos

Handling Aggregations in Feature Function

Hi,Is it possible to cater aggregation using Feature Functions somehow. As we know that the logic defined in feature function is applied on a single row when a join is being performed. But do we have any mechanism to handle to aggregations too someho...

Data Engineering
Feature Functions
Feature Store
  • 1723 Views
  • 2 replies
  • 0 kudos
Latest Reply
rafaelsass
New Contributor II
  • 0 kudos

Hi @NaeemS !Have you managed to achieve this by any means? I'm facing the same question right now.

  • 0 kudos
1 More Replies
Paul92S
by New Contributor III
  • 12394 Views
  • 6 replies
  • 5 kudos

Resolved! DELTA_EXCEED_CHAR_VARCHAR_LIMIT

Hi,I am having an issue of loading source data into a delta table/ unity catalog. The error we are recieving is the following:grpc_message:"[DELTA_EXCEED_CHAR_VARCHAR_LIMIT] Exceeds char/varchar type length limitation. Failed check: (isnull(\'metric_...

  • 12394 Views
  • 6 replies
  • 5 kudos
Latest Reply
willflwrs
New Contributor III
  • 5 kudos

Setting this config change before making the write command solved it for us:  spark.conf.set("spark.sql.legacy.charVarcharAsString", True) 

  • 5 kudos
5 More Replies
DataEnginerrOO1
by New Contributor II
  • 2685 Views
  • 5 replies
  • 0 kudos

Access for delta lake with serverless

I have an issue when trying to use the command display(dbutils.fs.ls("abfss://test@test.dfs.core.windows.net")). When I execute the command on my personal cluster, it works, and I can see the files. Before that, I set the following configurations:spa...

  • 2685 Views
  • 5 replies
  • 0 kudos
Latest Reply
Rjdudley
Honored Contributor
  • 0 kudos

Can your serverless compute access any storage in that storage account?  Something else to check is if your NCC is configured correctly: Configure private connectivity from serverless compute - Azure Databricks | Microsoft Learn.  However, if your se...

  • 0 kudos
4 More Replies
akshay716
by New Contributor III
  • 2174 Views
  • 7 replies
  • 1 kudos

Resolved! How to create Service Principal and access APIs like clusters list without adding to admin group

I have created a Databricks Managed Service Principal and trying to access the APIs like clusters list, job lists pipelines but without adding it to admin group I am getting empty list in response. There are other ways to get clusters by adding polic...

  • 2174 Views
  • 7 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Only admin access through account console can be given not read only unfortunately.

  • 1 kudos
6 More Replies
sachamourier
by Contributor
  • 2422 Views
  • 4 replies
  • 0 kudos

Use init script for Databricks job cluster via Azure Data Factory

Hello,I would like to install some libraries (both public and private) on a job cluster. I am using Azure Data Factory to run my Databricks notebooks and hence would like to use job clusters to run these jobs.I have passed my init script to the job c...

adf_init_script_config.png init_script.png
  • 2422 Views
  • 4 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @sachamourier, Have you considered using cluster libraries? The behavior you are observing you require additional debugging since init script is installed successfully, can you enable cluster logging and research through the logs: https://docs.dat...

  • 0 kudos
3 More Replies
staskh
by New Contributor III
  • 2743 Views
  • 1 replies
  • 1 kudos

Resolved! TIMESTAMP(NANOS,false) error

Hi,I'm getting Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false)) error while trying to read a parquet file (generated outside of DataBricks). Unfortunately, due to security configuration, I do not have the ability to read it with pandas or similar...

  • 2743 Views
  • 1 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Hi @staskh That error happens because data type is not supported natively. You can try with below spark setting: spark.conf.set(“spark.sql.legacy.parquet.nanosAsLong”, “true”)  

  • 1 kudos
Mauro
by New Contributor II
  • 2537 Views
  • 0 replies
  • 0 kudos

DLT change in hive metastore destination to unity catalog

A change recently came out in which Databricks necessarily requires using the Unity Catalog as the output of a DLT because previously it was HiveMetaStore. At first I was working using CDC plus expectations which resulted in the "allow_expectations_c...

  • 2537 Views
  • 0 replies
  • 0 kudos
LearningDatabri
by Contributor II
  • 14504 Views
  • 11 replies
  • 6 kudos

Resolved! Bootstrap Timeout during cluster start

sometimes while starting a cluster I am facing bootstrap timeout error, what is the reason? when I try the next time it starts the cluster.

  • 14504 Views
  • 11 replies
  • 6 kudos
Latest Reply
Amine8089
New Contributor II
  • 6 kudos

i have this issue, can someone help on that ?Instance bootstrap failed command: Bootstrap_e2e Instance bootstrap inferred timeout reason: Command_UpdateWorker_Slow Failure message (may be truncated): Bootstrap is terminated spontaneously by VM becaus...

  • 6 kudos
10 More Replies
narvinya
by New Contributor
  • 3518 Views
  • 1 replies
  • 0 kudos

What is the best approach to use Delta tables without Unity Catalog enabled?

Hello!I would like to work with delta tables outside of Databricks UI notebook. I know that the best option would be to use databricks-connect but I don’t have Unity Catalog enabled.What would be the most effective way to do so? I know that via JDBC ...

  • 3518 Views
  • 1 replies
  • 0 kudos
Latest Reply
NanthakumarYoga
New Contributor II
  • 0 kudos

Programatically, you can go for DeltaTables.forPath ( not forName which require Unity Catalog )... This works

  • 0 kudos
mjedy78
by New Contributor II
  • 1074 Views
  • 3 replies
  • 0 kudos

Transition from partitioned table to Liquid clustered table

Hi all,I have a table called classes, which is already partitioned on three different columns. I want to create a Liquid Clustered Table, but as far as I understand from the documentation—and from Dany Lee and his team—it was not possible as of 2024 ...

  • 1074 Views
  • 3 replies
  • 0 kudos
Latest Reply
MariuszK
Valued Contributor III
  • 0 kudos

I'm sorry, I missed that your tables are partitioned, in this case easiest  way will be to recreate it.

  • 0 kudos
2 More Replies
KSB
by New Contributor
  • 1309 Views
  • 1 replies
  • 0 kudos

databricks

Hi Team,Having excel file in sharepoint folder, and has to insert excel data into SQL table from databricks notebook . can i have clear steps on it. Dont have access to Azure Active Directory.  can anyone gives solution without using AZURE Active Dir...

  • 1309 Views
  • 1 replies
  • 0 kudos
Latest Reply
Stefan-Koch
Valued Contributor II
  • 0 kudos

Hi KSBYou could read direkt with Databricks from Excel Sharepoint with Graph API. Here is one possible way: https://community.databricks.com/t5/data-engineering/load-data-from-sharepoint-site-to-delta-table-in-databricks/td-p/16410However, you need t...

  • 0 kudos
OmarE
by New Contributor II
  • 2770 Views
  • 0 replies
  • 1 kudos

Streamlit Databricks App Compute Scaling

I have a streamlit Databricks app and I’m looking to increase the compute resources. According to the documentation and the current settings, the app is limited to 2 vCPUs and 6 GB of memory. Is there a way to adjust these limits or add more resource...

  • 2770 Views
  • 0 replies
  • 1 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels