cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
User16783853501
by • Databricks Employee
  • 2288 Views
  • 2 replies
  • 1 kudos

using Spark SQL or particularly %SQL in a databricks notebook, is there a way to use pagination or offset or skip ?

using Spark SQL or particularly %SQL in a databricks notebook, is there a way to use pagination or offset or skip ? 

Data Engineering
Reply
  • 2288 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

There is no offset support yet. Here are a few possible workarounds If you data is all in one partition ( rarely the case ) , you could create a column with monotonically_increasing_id and apply filter conditions. if there are multiple partitions...

  • 1 kudos
1 More Replies
Ryan_Chynoweth
by • Esteemed Contributor
  • 3031 Views
  • 1 replies
  • 1 kudos

Can you execute an Azure Databricks Notebook from Azure Data Factory? Can you return values back to the Data Factory from the Notebook?

Data Engineering
Reply
  • 3031 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 1 kudos

Yes, Azure Data Factory can execute code on Azure Databricks. The best way to return values from the notebook to Data factory is to use the dbutils.notebook.exit() function at the end of your notebook or whenever you want to terminate execution.

  • 1 kudos
User16753724663
by • Valued Contributor
  • 2356 Views
  • 1 replies
  • 1 kudos

Download private repo from GitHub Enterprise in Databricks notebook

We are trying to download our repository which is hosted on GitHub Enterprise to use its python libraries in our notebooks.Earlier we had issues with downloading our repository using the repos feature in Databricks platform since only notebooks can b...

Data Engineering
Reply
  • 2356 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
User16753724663
Valued Contributor
  • 1 kudos

To fix the issue, we need to pass the token in the header itself git clone https://<token>:x-oauth-basic@github.com/owner/repo.gitExample:%sh   git clone https://<token>@github.com/darshanbargal4747/databricks.git

  • 1 kudos
Anbazhagananbut
by • New Contributor II
  • 11688 Views
  • 1 replies
  • 1 kudos

How to handle Blank values in Array of struct elements in pyspark

Hello All, We have a data in a column in pyspark dataframe having array of struct typehaving multiple nested fields present.if the value is not blank it will savethe data in the same array of struct type in spark delta table.please advise on the bel...

Data Engineering
Reply
  • 11688 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
shyam_9
Databricks Employee
  • 1 kudos

Hi @Anbazhagan anbutech17,Can you please try as in below answers,https://stackoverflow.com/questions/56942683/how-to-add-null-columns-to-complex-array-struct-in-spark-with-a-udf

  • 1 kudos
Labels