cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
AJ270990
by • Contributor II
  • 3326 Views
  • 1 replies
  • 0 kudos

API for Databricks code functionality

I have a Databricks notebook for which I want to create an API. From that API I will have to call the notebook and perform certain operations. Result will be sent back to API. I dont want to do via Postman, as someone has to install Postman at their ...

Community Platform Discussions
Reply
  • 3326 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
I have a Databricks notebook for which I want to create an API. From that API I will have to call the notebook and perform certain operations. Result will be sent back to API. I dont want to do via Postman, as someone has to install Postman at their ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
Anonym
by • New Contributor II
  • 1088 Views
  • 1 replies
  • 0 kudos

Error ingesting files with databricks jobs

The source path that i want to ingest files with is:"gs://bucket-name/folder1/folder2/*/*.json"I have a file in this path that ends with ".json.gz" and the databricks job ingests this file even though it doesn't suppose to.How can i fix it?Thanks.

Community Platform Discussions
Reply
  • 1088 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonym
New Contributor II
  • 0 kudos

Thanks Kaniz

  • 0 kudos
Madhawa
by • New Contributor II
  • 1633 Views
  • 0 replies
  • 0 kudos

org.apache.spark.SparkException - FileReadException

Sometimes getting this kind of error "org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 12224.0 failed 4 times, most recent failure: Lost task 1.5 in stage 12224.0 (TID           ) (12.xxx.x.xxx executor 1): com.datab...

Community Platform Discussions
Reply
  • 1633 Views
  • 0 replies
  • 0 kudos
adrianhernandez
by • New Contributor III
  • 2419 Views
  • 3 replies
  • 1 kudos

Add Oracle Jar to Databricks cluster policy

I created a policy for users to use when they create their own Job clusters. When I'm editing the policy, I don't have the UI options for adding library (I can only see Definitions and Permissions tabs). I need to add via JSON the option to allows th...

Community Platform Discussions
Reply
  • 2419 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@adrianhernandez are you admin to workspace, if not you might be missing permissions, if you have policies enabled, admin can allow you.https://docs.databricks.com/en/administration-guide/clusters/policies.html#librariesif your workspace is Unity cat...

  • 1 kudos
2 More Replies
thuovi
by • New Contributor II
  • 1765 Views
  • 0 replies
  • 2 kudos

dbutils.fs.ls MAX_LIST_SIZE_EXCEEDED

Hi!I'm experiencing different behaviours between two DBX Workspaces when trying to list file contents from an abfss: location.In workspace A running len(dbutils.fs.ls('abfss://~~@~~~~.dfs.core.windows.net/~~/')) results in "Out[1]: 1551", while runni...

Community Platform Discussions
Reply
  • 1765 Views
  • 0 replies
  • 2 kudos
llvu
by • New Contributor III
  • 2878 Views
  • 3 replies
  • 1 kudos

getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4 LTS

Hello,I am trying to use the getArgument() function in a spark.sql query. It works fine if I run the notebook via an interactive cluster, but gives an error when executed via a job run in an instance Pool.query:OPTIMIZE <table>where date = replace(re...

Community Platform Discussions
Reply
  • 2878 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
llvu
New Contributor III
  • 1 kudos

Hi @Retired_mod,Would you be able to respond to my last comment? I couldn't manage to get it working yet.Thank you in advance.

  • 1 kudos
2 More Replies
AH
by • New Contributor III
  • 2110 Views
  • 1 replies
  • 0 kudos

AWS Databricks VS AWS EMR

HiWhich services should I use for data lake implementation?any cost comparison between Databricks and aws emr.which one is best to choose 

Community Platform Discussions
Reply
  • 2110 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@AH that depends on use case, if your implementation involves Data Lake, ML, Data engineering tasks better to go with databricks as it has got good UI and there good governance using unity catalog for your data lake and you have good consumer tool su...

  • 0 kudos
elgeo
by • Valued Contributor II
  • 2039 Views
  • 1 replies
  • 1 kudos

Resolved! System billing usage table - Usage column

Hello experts,Could someone please explain what is exactly contained into the column usage in the system.billing.usage table?We ran specific queries in a cluster trying to calculate the cost and we observe that the DBUs shown in the system table are ...

Community Platform Discussions
Reply
  • 2039 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@elgeo both should be same, untill if somehow we miss to pick proper plan DBU price, usage column will have complete information related to sku name and DBU units etc... if you use azure databricks calculator and compare we should see similar result 

  • 1 kudos
RyanHager
by • Contributor
  • 2970 Views
  • 4 replies
  • 2 kudos

Roadmap on export menu option for SQL Query and Dashboard Types in Workspace

Are there plans for an export option for SQL Query and SQL Dashboard in the Workspace explorer screen similar to notebooks?Background:  Need a way to export and backup any queries and dashboards to save design work and move from staging environments ...

Community Platform Discussions
Reply
  • 2970 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Th best option would be to have them just under git Repo (especially dashboards).

  • 2 kudos
3 More Replies
surya_1527
by • New Contributor
  • 1716 Views
  • 0 replies
  • 0 kudos

DataBricks Certification Exam Got Suspended. Require support for the same.

Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam, saying I have exceeded ...

Community Platform Discussions
Reply
  • 1716 Views
  • 0 replies
  • 0 kudos
kll
by • New Contributor III
  • 1153 Views
  • 1 replies
  • 0 kudos

how to save variables in one notebook to be imported into another?

Say, I have a list of values, dictionaries, variable names in `notebook1.ipynb` that I'd like to re-use / import in another `notebook2.ipynb`. For example, in `notebook1.ipynb`, I have the following:   var1 = "dallas" var_lst = [100, 200, 300, 400, ...

Community Platform Discussions
Reply
  • 1153 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Krishnamatta
New Contributor III
  • 0 kudos

You can use %run ./notebook2 after defining variables in notebook1So notebook2 will use the variables defined in notebook1

  • 0 kudos
elgeo
by • Valued Contributor II
  • 1567 Views
  • 1 replies
  • 0 kudos

Retrieve DBU per query executed

Hello experts,Do you know how we can retrieve the DBUs consumed for a specific query?Thank you

Community Platform Discussions
Reply
  • 1567 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
elgeo
Valued Contributor II
  • 0 kudos

I couldn't find a metadata table. However the workaround is to multiply the DBU of the current cluster (retrieve it either online or to be more accurate from the compute page at the right) and multiply it with the time in minutes that the query took ...

  • 0 kudos
NielsMH
by • New Contributor III
  • 1915 Views
  • 2 replies
  • 1 kudos

running notebook job from remote github repository fails, but do not fail on python script type

Hi allI am trying to run a notebook from a remote repository, but the job fails. I setup the job as follows:my project structure is as such:but the output i get is like such: The thing is if i set the job type to "Python Script" i dont encounter this...

job-setup.png folder_structure.png job_output.png
Community Platform Discussions
Reply
  • 1915 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@NielsMH if you want to run your jobs based o job name, please use new preview service that databricks released which are DAB format. there you can run your job based on your job name.remote repo in the sense, are you using github actions or api, loo...

  • 1 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors
View all