cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

tototox
by New Contributor III
  • 1801 Views
  • 3 replies
  • 0 kudos

Using dbutils.fs.ls gives overlap error.

I created a schema with that route as a managed location.(abfss://~~@~~.dfs.core.windows.net/dejeong)And an external table named 'first_table' was created in the corresponding path.(abfss://~~@~~.dfs.core.windows.net/dejeong/first_table)​The results ...

  • 1801 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @jin park​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we c...

  • 0 kudos
2 More Replies
iwan_aucamp
by New Contributor III
  • 743 Views
  • 2 replies
  • 1 kudos

Account SCIM API OpenAPI specification issues

I'm trying to get a list of all users, groups and service principals on Azure from a python script. As I understand things I should be using the Account SCIM API for this. According to the azure documentation [ref], the OpenAPI specification for this...

  • 743 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Iwan Aucamp​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 1 kudos
1 More Replies
Deiry
by New Contributor III
  • 614 Views
  • 0 replies
  • 0 kudos

Why is the whole list not displayed in dbutil.widgets.multiselect?

I have been studying the Apache Spark in Databricks Academy and I don't understand why the whole list is nos displayed? Creation of widgets:dbutils.widgets.text("name", "Brickster", "Name") dbutils.widgets.multiselect("colors","orange", ["orange", "r...

image
  • 614 Views
  • 0 replies
  • 0 kudos
db-avengers2rul
by Contributor II
  • 5279 Views
  • 2 replies
  • 6 kudos

Resolved! AttributeError: 'list' object has no attribute 'columns' - PySpark

Hi All,i am getting the below error when i am ingesting the data from source file , source file is also attached , i have tried in both Community edition and Azure databricks as well getting the same error , can any one suggest me the solution ? # ...

  • 5279 Views
  • 2 replies
  • 6 kudos
Latest Reply
Kaniz
Community Manager
  • 6 kudos

Hi @Rakesh Reddy Gopidi​ ​, We haven’t heard from you since the last response from me, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise...

  • 6 kudos
1 More Replies
vizoso
by New Contributor III
  • 789 Views
  • 2 replies
  • 3 kudos

Cluster list in Microsoft.Azure.Databricks.Client fails because ClusterSource enum does not include MODELS. When you have a model serving cluster, Clu...

Cluster list in Microsoft.Azure.Databricks.Client fails because ClusterSource enum does not include MODELS.When you have a model serving cluster, ClustersApiClient.List method fails to deserialize the API response because that cluster has MODELS as C...

  • 789 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @José Fernández Vizoso​, May I know are you facing any issue here or do you want to share some sort of information through this post?

  • 3 kudos
1 More Replies
Barb
by New Contributor III
  • 2487 Views
  • 2 replies
  • 0 kudos

How do I get a list of the tables that I personally created?

I know that I can get a list of all of the table names in a given 'database' by using (if the 'database' was named "scratch"): show tables from scratchHow do I get a list just like that, but that only lists the tables that I created?

  • 2487 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Barb Krienke​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
1 More Replies
KNP
by New Contributor
  • 1936 Views
  • 2 replies
  • 0 kudos

passing array as a parameter to PandasUDF

Hi Team,My python dataframe is as below.The raw data is quite a long series of approx 5000 numbers. My requirement is to go through each row in RawData column and calculate 2 metrics. I have created a function in Python and it works absolutely fine. ...

image
  • 1936 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hello @Kausthub NP​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
1 More Replies
Bency
by New Contributor III
  • 950 Views
  • 2 replies
  • 1 kudos

How to get the list of parameters passed from widget

Hi ,Could someone help me understand how I would be able to get all the parameters in the task (from the widget). ie I want to get input as parameter 'Start_Date' , but the case is that this will not always be passed . It could be 'Run_Date' as well ...

  • 950 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Bency Mathew​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 1 kudos
1 More Replies
celerity12
by New Contributor II
  • 3630 Views
  • 7 replies
  • 4 kudos

Pulling list of running jobs using JOBS API 2.1

I need to find out all jobs which are currently running and not get other jobsThe below command fetches all the jobscurl --location --request GET 'https://xxxxxx.gcp.databricks.com/api/2.1/jobs/list?active_only=true&expand_tasks=true&run_type=JOB_RUN...

  • 3630 Views
  • 7 replies
  • 4 kudos
Latest Reply
User16764241763
Honored Contributor
  • 4 kudos

Hi @Sumit Rohatgi​ It seems like active_only=true only applies to jobs/runs/list API and not to jobs/list.Can you please try the jobs/runs/list API?

  • 4 kudos
6 More Replies
Jack
by New Contributor II
  • 3062 Views
  • 2 replies
  • 1 kudos

Append an empty dataframe to a list of dataframes using for loop in python

I have the following 3 dataframes:I want to append df_forecast to each of df2_CA and df2_USA using a for-loop. However when I run my code, df_forecast is not appending: df2_CA and df2_USA appear exactly as shown above.Here’s the code:df_list=[df2_CA,...

image image
  • 3062 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Jack Homareau​ , We haven’t heard from you on the last response from @Arvind Ravish​ , and I was checking back to see if you have a resolution yet. If you have any solution, please do share that with the community as it can be helpful to others. ...

  • 1 kudos
1 More Replies
SailajaB
by Valued Contributor III
  • 6580 Views
  • 5 replies
  • 12 kudos

Resolved! how to convert each row of df to array of rows(list of rows)

Hi,How to convert each row of dataframe to array of rows?Here is our scenario , we need to pass each row of dataframe to one function as dict to apply the key level transformations. But as our data is very huge we can't use collect df.toJson().colle...

  • 6580 Views
  • 5 replies
  • 12 kudos
Latest Reply
SailajaB
Valued Contributor III
  • 12 kudos

@Hubert Dudek​ , Thank you for the reply. We are new to ADB. And using the below code, looking for an optimized way to do itdfJSONString = df.toJSON().collect()stringList = []  for row in dfJSONString:    # ==== Unflatten the JSON string ==== #    js...

  • 12 kudos
4 More Replies
Sandesh87
by New Contributor III
  • 1731 Views
  • 2 replies
  • 2 kudos

Resolved! create a dataframe with all the responses from the api requests within foreachPartition

I am trying to execute an api call to get an object(json) from amazon s3 and I am using foreachPartition to execute multiple calls in paralleldf.rdd.foreachPartition(partition => { //Initialize list buffer var buffer_accounts1 = new ListBuffer[St...

  • 1731 Views
  • 2 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Hi @Sandesh Puligundla​ ,Thank you for sharing the solution. We will mark it as "best" response so, in the future is another user has the same question, they will be able to find the solution right away.

  • 2 kudos
1 More Replies
SailajaB
by Valued Contributor III
  • 10538 Views
  • 9 replies
  • 6 kudos

How to send a list as parameter in databricks notebook task

Hi,How we can pass a list as parameter in data bricks notebook to run the notebook parallelly for list of values.Thank you

  • 10538 Views
  • 9 replies
  • 6 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

another another way (in databricks you can achieve everything many ways) is to encode list using json library:import json print type(json.dumps([1, 2, 3])) #>> <type 'str'>

  • 6 kudos
8 More Replies
User16869510359
by Esteemed Contributor
  • 1546 Views
  • 1 replies
  • 0 kudos
  • 1546 Views
  • 1 replies
  • 0 kudos
Latest Reply
amr
New Contributor III
  • 0 kudos

You can view them in the workspace Home Page --> Top Right Click on User Settings --> Access Tokens.Or you can use the List API to get them.

  • 0 kudos
Labels