cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
hv
by • New Contributor
  • 4315 Views
  • 1 replies
  • 0 kudos

Error-"'Column' object is not callable".

I am trying to lowercase one of the columns(A_description) of a dataframe(df) and getting the error-"'Column' object is not callable".Code: def new_desc():  for line in df:    line = df['A_description'].map(str.lower)  return line new_desc()Have used...

Data Engineering
Reply
  • 4315 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Chaitanya_Raju
Honored Contributor
  • 0 kudos

Hi @Himadri Verma​ Hope this below suggestion will help you in pyspark.Please let me know if you are looking for something elseHappy Learning!!

  • 0 kudos
theSoyf
by • New Contributor II
  • 4276 Views
  • 2 replies
  • 1 kudos

How to write to Salesforce object using Spark Salesforce Library

Hi I'm facing an issue when writing to a salesforce object. I'm using the springml/spark-salesforce library. I have the above libraries installed as recommended based on my research.I try to write like this:(_sqldf .write .format("com.springml.spar...

Screen Shot 2022-12-14 at 8.18.07 AM
Data Engineering
Reply
  • 4276 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Gauthy
New Contributor II
  • 1 kudos

Im facing the same issue while trying to write to Salesforce, if you have found a resolution could you please share it ?

  • 1 kudos
1 More Replies
Rishabh-Pandey
by • Esteemed Contributor
  • 1134 Views
  • 1 replies
  • 5 kudos

PrivilegesSELECT: gives read access to an object.CREATE: gives ability to create an object (for example, a table in a schema).MODIFY: gives ability to...

PrivilegesSELECT: gives read access to an object.CREATE: gives ability to create an object (for example, a table in a schema).MODIFY: gives ability to add, delete, and modify data to or from an object.USAGE: does not give any abilities, but is an add...

Data Engineering
Reply
  • 1134 Views
  • 1 replies
  • 5 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 5 kudos

thanks sir

  • 5 kudos
gpzz
by • New Contributor II
  • 2495 Views
  • 1 replies
  • 3 kudos

pyspark code error

rdd4 = rdd3.reducByKey(lambda x,y: x+y)AttributeError: 'PipelinedRDD' object has no attribute 'reducByKey'Pls help me out with this

Data Engineering
Reply
  • 2495 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 3 kudos

Is it a typo or are you really using reducByKey instead of reduceByKey ?

  • 3 kudos
Slalom_Tobias
by • New Contributor III
  • 10226 Views
  • 1 replies
  • 1 kudos

AttributeError: 'SparkSession' object has no attribute '_wrapped' when attempting CoNLL.readDataset()

I'm getting the error...AttributeError: 'SparkSession' object has no attribute '_wrapped'---------------------------------------------------------------------------AttributeError Traceback (most recent call last)<command-2311820097584616> in <cell li...

Data Engineering
Reply
  • 10226 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 1 kudos

this can happen in 10X version try to use 7.3 LTS and share your observationand if it not working there try to create init script and load it to your databricks cluster so whenever your machine go up you can get advantage of that library because some...

  • 1 kudos
Ancil
by • Contributor II
  • 7001 Views
  • 1 replies
  • 4 kudos

Resolved! TypeError: 'JavaPackage' object is not callable - DocumentAssembler() - Spark NLP

Hi,I am doing a grammar check using spark NLP using azure databricks. But am getting TypeError: 'JavaPackage' object is not callable - in DocumentAssembler() intialization line.document_assembler = DocumentAssembler()\ .setInputCol("text")\ .setOutpu...

image.png
Data Engineering
Reply
  • 7001 Views
  • 1 replies
  • 4 kudos
View Replies
Latest Reply
Ancil
Contributor II
  • 4 kudos

Hi its working after adding below configs in cluster. Please check URL for more info :- https://nlp.johnsnowlabs.com/docs/en/install#databricks-support

  • 4 kudos
Raymond_Garcia
by • Contributor II
  • 5283 Views
  • 1 replies
  • 1 kudos

Resolved! query array in SQL

Hello I have a databricks question I was not able to answer myselfI have this queryselect count(*) from tablewhere object[0].value is not null and object[0].value.value1 = "s"and created_year = 2022 and created_month = 7 and created_day = 4you can se...

Data Engineering
Reply
  • 5283 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Raymond_Garcia
Contributor II
  • 1 kudos

SELECT count(*)FROM ( SELECT explode(mmycolumn) FROM table WHERE created_year = 2022 and created_month = 7 and created_day = 5)WHERE col.field is not null and col.field.field! = "signal"

  • 1 kudos
chandan_a_v
by • Valued Contributor
  • 21527 Views
  • 7 replies
  • 3 kudos

Resolved! TypeError: 'JavaPackage' object is not callable

image
Data Engineering
Reply
  • 21527 Views
  • 7 replies
  • 3 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 3 kudos

By any chance, was the cluster restarted after installing the libraries or was it detached and reattached from/to the notebook? Notebook-scoped libraries do not persist across sessions. You must reinstall notebook-scoped libraries at the beginning of...

  • 3 kudos
6 More Replies
Jeff1
by • Contributor II
  • 2037 Views
  • 3 replies
  • 1 kudos

Resolved! Strange object returned using sparklyr

CommunityI'm running a sparklyr "group_by" function and the function returns the following info:# group by event_typeacled_grp_tbl <- acled_tbl %>% group_by("event_type") %>% summary(count = n())                   Length Cl...

Data Engineering
Reply
  • 2037 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Jeff1
Contributor II
  • 1 kudos

I should have deleted the post. While your are correct "event_type" should be without quotes the problem was the Summary function. I was using the wrong function it should have been "summarize."

  • 1 kudos
2 More Replies
marsjuli
by • New Contributor II
  • 18674 Views
  • 1 replies
  • 1 kudos

How to handle <IPython.core.display.HTML object>

Some libraries have intermediate IPython HTML-objects returned to the notebook cell output.Since this happens during training a machine learning model the statements are typically buried within in the library so I cannot easily interfere. (e.g. in or...

grafik.png
Data Engineering
Reply
  • 18674 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
marsjuli
New Contributor II
  • 1 kudos

Hi @Kaniz Fatma​ ,thanks for showing me the link. This helps if you are in control of the generated html-object. If the html-content comes from a library, that is where the problems start, because I cannot wrap displayHTML().(I can of course look for...

  • 1 kudos
amitdatabricksc
by • New Contributor II
  • 7286 Views
  • 2 replies
  • 0 kudos

AttributeError: 'NoneType' object has no attribute 'repartition'

I am using a framework and i have a query where i am doing,df = seg_df.select(*).write.option("compression", "gzip') and i am getting below error,When i don't do the write.option i am not getting below error. Why is it giving me repartition error. Wh...

Data Engineering
Reply
  • 7286 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

hi @AMIT GADHAVI​ ,could you provide more details? for example, what is your data source? how do you repartition? etc

  • 0 kudos
1 More Replies
Labels