Anonymous
Not applicable

@Kay Connolly​ :

It looks like you are trying to concatenate a string with a column object, which is causing the error. You need to convert the column object to a string first before concatenating it to the URL. Here's a modified code snippet that should work:

from pyspark.sql.functions import concat_ws
 
Idlist = spark.read.load("loadedfile.paquet")
locid = Idlist.select('IdColumn')
 
# Convert the IdColumn to string and concatenate with the URL
lookup_urls = locid.withColumn('url', concat_ws('', 'https://apilink/locations/', locid.IdColumn.cast('string')))
 
# Show the resulting URLs
lookup_urls.show()

This should create a new column called url that contains the complete API links for each location ID in your dataframe. You can then use this column to make the API calls in a loop.