Anonymous
Not applicable
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2023 10:33 PM
@Kay Connolly :
It looks like you are trying to concatenate a string with a column object, which is causing the error. You need to convert the column object to a string first before concatenating it to the URL. Here's a modified code snippet that should work:
from pyspark.sql.functions import concat_ws
Idlist = spark.read.load("loadedfile.paquet")
locid = Idlist.select('IdColumn')
# Convert the IdColumn to string and concatenate with the URL
lookup_urls = locid.withColumn('url', concat_ws('', 'https://apilink/locations/', locid.IdColumn.cast('string')))
# Show the resulting URLs
lookup_urls.show()This should create a new column called url that contains the complete API links for each location ID in your dataframe. You can then use this column to make the API calls in a loop.