Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-10-2022 01:54 AM
I'm facing an error in Delta Live Tables when I want to pivot a table. The error is the following:
And the code to replicate the error is the following:
import pandas as pd
import pyspark.sql.functions as F
pdf = pd.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo",
"bar", "bar", "bar", "bar"],
"B": ["one", "one", "one", "two", "two",
"one", "one", "two", "two"],
"C": ["small", "large", "large", "small",
"small", "large", "small", "small",
"large"],
"D": [1, 2, 2, 3, 3, 4, 5, 6, 7],
"E": [2, 4, 5, 5, 6, 6, 8, 9, 9]})
df = spark.createDataFrame(pdf)
df.write.mode('overwrite').saveAsTable('test_table')
import dlt
@dlt.view
def test_table():
return spark.read.table('test_table')
@dlt.table
def test_table_pivoted():
return (
spark.table('LIVE.test_table')
.groupBy('A', 'B')
.pivot('C')
.agg(F.first('D'))
)Does anybody know why I can not pivot a table in Delta Live Tables Pipelines?
Labels:
- Labels:
-
Delta Live Tables