Databricks Community

Mathias_Peters

We have implemented an asset bundle (DAB) that creates a wheel. During DAB deployment, the wheel is built and stored in the folder of the service principal running the deployment via GH workflow. The full path is/Workspace/Users/SERVICE-PRINCIPAL-ID/...

Mathias_Peters · 05-26-2024

Hi, I implemented a job that should incrementally read all the available data from a Kinesis Data Stream and terminate afterwards. I schedule the job daily. The data retention period of the data stream is 7 days, i.e., there should be enough time to ...

Mathias_Peters · 05-15-2024

Hi, I have a job consisting of three tasks: tasks: - task_key: Kinesis_to_S3_new spark_python_task: python_file: ../src/kinesis.py parameters: ["${var.stream_region}", "${var.s3_base_path}"] j...

Mathias_Peters · 05-10-2024

Hi, I am a bit stumped atm bc I cannot figure out how to get a DLT table definition picked up in a Python notebook. 1. I created a new notebook in python2. added the following code: %python import dlt from pyspark.sql.functions import * @dlt.table(...

Mathias_Peters · 04-26-2024

Hi, I am loading data from a kinesis data stream using DLT. CREATE STREAMING TABLE Consumers_kinesis_2 ( ..., unbase64(data) String, ... ) AS SELECT * FROM STREAM read_kinesis (...) Is it possible to directly cast, unbase64, and/or transform the resu...

Mathias_Peters

Update: When moving the whl to the shared workspace folder, the installation on the cluster works.

Mathias_Peters · 05-16-2024

Hi @Ajay-Pandey ,thank you for the hints. I will try to recreate the job via UI. I ran the tasks in a Github workflow. The file locations are mixed: the first two tasks (python and dlt) are located in the databricks/src folder. The dbt files come fro...

Mathias_Peters · 05-10-2024

Ok, it seems that the default language of the notebook and the language of a particular cell can clash. If the default is set to Python, switching a cell to SQL won't work in DLT and vice versa. This is super unintuitive tbh.

Mathias_Peters · 05-10-2024

Update: this seems to be an issue with the notebook type and the magic commands at the beginnning. I have created a new DLT pipeline without attaching a notebook to it. That triggered the creation of an empty Python notebook. Switching this to SQL ad...

Mathias_Peters · 05-08-2024

We have used the roleArn and role session name like this: CREATE STREAMING TABLE table_name as SELECT * FROM STREAM read_kinesis ( streamName => 'stream', initialPosition => 'earliest', roleArn => 'arn:aws:iam::ACCT_ID:role/R...

Databricks Community

User Stats

User Activity

Service principal seemingly cannot access its own workspace folder

How to properly implement incremental batching from Kinesis Data Streams

Asset Bundles: Adding project_directory in DBT task breaks previous python task

DLT table not picked in python notebook

On the fly transformations on DLT tables

Re: Service principal seemingly cannot access its own workspace folder

Re: Asset Bundles: Adding project_directory in DBT task breaks previous python task

Re: DLT table not picked in python notebook

Re: DLT table not picked in python notebook

Re: DLT can't authenticate with kinesis using instance profile