Databricks

dvmentalmadess

I am trying to setup s3 as a structured streaming source. The bucket receives ~17K files/day and the original load to the bucket was ~54K files. The bucket was first loaded 3 months ago and we haven't started reading from it since. So let's say there...

dvmentalmadess · 08-01-2023

We use a private PyPI repo (AWS CodeArtifact) to publish custom python libraries. We make the private repo available to DBR 12.2 clusters using an init-script as prescribed here in the Databricks KB. When we tried to upgrade to 13.2 this stopped wor...

dvmentalmadess · 06-28-2023

We are migrating our data into Databricks and I was looking at the recommendations for partitioning here: https://docs.databricks.com/tables/partitions.html. This recommends not specifying partitioning and allowing "Ingestion Time Partitioning" (ITP)...

dvmentalmadess · 04-25-2023

We have setup SCIM with Okta at the account-level and setup Unity Catalog and are in the process of migrating groups from workspace-local to account-level. I have an instance profile that was assigned to a workspace-local group. using `databricks_gro...

dvmentalmadess · 04-20-2023

Based on the instructions for creating an external table (see: https://docs.databricks.com/data-governance/unity-catalog/create-tables.html#create-a-table) I had assumed that external tables were a way to add an existing object store to Unity Catalog...

dvmentalmadess

Thanks,We were able to make things work by increasing the driver instance size so it has more memory for the initial load. After initial load we scaled the instance down for subsequent runs. We're still testing, if we aren't able to make it work we'l...

dvmentalmadess · 08-07-2023

@lucasvieira thanks for tracking down the fix! It solves the problem as described so I'm marking your post as the solution.In case it helps someone else, my original posts left out a detail from our setup in order to make the problem easy to replica...

dvmentalmadess · 08-02-2023

@Kaniz,I've been running some additional experiments.Experiment 1Create DBR 12.2 clusterOnce running, check to see if /home/libraries/.cache/pip exists using ls -lh (it does not)Use cluster configuration "Libraries" tab to install "s3_ingest==0.0.2" ...

dvmentalmadess · 08-02-2023

@Kanizthank you for your reply. In response to your suggestions:Library available? I can run pip install from a notebook on interactive compute using both DBR 12.2 and 13.2Library version? Same results as 1.Different version? Same results as 1 and 2 ...

dvmentalmadess · 08-02-2023

@Kanizthank you for your response.This only happens with a job compute cluster and if I change spark_version from 12.2.x-scala2.12 to 13.2.x-scala2.12. The same job definition works fine when configured as DBR 12.2. Also, if I spin up an interactive ...

Databricks

User Stats

User Activity

Structured Streaming of S3 source

Private PyPI repos on DBR 13+

Ingestion Time Clustering on initial load

Terraform: Grant account-level group access to instance profile

Do external tables automatically receive external updates?

Re: Structured Streaming of S3 source

Re: Private PyPI repos on DBR 13+

Re: Private PyPI repos on DBR 13+

Re: Private PyPI repos on DBR 13+

Re: Private PyPI repos on DBR 13+