Databricks

YSF · 04-30-2023

Does anyone have a workflow or pattern that works for developing with autoloader/DLT? I'm still new to but the fact that while testing it's creating checkpoints using schema locations makes it really tricky to develop with and hammer out a working ve...

YSF · 04-30-2023

I'm trying to setup autoloader to read some csv files. I tried with both autoloader with the DLT decorator as well as just autoloader by itself. The first column of the data is called "run_id", when I do a spark.read.csv() directly on the file it com...

YSF · 02-28-2023

I have a repo that have python files that use the built in logging module. Additionally in some of the notebooks of the repo I want to use logging.debug()/logging.info() instead of print statements everywhere. However when I use the root logger or cr...

YSF · 12-16-2022

I have images, zipped files, and arbitrary binary files in ADLS gen 2. In Unity Catalog I created a storage credential and an external location to the container that contains these files in directories.Is there any way I can view them within Unity Ca...

YSF · 12-25-2021

Has anyone successfully used Petastorm + Databricks-Connect + Delta Lake?The use case is being able to use DeltaLake as a data store regardless of whether I want to use the databricks workspace or not for my training tasks.I'm using a cloud-hosted ju...

YSF · 04-19-2023

I'm also getting this error for catalogs/schemas/tables I create. I get this when I click sample data. But I am able to otherwise query it from a notebook using SQL

YSF · 12-19-2022

In the Data Explorer, in my external location, the URL is:abfss://<container>@<storage_account>.dfs.core.windows.netWhen I click "Test Connection" it says all permissions confirmed and has full privileges. In my ADLS I have a text file that is inside...

YSF · 12-29-2021

because its janky or why? I don't need it for customer facing production. More so for if I'm using my own HPC or local workstation, but I want to access data from delta lake. Figured it was easier/preferable to setting up my own spark environment loc...

Databricks

User Stats

User Activity

Any elegant pattern for Autoloader/DLT development?

Delta Live Table & Autoloader adding a non-existent column

How do I use the Python Logging Module in a Repo?

Unity Catalog with unstructured "lake" data? (.png, .ggzip, .zarr)

Issues with using Databricks-Connect and Petastorm

Re: Unity Catalog - Error getting sample data in data explorer

Re: Unity Catalog with unstructured "lake" data? (.png, .ggzip, .zarr)

Re: Issues with using Databricks-Connect and Petastorm