cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Karin
by • New Contributor II
  • 3819 Views
  • 1 replies
  • 2 kudos

Resolved! Liquid clustering with boolean columns

Hi community Is it possible to use boolean columns as cluster keys for liquid clustering on Delta Tables? I've been trying to set a boolean column as cluster key since it's one of my most common queries when reading from the table. I'm getting the er...

Data Engineering
Liquid clustering
Reply
  • 3819 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
jeroenvs
New Contributor III
  • 2 kudos

Can confirm that boolean columns are note allowed for liquid clustering. This seems to be undocumented and the error message is not helpful: "couldn't find clustering column in stats schema"

  • 2 kudos
RobinK
by • Contributor
  • 13635 Views
  • 5 replies
  • 6 kudos

Resolved! How to set Python rootpath when deploying with DABs

We have structured our code according to the documentation (notebooks-best-practices). We use Jupyter notebooks and have outsourced logic to Python modules. Unfortunately, the example described in the documentation only works if you have checked out ...

Data Engineering
Reply
  • 13635 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Corbin
Databricks Employee
  • 6 kudos

Hello Robin, You’ll have to either use wheel files to package your libs and use those (see docs here), to make imports work out of the box. Otherwise, your entry point file needs to add the bundle root directory (or whatever the lib directory is) to ...

  • 6 kudos
4 More Replies
Kumarashokjmu
by • New Contributor II
  • 4074 Views
  • 4 replies
  • 0 kudos

need to ingest millions of csv files from aws s3

I have a need to ingest millions of csv files from aws s3 bucket. I am facing issue with aws s3 throttling issue and besides notebook process is running for 8 hours plus and sometimes failing. When looking at cluster performance, it is utilized 60%.I...

Data Engineering
Reply
  • 4074 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
kulkpd
Contributor
  • 0 kudos

If you want to load all the data at once use autoloader or DLT pipeline with directory listing if files are lexically ordered. ORIf you want to perform incremental load, divide the load into two job like historic data load vs live data load:Live data...

  • 0 kudos
3 More Replies
leelee3000
by • Databricks Employee
  • 932 Views
  • 0 replies
  • 0 kudos

Dynamic Filtering Criteria for Data Streaming

One of the potential uses for DLT is a scenario where I have a large input stream of data and need to create multiple smaller streams based on dynamic and adjustable filtering criteria. The challenge is to allow non-engineering individuals to adjust ...

Data Engineering
Reply
  • 932 Views
  • 0 replies
  • 0 kudos
leelee3000
by • Databricks Employee
  • 1538 Views
  • 0 replies
  • 0 kudos

Parameterizing DLT Jobs

I have observed the use of advanced configuration and creating a map as a way to parameterize notebooks, but these appear to be cluster-wide settings. Is there a recommended best practice for directly passing parameters to notebooks running on a DLT ...

Data Engineering
Reply
  • 1538 Views
  • 0 replies
  • 0 kudos
Geoff
by • New Contributor II
  • 1510 Views
  • 0 replies
  • 1 kudos

Bizarre Delta Tables pipeline error: ModuleNotFound

I received the following error when trying to import a function defined in a .py file into a .ipynb file. I would add code blocks, but the message keeps getting rejected for invalid HTML.# test_lib.py (same directory, in a subfolder)def square(x):ret...

Data Engineering
Reply
  • 1510 Views
  • 0 replies
  • 1 kudos
pankz-104
by • New Contributor
  • 1567 Views
  • 1 replies
  • 0 kudos

how to read deleted files in adls

We have soft delete enabled in adls for 3 days, And we have manually deleted some checkpoint files size 3 tb approx. Each file is just couple of bytes like 30 b, 40 b. The deleted file size is increasing day by day even after couple of days. Suppose ...

Data Engineering
Reply
  • 1567 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @pankz-104 , Just a friendly follow-up. Did you have time to test Kaniz's recommendations? do you still have issues? please let us know

  • 0 kudos
Chris_sh
by • New Contributor II
  • 876 Views
  • 1 replies
  • 0 kudos

DLT Missing Select tables button or Enhancement Request?

Currently when a Delta Live Table fails due to an error the option to select specific tables to run a full refresh is removed. This seems like an error. A full refresh can fix an error that might be caused and you should always be able to select to d...

Chris_sh_0-1700066428625.png Chris_sh_2-1700066503115.png
Data Engineering
Reply
  • 876 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Chris_sh, which DLT channel are you using? 

  • 0 kudos
Rajaniesh
by • New Contributor III
  • 2803 Views
  • 2 replies
  • 1 kudos

URGENT HELP NEEDED: Python functions deployed in the cluster throwing the error

Hi,I have created a python wheel with the following code. And the package name is rule_engine"""The entry point of the Python Wheel"""import sysfrom pyspark.sql.functions import expr, coldef get_rules(tag): """  loads data quality rules from a table ...

Data Engineering
Reply
  • 2803 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

You can find more details and examples here https://docs.databricks.com/en/workflows/jobs/how-to/use-python-wheels-in-workflows.html#use-a-python-wheel-in-a-databricks-job

  • 1 kudos
1 More Replies
dcc
by • New Contributor
  • 7785 Views
  • 1 replies
  • 0 kudos

DBT Jobs || API call returns "Internal Error"

Hey there,I am currently using the Databricks API to trigger a specific DBT job. For this, I am calling the API in a Web Activity on Azure datafactory and sending as headers the token and for the body I am sending the Job ID and the necessary vars I ...

Data Engineering
API
jobs
Reply
  • 7785 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Could you please share the driver logs? it will help us to narrow down the issue

  • 0 kudos
chari
by • Contributor
  • 13401 Views
  • 4 replies
  • 2 kudos

Resolved! Connect to data in one drive to Azure Databricks

Hello,A colleague of mine previously built a data pipeline for connecting data available on share point (one drive), coded in python in jupyter notebook. Now, its my job to transfer the code to Azure databricks and I am unable to connect/download thi...

Data Engineering
Workflows
Reply
  • 13401 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
gabsylvain
Databricks Employee
  • 2 kudos

@chari Also you ingest both Sharepoint and OneDrive data directly into Databricks using Partner Connect. You can refer to the documentation bellow for more information: Connect to Fivetran using Partner Connect Fivetran Sharepoint Connector Documenta...

  • 2 kudos
3 More Replies
rvo1994
by • New Contributor
  • 7424 Views
  • 0 replies
  • 0 kudos

Performance issue with spatial reference system conversions

Hi,I am facing a performance issue with spatial reference system conversions. My delta table has approximately 10 GB/46 files/160M records and gets +/- 5M records every week. After ingestion, I need to convert points (columns GE_XY_XCOR and GE_XY_YCO...

Data Engineering
Reply
  • 7424 Views
  • 0 replies
  • 0 kudos
BriGuy
by • New Contributor II
  • 1527 Views
  • 0 replies
  • 0 kudos

How can I efficiently write to easily queryable logs?

I've got a parallel running process loading multiple tables into the datalake. I'm writing my logs to a delta table using dataframewriter in append mode. The problem is that every save is taking a bit of time with what appears to be the calculation o...

Data Engineering
logging
Reply
  • 1527 Views
  • 0 replies
  • 0 kudos
MrDataMan
by • New Contributor II
  • 2087 Views
  • 2 replies
  • 0 kudos

Expand and read Zip compressed files not working

I am trying to unzip compressed files following this doc (https://docs.databricks.com/en/files/unzip-files.html) but I am getting the error.When I run:dbutils.fs.mv("file:/LoanStats3a.csv", "dbfs:/tmp/LoanStats3a.csv") I get the following error: java...

Data Engineering
Reply
  • 2087 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
gabsylvain
Databricks Employee
  • 0 kudos

Hey @MrDataMan, I wasn't able to reproduce the exact same error you did get, but I still got a similar error while trying to run the example. To solve it, I tweaked the code a little bit:   %sh curl https://resources.lendingclub.com/LoanStats3a.csv.z...

  • 0 kudos
1 More Replies
BriGuy
by • New Contributor II
  • 1492 Views
  • 1 replies
  • 0 kudos

process logging optimisation

I have created a process that runs a notebook multiple times in parallel with different parameters.  This was working quite quickly.  However I've added several logging steps that are appending log details to a dataframe then using dataframewriter to...

Data Engineering
logging
Reply
  • 1492 Views
  • 1 replies
  • 0 kudos
View Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels