cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Saurabh707344
by New Contributor III
  • 15731 Views
  • 2 replies
  • 1 kudos

Platform and Approach Comparison

Do anyone have structure and crisp comparison between benefits of performing MLOps using below ways and what are the strong areas of each platform:a) Standalone Databricks where all pipelines and orchestration done on Databricks and external third pa...

  • 15731 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Saurabh Singh​, Here is a structured and crisp comparison of the benefits and strong areas of each platform for performing MLOps:a) Standalone Databricks:Benefits: Unified platform: Databricks provides a unified environment for data engineering, ...

  • 1 kudos
1 More Replies
f2008700
by New Contributor III
  • 12450 Views
  • 7 replies
  • 7 kudos

Configuring average parquet file size

I have S3 as a data source containing sample TPC dataset (10G, 100G).I want to convert that into parquet files with an average size of about ~256MiB. What configuration parameter can I use to set that?I also need the data to be partitioned. And withi...

  • 12450 Views
  • 7 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @Vikas Goel​ We haven't heard from you since the last response from @Werner Stinckens​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 7 kudos
6 More Replies
mbejarano89
by New Contributor III
  • 1607 Views
  • 2 replies
  • 0 kudos

Running a K-means (.fit) gives error:Params must be either a param map or a list/tuple of param maps but got %s." % type(params)

 am running a k-means algorithm. My feature are DoubleType and have no nulls, but I get : raise TypeError("Params must be either a param map or a list/tuple of param maps but got %s." % type(params). Anyone have any idea how to solve this?File /datab...

  • 1607 Views
  • 2 replies
  • 0 kudos
Latest Reply
mbejarano89
New Contributor III
  • 0 kudos

I found the answer just by trying several things, although I do not understand exactly what the problem was. All I had to do was to cache the input data before fitting the model:assemble=VectorAssembler(inputCols=columns_input, outputCol='features')...

  • 0 kudos
1 More Replies
jch
by New Contributor III
  • 1399 Views
  • 2 replies
  • 0 kudos

Can Kaggle Kernels be used in Databricks notebooks?

suggestions).This one looks perfect kaggle kernels output rsrishav/starter-youtube-trending-video-dataset -p /path/to/dest but I'm not using CLI, I'm using a databricks notebook.I tried using this code but it doesn't work. data_path = 'rsrishav/youtu...

  • 1399 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @jch​, The error message you received, "Permission 'kernels.get' was denied," suggests that you don't have the necessary permissions to access or view the Kaggle kernel. This error typically occurs when you try to access a kernel that you don't ha...

  • 0 kudos
1 More Replies
Anonymous
by Not applicable
  • 1859 Views
  • 6 replies
  • 2 kudos

  Hello Everyone, I am thrilled to announce that we have our 6th winner for the raffle contest - @Yogita Chavan​ . Please join me in congratulating he...

 Hello Everyone,I am thrilled to announce that we have our 6th winner for the raffle contest - @Yogita Chavan​ . Please join me in congratulating her on this remarkable achievement!Your dedication and hard work have paid off, and we are delighted to ...

6
  • 1859 Views
  • 6 replies
  • 2 kudos
Latest Reply
yogu
Honored Contributor III
  • 2 kudos

@Suteja Kanuri​ i haven't received swag until now as per our last mail i provided size after that I didn't receive any mail for tracking details

  • 2 kudos
5 More Replies
bshirdi
by New Contributor II
  • 7474 Views
  • 1 replies
  • 2 kudos

Getting HTTP 502 bad gateway error!

Hello all,I am suddenly getting an HTTP 502 and DRIVER_LIBRARY_INSTALLATION_FAILURE error during the Python library installation when the cluster gets initialized. I have around 10 Python packages out of which 2-3, packages always failed to install a...

image.png
  • 7474 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Bhargav Shir​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
Priyag1
by Honored Contributor II
  • 748 Views
  • 2 replies
  • 10 kudos

How to improve visualization and dasboard concepts?

How to improve visualization and dasboard concepts?

  • 748 Views
  • 2 replies
  • 10 kudos
Latest Reply
Anonymous
Not applicable
  • 10 kudos

Hi @Priyadarshini G​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 10 kudos
1 More Replies
carlosjrestr
by New Contributor III
  • 2358 Views
  • 2 replies
  • 1 kudos

Does Unity Catalog on Azure require premium blob storage tier?

from the docs I read : Create a storage container where the metastore’s managed table data will be stored.This storage container must be in a Premium performance Azure Data Lake Storage Gen2 account in the same region as the workspaces you want to us...

  • 2358 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Carlos Restrepo​ We haven't heard from you since the last response from @Kaniz Fatma​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 1 kudos
1 More Replies
pranavyadavbugy
by New Contributor
  • 2119 Views
  • 1 replies
  • 0 kudos

Regarding Discount on certifications for students

Hi team,I'm a student is there any student discounts for students on certification if yes please let me know.Thanks

  • 2119 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Pranav Yadav​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
Phani1
by Valued Contributor
  • 579 Views
  • 1 replies
  • 0 kudos

Databricks Cross cloud (Databricks on AWS and Storage is on Azure)

We have a Databricks account on AWS and storage is on Azure Cloud (ADLS), because of cross-platform do we face any security and performance-related issues also is this feasible and supported? could you please provide your suggestions on this?

  • 579 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Janga Reddy​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
guille-ci
by New Contributor
  • 549 Views
  • 1 replies
  • 0 kudos

[bug] databricks jobs list not desplaying 2.0 created jobs

Hi! When I use `databricks jobs list --version=2.0` I get all jobs deployed using 2.0 and 2.1 API, however, when I use `databricks jobs list --version=2.1` I only get jobs deployed using 2.1 API. This is a behaviour that we've only experienced recent...

  • 549 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Guillermo Sanchez​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
marksachin_k
by New Contributor
  • 1565 Views
  • 1 replies
  • 0 kudos

Python custom Logging on Databricks

I am planning to introduce a custom logging to the databricks workload. To achieve this I am using a python logging module. I am storing logs in driver memory "file:/tmp/" directory before I move those logs to blob storage. In my personal databricks ...

  • 1565 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @MARKSACHIN K​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
abhilashmhaisne
by New Contributor II
  • 960 Views
  • 1 replies
  • 1 kudos

Error while creating workspace with 'Quickstart' on AWS

Cloudformation runs into error during 'assign Metastore' step.Following is the error from Cloudwatch:HTTP content: b'{ "error_code": "PERMISSION_DENIED", "message": "Cannot assign metastore to STANDARD tier workspace xyz" }What is the solution for th...

  • 960 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Abhilash Mhaisne​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
rdobbss
by New Contributor II
  • 3145 Views
  • 4 replies
  • 3 kudos

How to use foreachbatch in deltalivetable or DLT?

I need to process some transformation on incoming data as a batch and want to know if there is way to use foreachbatch option in deltalivetable. I am using autoloader to load json files and then I need to apply foreachbatch and store results into ano...

  • 3145 Views
  • 4 replies
  • 3 kudos
Latest Reply
TomRenish
New Contributor III
  • 3 kudos

Not sure if this will apply to you or not...I was looking at the foreachbatch tool to reduce the workload of getting distinct data from a history table of 20million + records because the df.dropDuplicates() function was intermittently running out of ...

  • 3 kudos
3 More Replies
Oliver_Angelil
by Valued Contributor II
  • 6956 Views
  • 3 replies
  • 3 kudos

New SQL Dashboard feature never loads

I've created the sample Taxi dashboard, but the charts never load?

Screenshot 2023-05-11 at 13.39.59
  • 6956 Views
  • 3 replies
  • 3 kudos
Latest Reply
jose_gonzalez
Moderator
  • 3 kudos

Do you have the full error stack trace? does this happens frequently? whats the size of your cluster?

  • 3 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels