cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

shubham_meshram
by New Contributor II
  • 1111 Views
  • 2 replies
  • 2 kudos

Databrick Dashboards Maps

I am currently unable to use the MAPS (Choropleth) feature in databricks dashboard, however I am able to use it in the legacy dashboard. I would like my users to use a single dashboard that has all the combined features than exposing them to 2 differ...

  • 1111 Views
  • 2 replies
  • 2 kudos
Latest Reply
shubham_meshram
New Contributor II
  • 2 kudos

Thanks for your input @Brahmareddy , I just met some folks from Databricks at a booth in Missouri and they confirmed April end would be the planned timeline for the maps feature release. Thanks for your input

  • 2 kudos
1 More Replies
jeremy98
by Honored Contributor
  • 3243 Views
  • 1 replies
  • 0 kudos

where saving the wheel package?

Hi community,We have deployed the wheel package internally in our bundle repository: artifacts: rnc_lib: type: whl build: poetry build path: . # For passing wheel package to workspace sync: include: - ./dist/*.whlThe problem is t...

  • 3243 Views
  • 1 replies
  • 0 kudos
Latest Reply
Renu_
Valued Contributor II
  • 0 kudos

Hi @jeremy98 You can upload the wheel to a shared workspace location and configure it for cluster-level installation by attaching it as a library.Or you can also automate the process by adding the wheel to the libraries section of your databricks.yml...

  • 0 kudos
naineel
by New Contributor
  • 525 Views
  • 1 replies
  • 0 kudos

Scheduling a Complete Python Project in Databricks

Hi everyone,I have a simple Python project with the following structure:root/ │── src/ │ ├── package_name/ │ │ ├── __init__.py │ │ ├── main.py │ │ ├── submodules1/ │ │ │ ├── __init__.py │ │ │ ├── base1.py │ ...

  • 525 Views
  • 1 replies
  • 0 kudos
Latest Reply
ashraf1395
Honored Contributor
  • 0 kudos

Hi there @naineel , one approach can be you can convert your project into a whl fileand then create a python whl task for it and schedulehttps://docs.databricks.com/aws/en/jobs/python-wheel

  • 0 kudos
Subhasis
by New Contributor III
  • 731 Views
  • 2 replies
  • 0 kudos

Unity catalog codebase

SpoilerHow can I get the code base of unity catalog?How can I get the code base of unity catalog?

  • 731 Views
  • 2 replies
  • 0 kudos
Latest Reply
Subhasis
New Contributor III
  • 0 kudos

Here I am mentioning about the .dbc file which databricks used to provide earlier for practice the query. Hence is it possible to get that query? 

  • 0 kudos
1 More Replies
gchandra
by Databricks Employee
  • 2848 Views
  • 5 replies
  • 3 kudos

Resolved! Databricks Community Edition - DBFS Alternative Solutions

Option 1: Mount AWS S3 bucket access_key = ""secret_key = ""encoded_secret_key = secret_key.replace("/", "%2F") aws_bucket_name = "yourawsbucketname/"mount_name = "youraliasmountname" # #dbutils.fs.unmountf"/mnt/{mount_name}")dbutils.fs.mount(f"s3a:/...

  • 2848 Views
  • 5 replies
  • 3 kudos
Latest Reply
DanT
New Contributor II
  • 3 kudos

Seems to be removed again? Can't see options. 

  • 3 kudos
4 More Replies
jhgorse
by New Contributor III
  • 2689 Views
  • 1 replies
  • 0 kudos

mqtt to Delta Live Table

Greetings,I see that Delta Live Tables has various real-time connectors such as Kafka, Kinesis, Google's Pub Sub, and so on. I also see that Apache had maintained an mqtt connector to Spark through the 2.x series called Bahir, but dropped it in versi...

  • 2689 Views
  • 1 replies
  • 0 kudos
Latest Reply
KK0001
New Contributor II
  • 0 kudos

How this solution end up?

  • 0 kudos
DataGirl
by New Contributor
  • 16221 Views
  • 6 replies
  • 2 kudos

Multi value parameter on Power BI Paginated / SSRS connected to databricks using ODBC

Hi All, I'm wondering if anyone has had any luck setting up multi valued parameters on SSRS using ODBC connection to Databricks? I'm getting "Cannot add multi value query parameter" error everytime I change my parameter to multi value. In the query s...

  • 16221 Views
  • 6 replies
  • 2 kudos
Latest Reply
ssrsnat
New Contributor II
  • 2 kudos

Hi I am working on having SSRS reports access Databricks and facing similar challenges. I see you tried this back in 2022. Can you please advice the approach to handle the multi value parameters? Thanks Sam

  • 2 kudos
5 More Replies
alex_crow
by New Contributor II
  • 17583 Views
  • 7 replies
  • 1 kudos

ModuleNotFoundError: No module named 'databricks.sdk' in module installed via Pip

Hello. I'm currently having an issue that I simply cannot understand nor find an adequate work-around for. Recently, my team within our organization has undergone the effort of migrating our Python code from Databricks notebooks into regular Python m...

  • 17583 Views
  • 7 replies
  • 1 kudos
Latest Reply
ferdinand
New Contributor II
  • 1 kudos

Lol, OK so in my case it was because I had a file called databricks.py with clashed with the installed databricks. Renaming my file to databricks_utils.py solved it. 

  • 1 kudos
6 More Replies
Niil
by New Contributor II
  • 1687 Views
  • 1 replies
  • 2 kudos

Resolved! AI Agents in ETL

Hi, I recently found a blog online about Databricks using AI Agents to automate ETL, but I can't find where these capabilities are located in Databricks. Does anyone know?Here is the blog,https://www.heliverse.com/blog/databricks-ai-agents-streamlini...

  • 1687 Views
  • 1 replies
  • 2 kudos
Latest Reply
santhakumar11
New Contributor III
  • 2 kudos

Hi NiilDatabrick has introduced AI Agent sub-categories as part of its Generative AI capabilities. We can now automate tasks such as Extract, Transformation, and Load (ETL).Ex,  Information Extraction Agent - we can able to transform large a volume  ...

  • 2 kudos
marcelhfm
by New Contributor II
  • 2630 Views
  • 7 replies
  • 0 kudos

Internal GRPC errors when using databricks connect

Hey there, in our local development flow we heavily rely on databricks asset bundles and databricks connect. Recently, locally run workflows (i.e. just pyspark python files) have begun to frequently fail with the following grpc error:pyspark.errors.e...

  • 2630 Views
  • 7 replies
  • 0 kudos
Latest Reply
ChrisChieu
Databricks Employee
  • 0 kudos

@marcelhfm it might be a Spark Connect issueI would say it is the same for the rest of you, guys Nothing much to do until the situation is fixed by Databricks 

  • 0 kudos
6 More Replies
sparkycloud
by New Contributor II
  • 1437 Views
  • 3 replies
  • 1 kudos

Observability and monitoring accross multiple workspaces(both job clusters and serverless compute)

Hi all, Today what are the best option available today for observability and monitoring databricks jobs accross all workspaces. We have 100 of workspaces and it hard to do monitoring to check failed and successeded jobs.We tried using: 1. Team webhoo...

  • 1437 Views
  • 3 replies
  • 1 kudos
Latest Reply
sparkycloud
New Contributor II
  • 1 kudos

Hey Brahmareddy, Thanks so much for responding. Sorry I forgot to mention we are in Azure. Lets go through one by one.1. Audit logs(Azure Monitor) : AFAIK this requires init scripts  and jar build that will not support in serverless or its not the ca...

  • 1 kudos
2 More Replies
ashraf1395
by Honored Contributor
  • 1663 Views
  • 2 replies
  • 1 kudos

Resolved! Not readable format of event_logs traceback in dlt pipeline

This is my dlt pipeline event_log - why is it not in readable foramt how can i correct it.This is my pipeline code :   import logging logger = logging.getLogger(__name__) logger.info("Error") raise "Error is error"    

ashraf1395_0-1740044260033.png
  • 1663 Views
  • 2 replies
  • 1 kudos
Latest Reply
jorperort
Contributor
  • 1 kudos

Hi @ashraf1395 ,I'm working with Delta Live Tables (DLT) and the event_log table. I would like to know if it is possible to access the event handler that DLT uses to write custom logs and send them to this table when events are published.If this is n...

  • 1 kudos
1 More Replies
susanne
by Contributor
  • 955 Views
  • 2 replies
  • 1 kudos

How to write event_log destination into DLT Settings JSON via Asset Bundles

Hi all,I would like to publish the event_log of my DLT Pipeline to a specific schema in Unity Catalog.Following this article (https://docs.databricks.com/gcp/en/dlt/observability#query-the-event-log) this can be done by writing this into the DLTs set...

  • 955 Views
  • 2 replies
  • 1 kudos
Latest Reply
ashraf1395
Honored Contributor
  • 1 kudos

Hi @susanne , indeed , i tried to create it using dabs as well. This feature is not available using dabs I guess, maybe they will add it once event_logs moves to ga from public preview.databricks API will be a good alternative but if you try it using...

  • 1 kudos
1 More Replies
21f3001806
by New Contributor III
  • 806 Views
  • 1 replies
  • 1 kudos

Resolved! Creating event_log for dlt pipeline using dabs

Hi there, i have dlt pipeline and I recently came to know about the event_log feature, i want to deploy my dlt pipeline along with the event_log using databricks asset bundles but i am not able to find any resources for it.If anyone has tried it , yo...

  • 806 Views
  • 1 replies
  • 1 kudos
Latest Reply
ashraf1395
Honored Contributor
  • 1 kudos

Hi there @21f3001806 ,I guess you are talking about the this : https://docs.databricks.com/api/workspace/pipelines/create#event_logIts still in public preview I tried creating it through UI or by chaing the pipeline settings it worked. But when I imp...

  • 1 kudos
DBStudent
by New Contributor II
  • 760 Views
  • 3 replies
  • 0 kudos

Data migration from S3 to Databricks

I currently have an S3 bucket with around ~80 tables, each of which has hive-style partition columns  S3RootFolder/Table1Name/Year=2024/Month=12/Day=1/xxx.parquetS3RootFolder/Table1Name/Year=2024/Month=12/Day=2/xxx.parquetS3RootFolder/Table2Name/Year...

  • 760 Views
  • 3 replies
  • 0 kudos
Latest Reply
Brahmareddy
Esteemed Contributor
  • 0 kudos

Hi DBStudent,You’re absolutely right—coalescing during write won’t help with the initial bottleneck, since the issue is really with the read side, where Spark has to list and open 110k small files. If you can’t pre-compact them before reading, then o...

  • 0 kudos
2 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels