- 749 Views
- 1 replies
- 0 kudos
Resolved! How to build Data Pipeline to consume data from Adobe Campaign to Azure Databricks
May Techie please help me design the pipeline with Databricks.I don't have any control over Adobe.How to set up a data pipeline that moves csv files from Adobe to ADLS Gen2 via a cron job, using Databricks.where this cron job will execute ? how ADLS ...
- 749 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Datalight Good day!Can I please know what do you mean by "you dont have any control over adobe"? I found similar case study over here: https://learn.microsoft.com/en-us/answers/questions/5533633/data-pipeline-to-push-files-from-external-system...
- 0 kudos
- 2311 Views
- 8 replies
- 5 kudos
Resolved! Cannot deploy DAB with the Job branch using a feature branch in Workspace UI
Hi, I tried to deploy DAB on Workspace UI with a feature branch (sf-trans-seq) targeted to Dev. After deploying successfully, the Job branch is, however, using the master branch (see the screenshot below).Is there any option to force the Job branch t...
- 2311 Views
- 8 replies
- 5 kudos
- 5 kudos
I agree.Can you mark your (or someone else´s) answer as solved? Because I think you won´t be the only one with this issue/feature.
- 5 kudos
- 1329 Views
- 1 replies
- 1 kudos
Resolved! Mongodb connection in GCP Databricks
I am trying to connect with Mongodb from databricks which is UC enabled, and both the mongodb and databricks are in same VPC, I am using the below code, df = ( spark.read.format("mongodb") .option( "connection.uri", f'''mongodb://{username}:{password...
- 1329 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @xavier_db ,Standard access mode has more limitations compared to dedicate access mode. For example, look at the limitations list of standard access mode:Standard compute requirements and limitations | Databricks on AWSNow, compare it to dedicated...
- 1 kudos
- 4909 Views
- 3 replies
- 0 kudos
Cannot run another notebook from same directory
Hello, I am having a similar problem from this thread which was never resolved: https://community.databricks.com/t5/data-engineering/unexpected-error-while-calling-notebook-string-matching-regex-w/td-p/18691 I renamed a notebook (utility_data_wrangli...
- 4909 Views
- 3 replies
- 0 kudos
- 0 kudos
I am running into the same issue. It seems like the `%run` magic command is trying to parse the entire cell content as its arguments. So if you have%run "my_notebook" print("hello")in the same cell, you will get the following error: `Failed to parse...
- 0 kudos
- 6041 Views
- 9 replies
- 12 kudos
Resolved! Pass Notebook parameters dynamically in Job task.
Hi Everyone, I'm working on scheduling a job and would like to pass parameters that I've defined in my notebook. Ideally, I'd like these parameters to be dynamic meaning that if I update their values in the notebook, the scheduled job should automati...
- 6041 Views
- 9 replies
- 12 kudos
- 12 kudos
I see you're using dbutils.widgets. text and dropdown—perfect! You're already on the right track.Quick SolutionYour widgets are already dynamic! Just pass parameters in your job configuration:In your notebook (slight refactor of your code):# Define w...
- 12 kudos
- 21602 Views
- 13 replies
- 8 kudos
Grafana + databricks = True?
We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find any way to con...
- 21602 Views
- 13 replies
- 8 kudos
- 8 kudos
@Erik wrote:We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find an...
- 8 kudos
- 3773 Views
- 6 replies
- 5 kudos
Resolved! Can schemaHints dynamically handle nested json structures?
Hi there, as I'm learning more about schemaHints, it seems like an incredibly useful way to unpack some of my json data. However, I've hit what is either a limitation of schemaHints or of my understanding of how to use it properly.Below I have an exa...
- 3773 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @ChristianRRL,Would you be able to share a small sample of your JSON file (with any sensitive data removed)? That way, I can try to replicate your use case and see if we can get schemaHints working across multiple nested fields without losing data...
- 5 kudos
- 14096 Views
- 8 replies
- 3 kudos
How to fix intermittent 503 errors in 10.4 LTS
I sometimes get the below error recently in version 10.4 LTS. Any solution to fix the intermittent failure? I added retry logic in our code, but Databricks query succeeded (even though it threw an exception) and it leads to the unexpected table statu...
- 14096 Views
- 8 replies
- 3 kudos
- 993 Views
- 2 replies
- 1 kudos
Prakash Hinduja ~ How do I create an empty DataFrame in Databricks—are there multiple ways?
Hello, I'm Prakash Hinduja, an Indian-born financial advisor and consultant based in Geneva, Switzerland (Swiss). My career is focused on guiding high-net-worth individuals and business leaders through the intricate world of global investment and wea...
- 993 Views
- 2 replies
- 1 kudos
- 1 kudos
Best Practices from Experience:Use predefined schema if you know your column types upfront—prevents errors when appending new data.For ad-hoc exploration, toDF or createDataFrame([], None) works fine.Always check printSchema()—it helps avoid silent t...
- 1 kudos
- 5782 Views
- 1 replies
- 0 kudos
Azure Databricks to Adobe Experience Platform
I'm using Azure databricks and am attempting to stream near real-time data from databricks into the Adobe Experience Platform to ingest into the AEP schema for profile enrichment.We are running into an issue with the API and streaming, so we are curr...
- 5782 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Tony... there are several ways to accomplish this based on the non-functional requirements of your Use Case. What does near real-time mean from a standpoint of signal to activation time from when the data is present in Databricks to when the acti...
- 0 kudos
- 2648 Views
- 2 replies
- 0 kudos
Use Salesforce Lakeflow Connector with a Salesforce Connected App
Hello, I'm trying to use the new Salesforce Lakeflow connector to ingest data into my Databricks account.However I see only the option to connect using a normal user, whereas I want to use a Salesforce App, just like how it is described here Run fede...
- 2648 Views
- 2 replies
- 0 kudos
- 0 kudos
@slimbnsalah Please select Connection type as of Salesforce Data Cloud then you will be asked for details
- 0 kudos
- 1243 Views
- 4 replies
- 2 kudos
Resolved! Silver to Gold Layer | Running ML - Debug Help Needed
Problem I am solving:Reads the raw sports data IPL CSV → bronze layerCleans and aggregates → silver layerSummarizes team stats → gold layerPrepares ML-ready features and trains a Random Forest classifier to predict match winners Getting error: [PARS...
- 1243 Views
- 4 replies
- 2 kudos
- 2 kudos
@ManojkMohan thanks for sharing this, I'm looking at starting an ML project in the coming weeks, I might have to bring this forward . Feeling motivated with that confusion matrix in your output .Congrats on getting it working!All the best,BS
- 2 kudos
- 1512 Views
- 6 replies
- 3 kudos
Jar File Upload To Workspace
Spoiler #dbfsI am unable to upload jar file dbfs to job cluster as it's deprecated now I need to upload it to workspace and install it to cluster, hower my jar size is 70mb i can't upload it through api or cli as max size is 50mb. Is there alternati...
- 1512 Views
- 6 replies
- 3 kudos
- 3 kudos
Hi @Srinivas5! Were you able to find a solution or approach that worked? If so, please mark the helpful reply as the Accepted Solution, or share your approach so others can benefit as well.
- 3 kudos
- 740 Views
- 2 replies
- 0 kudos
Notebook exposure
i have created a notebook as per client requirement. I have to migrate the notebook in the client env for testing with live data but do not want to expose the Databricks notebook code to the testers in the client env.Is there a way to package the not...
- 740 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @ShankarM,I’ve had to do something similar—packaging a Python class as a wheel. This documentation might help: https://docs.databricks.com/aws/en/dev-tools/bundles/python-wheel
- 0 kudos
- 2370 Views
- 2 replies
- 1 kudos
Resolved! databricks assets bundles issue
Hii all,I’m working with Databricks Asset Bundles (DAB) and trying to move from a single repository-level bundle to a structure where each workflow (folder under resources/jobs) has its own bundle.• My repository contains:• Shared src/variables.yml a...
- 2370 Views
- 2 replies
- 1 kudos
- 1 kudos
I solved it.For some reason, the Terraform folder created under the bundles wasn’t set up correctly.I copied it from a working bundle, and everything completed successfully.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
