- 1549 Views
- 2 replies
- 2 kudos
Resolved! Read Files from Adobe and Push to Delta table ADLS Gen2
The Upstream is sending 2 files of different schema. The Storage Account has Private Endpoints. there is no public access.no public IP (NPIP) = yes.How to design using only Databricks :-1. Databricks API to read data file from Adobe and Push it to AD...
- 1549 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Pratikmsbsvm ,Okay, since you’re going to use Databricks compute for data extraction and you wrote that your workspace is deployed with the secure connectivity cluster (NPIP) option enabled, you first need to make sure that you have a stable egre...
- 2 kudos
- 6056 Views
- 5 replies
- 2 kudos
Resolved! Managing libraries in workflows with multiple tasks - need to configure a list of libs for all tasks
I have workflows with multiple tasks, each of which need 5 different libraries to run. When I have to update those libraries, I have to go in and make the update in each and every task. So for one workflow I have 20 different places where I have to g...
- 6056 Views
- 5 replies
- 2 kudos
- 2 kudos
Actually I think I found most of a solution here in one of the replies: https://community.databricks.com/t5/administration-architecture/installing-libraries-on-job-clusters/m-p/37365/highlight/true#M245It seems like I only have to define libs for the...
- 2 kudos
- 1027 Views
- 1 replies
- 2 kudos
How to work with 300 billions rows and 5 columns?
Hi guys!I'm having a problem at work where I need to process a customer data dataset with 300 billion rows and 5 columns. The transformations I need to perform are "simple," like joins to assign characteristics to customers. And at the end of the pro...
- 1027 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @guilhermecs001 ,Wow, that's massive amount of rows. Can you somehow preprocess first this huge CSV file? For example, read CSV, partition by some columns that makes sense (maybe country from which customer is coming from) and save that data as de...
- 2 kudos
- 958 Views
- 1 replies
- 2 kudos
Is Photon Acceleration Helpful for All Maintenance Tasks (OPTIMIZE, VACUUM, ANALYZE_COMPUTE_STATS)?
Hi everyone,We’re currently reviewing the performance impact of enabling Photon acceleration on our Databricks jobs, particularly those involving table maintenance tasks. Our job includes three main operations: OPTIMIZE, VACUUM, and ANALYZE_COMPUTE_S...
- 958 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Sainath368 ,I wouldn't use photon for this kind of task. You should use it primarly for ETL transformations where it shines.VACUUM and OPTIMIZE are more of maintenance tasks and using photon would be pricey overkill here.According to documentatio...
- 2 kudos
- 17794 Views
- 13 replies
- 7 kudos
Value array {{QUERY_RESULT_ROWS}} in Databricks SQL alerts custom template
Please include in documentation an example how to incorporate the `QUERY_RESULT_ROWS` variable in the custom template.
- 17794 Views
- 13 replies
- 7 kudos
- 7 kudos
Databricks confirmed this was an issue on their end and it should be resolved now. It is working for me.
- 7 kudos
- 1653 Views
- 2 replies
- 1 kudos
Resolved! cosmosdb metadata integration with unity catalog
Hi Team,How can we integrate Cosmos DB Meta data with Unity Catalog, can you please provide some insights on this?Regards,Phani
- 1653 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Phani1 Good day:I have found a whole document on your requirementshttps://community.databricks.com/t5/technical-blog/optimising-data-integration-and-serving-patterns-with-cosmos-db/ba-p/91977 It has a project with it as well.
- 1 kudos
- 1089 Views
- 1 replies
- 0 kudos
Resolved! How to build Data Pipeline to consume data from Adobe Campaign to Azure Databricks
May Techie please help me design the pipeline with Databricks.I don't have any control over Adobe.How to set up a data pipeline that moves csv files from Adobe to ADLS Gen2 via a cron job, using Databricks.where this cron job will execute ? how ADLS ...
- 1089 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Datalight Good day!Can I please know what do you mean by "you dont have any control over adobe"? I found similar case study over here: https://learn.microsoft.com/en-us/answers/questions/5533633/data-pipeline-to-push-files-from-external-system...
- 0 kudos
- 2938 Views
- 8 replies
- 5 kudos
Resolved! Cannot deploy DAB with the Job branch using a feature branch in Workspace UI
Hi, I tried to deploy DAB on Workspace UI with a feature branch (sf-trans-seq) targeted to Dev. After deploying successfully, the Job branch is, however, using the master branch (see the screenshot below).Is there any option to force the Job branch t...
- 2938 Views
- 8 replies
- 5 kudos
- 5 kudos
I agree.Can you mark your (or someone else´s) answer as solved? Because I think you won´t be the only one with this issue/feature.
- 5 kudos
- 1778 Views
- 1 replies
- 1 kudos
Resolved! Mongodb connection in GCP Databricks
I am trying to connect with Mongodb from databricks which is UC enabled, and both the mongodb and databricks are in same VPC, I am using the below code, df = ( spark.read.format("mongodb") .option( "connection.uri", f'''mongodb://{username}:{password...
- 1778 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @xavier_db ,Standard access mode has more limitations compared to dedicate access mode. For example, look at the limitations list of standard access mode:Standard compute requirements and limitations | Databricks on AWSNow, compare it to dedicated...
- 1 kudos
- 5518 Views
- 3 replies
- 0 kudos
Cannot run another notebook from same directory
Hello, I am having a similar problem from this thread which was never resolved: https://community.databricks.com/t5/data-engineering/unexpected-error-while-calling-notebook-string-matching-regex-w/td-p/18691 I renamed a notebook (utility_data_wrangli...
- 5518 Views
- 3 replies
- 0 kudos
- 0 kudos
I am running into the same issue. It seems like the `%run` magic command is trying to parse the entire cell content as its arguments. So if you have%run "my_notebook" print("hello")in the same cell, you will get the following error: `Failed to parse...
- 0 kudos
- 8999 Views
- 9 replies
- 12 kudos
Resolved! Pass Notebook parameters dynamically in Job task.
Hi Everyone, I'm working on scheduling a job and would like to pass parameters that I've defined in my notebook. Ideally, I'd like these parameters to be dynamic meaning that if I update their values in the notebook, the scheduled job should automati...
- 8999 Views
- 9 replies
- 12 kudos
- 12 kudos
I see you're using dbutils.widgets. text and dropdown—perfect! You're already on the right track.Quick SolutionYour widgets are already dynamic! Just pass parameters in your job configuration:In your notebook (slight refactor of your code):# Define w...
- 12 kudos
- 23364 Views
- 13 replies
- 8 kudos
Grafana + databricks = True?
We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find any way to con...
- 23364 Views
- 13 replies
- 8 kudos
- 8 kudos
@Erik wrote:We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find an...
- 8 kudos
- 5288 Views
- 6 replies
- 5 kudos
Resolved! Can schemaHints dynamically handle nested json structures?
Hi there, as I'm learning more about schemaHints, it seems like an incredibly useful way to unpack some of my json data. However, I've hit what is either a limitation of schemaHints or of my understanding of how to use it properly.Below I have an exa...
- 5288 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @ChristianRRL,Would you be able to share a small sample of your JSON file (with any sensitive data removed)? That way, I can try to replicate your use case and see if we can get schemaHints working across multiple nested fields without losing data...
- 5 kudos
- 15286 Views
- 8 replies
- 3 kudos
How to fix intermittent 503 errors in 10.4 LTS
I sometimes get the below error recently in version 10.4 LTS. Any solution to fix the intermittent failure? I added retry logic in our code, but Databricks query succeeded (even though it threw an exception) and it leads to the unexpected table statu...
- 15286 Views
- 8 replies
- 3 kudos
- 1346 Views
- 2 replies
- 1 kudos
Prakash Hinduja ~ How do I create an empty DataFrame in Databricks—are there multiple ways?
Hello, I'm Prakash Hinduja, an Indian-born financial advisor and consultant based in Geneva, Switzerland (Swiss). My career is focused on guiding high-net-worth individuals and business leaders through the intricate world of global investment and wea...
- 1346 Views
- 2 replies
- 1 kudos
- 1 kudos
Best Practices from Experience:Use predefined schema if you know your column types upfront—prevents errors when appending new data.For ad-hoc exploration, toDF or createDataFrame([], None) works fine.Always check printSchema()—it helps avoid silent t...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
