- 2831 Views
- 3 replies
- 0 kudos
Log jobs stdout to an Azure Logs Analytics workspace
Hello,I have enabled cluster logs sending through an mspnp/spark-monitoring, but I don't see there stdout/stderr/log4j logs.Is it supported?
- 2831 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @templier2 If it works, it’s not duct tape and chewing gum; it’s a paperclip away from advanced engineering! You're right, I forgot this option is only there for AWS/S3. So, yeah I think that's the current and only way, mount points.
- 0 kudos
- 3143 Views
- 6 replies
- 2 kudos
Resolved! Keep history of task runs in Databricks Workflows while moving it from one job to another
We are using Databricks Asset Bundles (DAB) to orchestrate multiple workflow jobs, each containing multiple tasks.The execution schedules is managed on the job level, i.e., all tasks within a job start together.We often face the issue of rescheduling...
- 3143 Views
- 6 replies
- 2 kudos
- 2 kudos
You can submit it through https://docs.databricks.com/en/resources/ideas.html#ideas
- 2 kudos
- 1863 Views
- 3 replies
- 0 kudos
Adobe query support from databricks
Hi Team, We are working with Adobe tool for campaign metrics. which needs to pull data from AEP using explode option, when we pass query it is taking long time and performance is also very. Is there any better way to pull data from AEP, Please le...
- 1863 Views
- 3 replies
- 0 kudos
- 5061 Views
- 2 replies
- 0 kudos
Invalid Path when getting Notebook Path
The undocumented feature to get a notebook path as is great but it does not actually return a valid path that can be used in python, e.g.:from pathlib import Pathprint(Path(dbutils.notebook.entry_point.getDbutils().notebook().getContext().notebookPat...
- 5061 Views
- 2 replies
- 0 kudos
- 0 kudos
I actually think the major issue is that the above is undocumented and not supported. A supported and documented way of doing this would be much appreciated.
- 0 kudos
- 11498 Views
- 10 replies
- 10 kudos
Delta Live Table name dynamically
Hi Team,Can we pass Delta Live Table name dynamically [from a configuration file, instead of hardcoding the table name]? We would like to build a metadata-driven pipeline.
- 11498 Views
- 10 replies
- 10 kudos
- 10 kudos
Is this post referring to Direct Publishing Mode? As we are multi-tenanted we have to have separate schema per client, which currently means a single pipeline per client. This is not cost effective at all, so we are very much reliant on DPM. I believ...
- 10 kudos
- 1808 Views
- 4 replies
- 0 kudos
Resolved! DABs job name must start with a letter or underscore
Hi,In UI I used the pipeline name 00101_source_bronze. I wanted to do the same in the Databricks Asset Bundles.but when the configuration is refreshed against Databricks Workspace I see this error:I found that this issue can be connect to Terraform v...
- 1808 Views
- 4 replies
- 0 kudos
- 0 kudos
As mentioned above, this is a limitation directly with Terraform due to this our engineering team is limited on the actions that can be done, you can find more information about this limitation on the Terraform documentation: https://developer.hashic...
- 0 kudos
- 5449 Views
- 1 replies
- 1 kudos
Resolved! workflow set maximum queued items
Hi all,I have a question regarding Workflows and queuing of job runs. I'm running into a case where jobs are running longer than expected and result in job runs being queued, which is expected and desired. However, in this particular case we only nee...
- 5449 Views
- 1 replies
- 1 kudos
- 1 kudos
Unfortunately there is no way to control the number of jobs that will be moved to queue status when queuing is enabled.
- 1 kudos
- 3402 Views
- 3 replies
- 0 kudos
Guidance Needed for Developing CI/CD Process in Databricks Using Azure DevOps
Hi everyone,I am working on setting up a complete end-to-end CI/CD process for my Databricks environment using Azure DevOps. So far, I have developed a build pipeline to create a Databricks artifact (DAB). Now, I need to create a release pipeline to ...
- 3402 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @alcatraz96 ,One question, why don't you use Databricks Assets Bundles? Then the whole process would be much simplerHere you have a good end to end example:CI/CD Integration with Databricks Workflows - Databricks Community - 81821
- 0 kudos
- 2666 Views
- 3 replies
- 1 kudos
Update set in foreachBatch
I need to track codes of records that were ingested in foreachBatch function, and pass it as a task value, so downstream tasks can take actions based on this output. What would be the best approach to achieve that? Now, I have a following solution, b...
- 2666 Views
- 3 replies
- 1 kudos
- 1 kudos
Another approach is to persist the collected codes in a Delta table and then read from this table in downstream tasks. Make sure to add ample logging and counts. Checkpointing also would help if you suspect the counts in set are not the same as what ...
- 1 kudos
- 3923 Views
- 1 replies
- 0 kudos
Databricks sql variables and if/else workflow
I have 2 tasks in databricks job workflow first task is of type SQL and SQL task is query.In that query I've declared 2 variables and SET the values by running query.e.g:DECLARE VARIABLE max_timestamp TIMESTAMP DEFAULT '1970-01-01'; SET VARIABLE max_...
- 3923 Views
- 1 replies
- 0 kudos
- 0 kudos
Please try with max_timestamp = dbutils.jobs.taskValues("sql_task_1")["max_timestamp"] dbutils.jobs.taskValues("python_task_1", {"max_timestamp": max_timestamp}) Reference- https://docs.databricks.com/en/jobs/task-values.html
- 0 kudos
- 2470 Views
- 3 replies
- 1 kudos
ABFS Authentication with a SAS token -> 403!
Hi guys,I'm running a streamReader/Writer with autoloader from StorageV2 (general purpose v2) over abfss instead of wasbs. My checkpoint location is valid, the reader properly reads the file schema and autoloader is able to sample 105 files to do so....
- 2470 Views
- 3 replies
- 1 kudos
- 1 kudos
Would you mind to paste the sample code please. I am trying to use abfs with autoloader and getting error like yours.
- 1 kudos
- 5408 Views
- 3 replies
- 1 kudos
Resolved! SSIS packages migration to Databricks Workflows
We are doing POC to migrate SSIS packages to Databricks workflows as part of our effort to build the analytics layer, including dimension and fact tables. How can we accelerate or automate the SSIS package migration to Databricks environment?
- 5408 Views
- 3 replies
- 1 kudos
- 7385 Views
- 5 replies
- 2 kudos
Resolved! Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task`
If I create a job from the web UI and I select Python wheel, I can add kwargs parameters. Judging from the generated JSON job description, they appear under a section named `namedParameters`.However, if I use the REST APIs to create a job, it appears...
- 7385 Views
- 5 replies
- 2 kudos
- 2 kudos
@GabrieleMuciacc , in case of serverless compute job this can be pass as external dependency you can't use libraries. "tasks": [{ "task_key": task_id, "spark_python_task": { "python_file": py_file, ...
- 2 kudos
- 4689 Views
- 1 replies
- 0 kudos
Pool clusters and init scripts
Hey, just trying out pool clusters and providing the instance_pool_type and driver_instance_pool_id configuration to the Airflow new_cluster fieldI also pass the init_scripts field with an s3 link as usual but it this case of pool clusters it doesn't...
- 4689 Views
- 1 replies
- 0 kudos
- 0 kudos
When using a non pool cluster are you able to see the init script being deployed? You could set init script logging to see if it is being called or not at all https://docs.databricks.com/en/init-scripts/logs.html
- 0 kudos
- 6444 Views
- 1 replies
- 0 kudos
Managing Secrets for Different Groups in a Databricks Workspace
Hi everyone,I'm looking for some advice on how people are managing secrets within Databricks when you have different groups (or teams) in the same workspace, each requiring access to different sets of secrets.Here’s the challenge:We have multiple gro...
- 6444 Views
- 1 replies
- 0 kudos
- 0 kudos
Managing secrets within Databricks when you have different groups or teams in the same workspace can be approached in several ways, each with its own advantages. Here are some best practices and methods based on the context provided: Using Azure Key...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
