- 3457 Views
- 3 replies
- 0 kudos
Jobs API "run now" - How to set task wise parameters
I have a job with multiple tasks like Task1 -> Task2 -> Task3. I am trying to call the job using api "run now". Task details are belowTask1 - It executes a Note Book with some input parametersTask2 - It runs using "ABC.jar", so its a jar based task ...
- 3457 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,It would be a good feature to pass parameters at task level. We have scenarios where we would like to create a job with multiple tasks (notebook/dbt) and pass parameters at task level.
- 0 kudos
- 1982 Views
- 1 replies
- 0 kudos
Uploading wheel using `dbutils.fs.cp` to workspace and install it in Runtime>15
I have been trying to find an alternative to copying a wheel file from my local file system to Databricks and then installing it into the cluster. Doing this databricks_client.dbutils.fs.cp("file:/local..../..whl", "dbfs:/Workspace/users/..../..whl")...
- 1982 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @safoineext , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedb...
- 0 kudos
- 1464 Views
- 1 replies
- 0 kudos
System Access Column lineage showing inaccurate results
Hi All,I have been trying to leverage the system column lineage table to check the overall journey of a column. But i am getting inaccurate results wherever unpivot transformations are used.Instead of showing the results in a way that 20 columns are ...
- 1464 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mahesh_Yadav , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your fee...
- 0 kudos
- 1330 Views
- 1 replies
- 0 kudos
How to get data permissions from Sharepoint and Confluence to Unity Catalog for RAG LLM chatbot
We're implementing a chatbot where documents in SharePoint and pages in Confluence augment the results. We want to adhere to existing RBAC policies in these data sources so that the chatbot doesn't produce results that someone should not see. Are you...
- 1330 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @beautrincia , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feed...
- 0 kudos
- 4009 Views
- 5 replies
- 3 kudos
How can I preserve the data type of the delta tables while writing to Azure blob storage ?
I am writing a file using this but the data type of columns get changed while reading. df.write.format("com.crealytics.spark.excel").option("header", "true").mode("overwrite").save(path) Due to this I have to manual change every time as I can't chang...
- 4009 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Tiwarisk , Thank you for reaching out to our community! We're here to help you.To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...
- 3 kudos
- 23282 Views
- 3 replies
- 2 kudos
Resolved! Executor memory increase limitation based on node type
Hi Databricks community,I'm using Databricks Jobs Cluster to run some jobs. I'm setting the worker and driver type to AWS m6gd.large, which has 2 cores and 8G of memory each.After seeing it's defaulting executor memory to 2G, I wanted to increase it,...
- 23282 Views
- 3 replies
- 2 kudos
- 2 kudos
I think I found the right answer here: https://kb.databricks.com/en_US/clusters/spark-shows-less-memoryIt seems it sets fixed size of ~4GB is used for internal node services. So depending on the node type, `spark.executor.memory` is fixed by Databric...
- 2 kudos
- 1286 Views
- 0 replies
- 0 kudos
SQL AI functions in EU region
I know that currently foundation model with pay-per-token are not available in EU only in US. In EU I should create serving point and use provisioned foundation model. But even creating a serving point with llm from catalog (share models). I used the...
- 1286 Views
- 0 replies
- 0 kudos
- 1225 Views
- 0 replies
- 0 kudos
Databricks Asset Bundle Error
Hello,I am trying the Databricks Asset bundle for the first time. I am using Databricks CLI and can able to validate the bundle but when I am trying to run it it errors out error="expected a KEY of the resource to run" .In the resource yml file I ha...
- 1225 Views
- 0 replies
- 0 kudos
- 1116 Views
- 1 replies
- 0 kudos
Autoloader - Ingestion of CSV files when there is not operation column
Hi,We are working on ingesting multiple files from S3. The files name are fixed based on our source system, Files get replaced frequently with full feed. In DLT when we process new file we have to delete the records processed earlier of the same file...
- 1116 Views
- 1 replies
- 0 kudos
- 5747 Views
- 3 replies
- 1 kudos
Resolved! Databricks JDBC SQL Warehouse Encoding Issue
Hi Everyone.I am trying to connect and read data from the Databricks table using SQL Warehouse and return it using Azure API.However, the non-English characters, for example, 'Ä', are present in the response as following: ��.I am using the databricks...
- 5747 Views
- 3 replies
- 1 kudos
- 1 kudos
If Databricks support/Product Management following the forum, note that PDF from SIMBA in 2.6.28 does not discuss the name-value pairs in the above solution.Other errata includes PreparedMetadataLimitZero.
- 1 kudos
- 1045 Views
- 1 replies
- 0 kudos
Serverless Compute Flexibility
If I want to use SSD or GPU in my cluster,is that possible to choose?
- 1045 Views
- 1 replies
- 0 kudos
- 0 kudos
As of now this setting is not configurable.
- 0 kudos
- 12507 Views
- 12 replies
- 3 kudos
Resolved! Unity catalog - How do you modify groups properly ?
Hello,What is the best practice to modify/delete/recreate groups properly ?In order to rename a group, the only mean was to delete/recreate. But after deletion in the account console, the permissions granted to the deleted groups in the tables were i...
- 12507 Views
- 12 replies
- 3 kudos
- 3 kudos
Hello,I have exactly the same issue - I am also using terraform.I deleted a group and the catalog permissions are in bad state. I am not able to revoke access to this group using the Databricks UI nor REST API. I also tried to recreate the group wit...
- 3 kudos
- 2938 Views
- 1 replies
- 2 kudos
Liquid Clustering - Number of files are increasing
We enabled liquid clustering on one of the large tables (380GBs). This table goes many operations daily, which improved many folds after liquid clustering. However, after enabling liquid clustering and optimizing it number of files are increased.Prev...
- 2938 Views
- 1 replies
- 2 kudos
- 2 kudos
Thank you for detailed explanation @Retired_mod .
- 2 kudos
- 4467 Views
- 1 replies
- 1 kudos
Resolved! Asset Bundles : how to conditionally set content of a template file
Hello,since Asset Bundles is based on GO templating mechanism, I am wondering how it is possible to use IF-ELSE construct within a template file, to define which file content will be set in the generated file ( I want to have that in my custom templa...
- 4467 Views
- 1 replies
- 1 kudos
- 1 kudos
Never mind....I figured out the solution I just have to prefix my template file with ".tmpl", then it gets rendered correctly
- 1 kudos
- 2079 Views
- 4 replies
- 0 kudos
Databricks Job Params
Hi,Job params override the task params(same name params). Is there a way task params override the job params.Use case:job params: a = "param-1".job has 12 tasks. 10 of them should use job param(a = "param-1").2 of them should override the job param(a...
- 2079 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @ajithgaade,Unfortunately, I don't think it is currently possible. It's clearly stated in documentation that:" Job parameters take precedence over task parameters. If a job parameter and a task parameter have the same key, the job parameter overri...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
