- 2381 Views
- 3 replies
- 0 kudos
Resolved! How to grant custom container AWS credentials for reading init script?
I'm using a customer container *and* init scripts. At runtime, I get this error:Cluster '...' was terminated. Reason: INIT_SCRIPT_FAILURE (CLIENT_ERROR). Parameters: instance_id:i-0440ddd3a2d5cce79, databricks_error_message:Cluster scoped init script...
- 2381 Views
- 3 replies
- 0 kudos
- 0 kudos
Followup: I got the AWS creds working by amending our AWS role to permit read/write access to our S3 bucket. Woohoo!
- 0 kudos
- 2724 Views
- 3 replies
- 0 kudos
Resolved! Format when specifying docker_image url?
I am providing a custom Docker image to my Databricks/Spark job. I've created the image and uploaded it to our private ECR registry (the URL is `472542229217.dkr.ecr.us-west-2.amazonaws.com/tectonai/mrstevegross-testing:latest`). Based on the docs (h...
- 2724 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks, that's pretty much what I did; a lot of terraform configuration to get the AWS account set up properly, and now I'm able to tell DBR to load the container. (FWIW, I'm encountering *new* access issues; I started a thread here (https://communit...
- 0 kudos
- 1743 Views
- 1 replies
- 1 kudos
Unable to Register Models After Uploading Artifacts to DBFS in Databricks
Hi everyone,I'm currently working on a project where I'm migrating models and artifacts from a source Databricks workspace to a target one. I've written a script to upload the model artifacts from my local system to DBFS in the target workspace (usi...
- 1743 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Sudheer2, Does it give you any error while trying to register the model?
- 1 kudos
- 1645 Views
- 2 replies
- 0 kudos
Scheduling multiple jobs (workflows) in DABs
Hello, I'm wondering how can I schedule multiple jobs (workflow).I'd like to do something like this but on a workflow level. tasks: - task_key: task_1 sql_task: warehouse_id: ${var.warehouse_id} paramet...
- 1645 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Greg_c, You can try with this sctructure: - In the main databricks.yml # databricks.ymlbundle:name: master-bundle include:- resources/*.yml # Other bundle configurations... In resource directory, create a YAML for each job: # resources/job1.ymlre...
- 0 kudos
- 5796 Views
- 4 replies
- 1 kudos
ALIAS Not accepted 42601
I am unable to run the following query generated from my backend at databricks sideQuery: SELECT "A".`cut` AS "Cut" , "A".`color` AS "Color" , "A".`carat` AS "Carat" , "A".`clarity` AS "Clarity" FROM databricksconnect.default.diamonds "A" Error logs...
- 5796 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @malhm ,Double quotes are not supported in column alias. In Databricks SQL/Spark SQL one uses backticks instead of double quotes like in PostgreSQL.Check the docs:https://spark.apache.org/docs/3.5.1/sql-ref-identifier.html
- 1 kudos
- 6442 Views
- 3 replies
- 0 kudos
Resolved! DataBricks x Query Folding Power BI
I ran a native Power BI query in DataBricks in import mode and query folding was not enabled. No query folding?
- 6442 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Iguinrj11 ,The trick is to configure Databricks.Query instead of Databricks.Catalogs.Check this article and let us know if that helps:https://www.linkedin.com/pulse/query-folding-azure-databricks-tushar-desai/
- 0 kudos
- 2357 Views
- 2 replies
- 0 kudos
Dynamic Bloom Filters for Inner Joins
I have a question regarding combining the use of Bloom filters with Liquid Clustering to further reduce the data read during a join/merge on top of dynamic file pruning. Testing both combined worked extremely well together for point queries. However ...
- 2357 Views
- 2 replies
- 0 kudos
- 0 kudos
We do not recommend Bloom filters Index on the Delta Tables as they have to be manually maintained. If you prefer photon - please try predictive I/O with Liquid Clustering.
- 0 kudos
- 2576 Views
- 4 replies
- 0 kudos
Automate run as workflow parameter to default to current user
I am trying to run a workflow within Databricks. I have 2 workflows, workflow one which always runs as the service principal, as all data gets accessed and wrangled within this workflow, and workflow 2 which always defaults to the last run account. I...
- 2576 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, how are you expecting to achieve this? Do you want users who are manually triggering this workflow first update to their run_as? or you want to make this happen programatically?
- 0 kudos
- 1684 Views
- 2 replies
- 0 kudos
Create csv and upload on azure
Can some write a sql query , which queries a table like select * from stages.benefit , creates a csv and upload on azure
- 1684 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @subhadeep ,You can achieve this in SQL similarly to how you write a dataframe into a table or blob path. We will create an external table pointing to the blob path or mounted blob path. Note that this table does not support ACID transactions and ...
- 0 kudos
- 3679 Views
- 10 replies
- 2 kudos
01_demo_setup error
HelloI was following "Demo: Creating and Working with a Delta Table"while I have a community edition user.The first command in the Notebook is: %run ./setup/01_demo_setup But I got the following error:Notebook not found: Users/<my-email-was-here..>/s...
- 3679 Views
- 10 replies
- 2 kudos
- 2 kudos
Hey!Sad news guys... if you go to Course Logistics Review you can read:"We are pleased to offer a version of this course that also contains hands-on practice via a Databricks Academy Labs subscription. With a Databricks Academy Labs subscription, you...
- 2 kudos
- 1835 Views
- 1 replies
- 0 kudos
Databricks app giving 'upstream request timeout '
Hello all,We are developing an app which is based on flask, which is used to download logs from databricks dbfs location. For this useful case we are using databricks inbuilt App feature to deploy our app.While we pass a smaller file it is getting do...
- 1835 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey!It looks like the issue you’re facing might be related to the proxy timeout when downloading large files from DBFS. Since modifying the proxy settings might not be an option, there are a couple of alternative approaches you could consider to miti...
- 0 kudos
- 8210 Views
- 3 replies
- 0 kudos
50%-off Databricks certification voucher
Hello Databricks Community Team, I am reaching out to inquire about the Databricks certification voucher promotion for completing the Databricks Learning Festival (Virtual) courses.I completed one of the Databricks Learning Festival courses July 2024...
- 8210 Views
- 3 replies
- 0 kudos
- 2424 Views
- 1 replies
- 0 kudos
How to Create Azure Key Vault and Assign Key Vault Administrator Role Using Terraform
Hi all,I’m currently working with Terraform to set up Azure resources, including OpenAI services, and I’d like to extend my configuration to create an Azure Key Vault. Specifically, I want to:Create an Azure Key Vault to store secrets/keys.Assign the...
- 2424 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @naveen0142 , 1. Create the Key Vault resource "azurerm_key_vault" "example" { name = var.key_vault_name location = azurerm_resource_group.example.location resource_group_name = azurerm_resource_group.example....
- 0 kudos
- 735 Views
- 0 replies
- 0 kudos
Atualização Incremental e/ou Modelos Compostos (Databricks x Power BI)
Gostaria de deixar meu modelo mais performático no Power BI, mas tenho encontrado algumas dificuldades ao conectá-lo em uma fonte no DataBricks. Queria saber se é possÃvel fazer atualização incremenal e/ou trabalhar com modelos compostos (Direct Quer...
- 735 Views
- 0 replies
- 0 kudos
- 3763 Views
- 1 replies
- 0 kudos
User Unable to Access Key Vault Secrets Despite Role Assignment in Terraform
Hi All,I'm encountering an issue where a user is unable to access secrets in an Azure Key Vault, even though the user has been assigned the necessary roles using Terraform. Specifically, the user gets the following error when trying to access the sec...
- 3763 Views
- 1 replies
- 0 kudos
- 0 kudos
Are they accessing the Key Vault directly and not through Databricks? If so, based on your Terraform code, they should be able to directly read Secrets in the Azure Key Vault. You've configured the Key Vault with RBAC Authorization and assigned Key ...
- 0 kudos
