by
MYB24
• New Contributor III
- 3564 Views
- 4 replies
- 0 kudos
Good Evening, I am configuring databricks_mws_credentials through Terraform on AWS. I am getting the following error:Error: cannot create mws credentials: invalid Databricks Account configuration││ with module.databricks.databricks_mws_credentials.t...
- 3564 Views
- 4 replies
- 0 kudos
Latest Reply
MYB24
New Contributor III
Managed to fix the issue by updating the provider.tf while. Had to create a Service Principle token and add that into my provider.tf file. provider "databricks" {alias = "accounts"host = "https://accounts.cloud.databricks.com"client_id = "service-pri...
3 More Replies
- 2649 Views
- 4 replies
- 3 kudos
My earlier question was about creating a Databricks Asset Bundle (DAB) from an existing workspace. I was able to get that working but after further consideration and some experimenting, I need to alter my question. My question is now "how do I create...
- 2649 Views
- 4 replies
- 3 kudos
Latest Reply
This is how I solved this:. Hope this works for you.- bundle:- jobs:
3 More Replies
by
ac0
• New Contributor III
- 1297 Views
- 4 replies
- 0 kudos
I'm trying to use the Global Init Scripts in Databricks to set an environment variable to use in a Delta Live Table Pipeline. I want to be able to reference a value passed in as a path versus hard coding it. Here is the code for my pipeline:CREATE ST...
- 1297 Views
- 4 replies
- 0 kudos
Latest Reply
I was able to accomplish this by creating a Cluster Policy that put in place the scripts, config settings, and environment variables I needed.
3 More Replies
by
ChrisS
• New Contributor III
- 1981 Views
- 7 replies
- 8 kudos
I learning data bricks for the first time following the book that is copywrited in 2020 so I imagine it might be a little outdated at this point. What I am trying to do is move data from an online source (in this specific case using shell script but ...
- 1981 Views
- 7 replies
- 8 kudos
Latest Reply
In Databricks, you can install external libraries by going to the Clusters tab, selecting your cluster, and then adding the Maven coordinates for Deequ. This represents the best b2b data enrichment services in Databricks.In your notebook or script, y...
6 More Replies
- 664 Views
- 3 replies
- 0 kudos
Hey all, my team has settled on using directory-scoped SAS tokens to provision access to data in our Azure Gen2 Datalakes. However, we have encountered an issue when switching from a first SAS token (which is used to read a first parquet table in the...
- 664 Views
- 3 replies
- 0 kudos
Latest Reply
Hi @aockenden, The data in the Data Lake is not actually retrieved into cluster memory by the Spark dataframes until an action (like .show()) is executed. At this point, the fs.azure.sas.fixed.token Spark configuration setting has been switched to a ...
2 More Replies
by
JKR
• New Contributor III
- 928 Views
- 1 replies
- 0 kudos
I have 2 tasks in databricks job workflow first task is of type SQL and SQL task is query.In that query I've declared 2 variables and SET the values by running query.e.g:DECLARE VARIABLE max_timestamp TIMESTAMP DEFAULT '1970-01-01';
SET VARIABLE max_...
- 928 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @JKR, You can indeed pass variables between tasks in a Databricks job workflow. This is done using the taskValues subutility in Databricks Utilities. This utility allows tasks to output values that can be referenced in subsequent tasks.
However,...
- 595 Views
- 2 replies
- 0 kudos
I created simplistic DLT pipeline that create one table. When I delete the pipeline the tables is dropped as well. That's not really desired behavior. Since I remember there was a strong distinction between data (stored in tables) and processing (spa...
- 595 Views
- 2 replies
- 0 kudos
Latest Reply
Hi @BobEng, Here are a few things that might help:
Pipeline Settings: Delta Live Tables provides a user interface for configuring and editing pipeline settings. You can configure most settings with either the UI or a JSON specification.
Table Mana...
1 More Replies
- 905 Views
- 4 replies
- 0 kudos
from pyspark.sql import functions as F
from pyspark.sql import types as T
from pyspark.sql import DataFrame, Column
from pyspark.sql.types import Row
import dlt
S3_PATH = 's3://datalake-lab/xxxx/'
S3_SCHEMA = 's3://datalake-lab/xxxx/schemas/'
@dl...
- 905 Views
- 4 replies
- 0 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
3 More Replies
by
pyter
• New Contributor III
- 4059 Views
- 6 replies
- 2 kudos
Hello everyone,We use unity catalog, separating our dev, test and prod data into individual catalogs.We run weekly vacuums on our prod catalog using a service principal that only has (read+write) access to this production catalog, but no access to ou...
- 4059 Views
- 6 replies
- 2 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
5 More Replies
- 364 Views
- 2 replies
- 0 kudos
I am implementing Structured Streaming using Delta Live Table. I want to delete the parquet files once they are used. What options should I set so that the files loaded in S3 are not deleted?
- 364 Views
- 2 replies
- 0 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
- 554 Views
- 4 replies
- 1 kudos
hello everyone, how to collect metrics provided by clusters metrics databricks runtime 13.1 using bash script
- 554 Views
- 4 replies
- 1 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
3 More Replies
- 433 Views
- 1 replies
- 1 kudos
Hi Databricks community,Reverso is a language translation tool that converts texts from one language to another. I need to convert hundreds of text but its time consuming. Hence, I want to use its API to automate the process.How can I achieve this in...
- 433 Views
- 1 replies
- 1 kudos
Latest Reply
Hi @chari, To use the Reverso API inside Databricks notebooks, you can use the requests library in Python.
- 321 Views
- 2 replies
- 0 kudos
I have a number of functions in a schema in a catalog in Unity Catalog, is there a coding way to be able to change the owner of the functions created without having to do it manually via the gui?
- 321 Views
- 2 replies
- 0 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
1 More Replies
- 1208 Views
- 7 replies
- 0 kudos
Hi Team,My requirement is to move build a solution to move zos(db2) CDC data to Delta table on Realtime bases(at least near realtime) , data volume and number of tables are little huge (100 tables) I have researched I dont find any inbuild options in...
- 1208 Views
- 7 replies
- 0 kudos
Latest Reply
Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...
6 More Replies
