Hi Team, I’m trying to load data from Databricks into Snowflake using the Snowflake Spark connector. I’m using a generic username and password, but I’m unable to log in using these credentials directly. In the Snowflake UI, I can only log in through ...
As part of our Azure Data Factory pipeline, we utilize Databricks to run some scripts that identify which files we need to load from a certain source. This list of files is then passed back into Azure Data Factory utilizing the Exit status from the n...
Hello, as of today, is it still not possible to unredact secret names?
When I am login to the course it shows me this error.You are not authorized to access https://customer-academy.databricks.com. Please select a platform you can access from the following list.
Hello @osamR! If you’re encountering this error while trying to log in to Customer Academy, it may indicate that your account is registered under the Partner Academy. You can try accessing your courses through the Partner Academy.However, if your org...
Good Day all,After having issues with Cloud resources allocated to Lakeflow jobs and Gateways I am trying to apply a policy to the cluster that is allocated to the Job. I am very new to a lot of the databricks platform and the administration so all h...
I am trying to run a SQL alert task in data bricks job, Service principal is the run_as as set to the job, and task fails with below error message. I also checked if SPN can be granted permission on SQL alert and SQL query, turns out only user id or...
unfortunately, only these options are available in my UI, is it the new UI?
Hi, I have very interesting scenario. I am trying to use Declarative pipelines for first time. The platform team has made workspace artefacts as devops based deployment [infra as code], meaning, I cannot create compute. I have to create compute with ...
@crami wrote:Hi, I have very interesting scenario. I am trying to use Declarative pipelines for first time. The platform team has made workspace artefacts as devops based deployment [infra as code], meaning, I cannot create compute. I have to create ...
Hi everyone,I’m facing an issue when executing a Databricks job where my notebook uses %run to include other notebooks. I have a final notebook added as a task in a job, and inside that notebook I use %run to call another notebook that contains all ...
Hi guys I've added a parameter "files_mask " to a notebook, with a default value.The job running this notebook broke with error: com.databricks.dbutils_v1.InputWidgetNotDefined: No input widget named files_mask is definedCode: mask = dbutils.widgets....
Hi, When a Job is running, I would like to change the parameters with an API call.I know that I can set parameters value from API when I start a job from API, or that I can update the default value if the job isn't running, but I didn't find an API c...
Hi, Alessandro, I am trying to set job parameters value from API when I start a job from API call within python notebook, however, it has never succeeded, the job can be triggered, but always with job parameters' default values instead values from AP...
I have a stored procedure that is saved as a query file. I can run it and the proc is created. However I want to take this one step further. I want my notebook to run the query file called sp_Remit.sql so if there is any changes to the proc between t...
Something like this?import os query_name = "test_query.sql" query_path = os.path.abspath(query_name) # Read query contents with open(query_path, "r") as f: query_str = f.read() # Run it spark.sql(query_str)You can read the script from the sql ...
Hi,In the course Introduction to Apache Spark; according to Apache Spark Runtime Architecture; Page 6 of 15. It says that :The cluster manager allocates resources and assigns tasks......Workers perform tasks assigned by the driverCan you help me plea...
Hi @fundat Perhaps the picture is useful here:Give this blog a read, I think this will answer some of your questions: https://medium.com/@knoldus/understanding-the-working-of-spark-driver-and-executor-4fec0e669399 .All the best,BS
We have a use case where Job C should start only after both Job A and Job B have successfully completed.In Airflow, we achieve this using an ExternalTaskSensor to set dependencies across different DAGs.Is there a way to configure something similar in...
Hey @dhruvs2 You could use Lakeflow Jobs for this. You can add a job as a task:Then you can just follow the docs from here: https://docs.databricks.com/aws/en/jobs/ there's loads of great sections / tutorials.To answer your specific question:When con...
Hi all,I’m working with Delta Live Tables (DLT) and want to enable Liquid Clustering on an existing DLT table that was already created without it.Could someone please clarify:How can I enable Liquid Clustering on an existing DLT table (without recre...
Hi @Sakthi0311 ,For SQL language you can enable LC for materialized views and streaming tables. So the syntax looks following:If you want to use automatic clustering then use CLUSTER BY AUTO.
Hi!I am trying to replicate an AWS RDS PostgreSQL database in Databricks. I have successfully manage to enable CDC using AWS DMS that writes an initial load file and continuous CDC files in parquet.I have been trying to follow the official guide Repl...
hey @excavator-matt Let's remember that the Bronze layer is for mere raw ingestion; this provides a baseline for auditing and to start applying transformations based on the different use cases you need to serve. Systems and their requirements change...
Good Day all,I am having an issue with our first Data Ingestion Pipelines, I am wanting to connect to our Azure SQL Server with our Unity Connector (I can access the data in Unity Catalog). When I am on Step 3 of the process (Source) when it is scann...
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 477 | |
| 349 | |
| 287 |
