- 2451 Views
- 2 replies
- 0 kudos
Not able to Run jobs using M2M authentication form our code
Hi,I am using OAuth machine-to-machine (M2M) authentication with the JDBC approach.String url = "jdbc:databricks://<server-hostname>:443";Properties p = new java.util.Properties();p.put("httpPath", "<http-path>");p.put("AuthMech", "11");p.put("Auth_F...
- 2451 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Balram-snaplogi ,It looks like permission problem. Could you check if the service principal has the necessary permissions to execute jobs?In Databricks, permissions for jobs can be managed to control access. The following permissions are availabl...
- 0 kudos
- 3219 Views
- 2 replies
- 0 kudos
Resolved! Delta Live Table "Default Schema" mandatory but not editable
Hi,We have an issue with a DLT pipeline. We want to add some source code to an existing pipeline. However, when we save, error message shows that "Default schema" is a mandatory field. However, we are not able to edit the field. The DLT pipeline does...
- 3219 Views
- 2 replies
- 0 kudos
- 0 kudos
Does the pipeline settings JSON includes the "schema" field. If you have full admin rights, you can update the existing pipeline settings to include the "schema" field. Like curl -X PATCH https://<databricks-instance>/api/2.0/pipelines/<pipeline-id>...
- 0 kudos
- 1519 Views
- 2 replies
- 0 kudos
Unable to Read Collection/Files from MongoDB using Azure Databricks
Hi there,Can someone help to read data from MongoDB using Azure Databricks? Surprisingly, I am able to connect from Jupyter Notebook and read data, but not from the Azure Databricks.1) I have install the required spark-connector packages in the clust...
- 1519 Views
- 2 replies
- 0 kudos
- 0 kudos
I am using following code to read the data from mongoDB using Databricksfrom pyspark.sql import SparkSessionspark = SparkSession \.builder \.appName("myApp") \.config("spark.mongodb.connection.uri", "mongodb+srv://username:password@cluster.xxxx.mongo...
- 0 kudos
- 1375 Views
- 2 replies
- 1 kudos
Issue with Generic DLT Pipeline Handling Multiple BUs
We are implementing a data ingestion framework where data flows from a foreign catalog (source) to a raw layer (Delta tables) and then to a bronze layer (DLT streaming tables). Currently, each Business Unit (BU) has a separate workflow and DLT pipeli...
- 1375 Views
- 2 replies
- 1 kudos
- 1 kudos
You can create separate schemas within the same catalog for each BU. For example, you can have schemas like BU1_schema, BU2_schema, etc., within the same catalog. By using Unity Catalog, you can segregate BU-specific tables within the same DLT pipeli...
- 1 kudos
- 8606 Views
- 8 replies
- 0 kudos
Driver Crash on processing large dataframe
I have a dataframe with abt 2 million text rows (1gb). I partition it into about 700 parititons as thats the no of cores available on my cluster exceutors. I run the transformations extracting medical information and then write the results in parquet...
- 8606 Views
- 8 replies
- 0 kudos
- 0 kudos
Hey @Svish ,Your problem is probably caused by using Pandas. Pandas loads all the data into the driver memory, which is likely why you are experiencing issues. If you can modify your code to use Spark instead, you will probably avoid this problem.How...
- 0 kudos
- 1140 Views
- 2 replies
- 0 kudos
getting "NoSuchMethodError" while using tsl 15.4 and spark 3.5
hi, i am using data bricks version 15.4 and spark 3.5 and getting "NoSuchMethodError" and all the resources i found only solution is to downgrade spark and data bricks version. is there any solution apart from this as i cant do this in my case. regar...
- 1140 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @developer321, Are you upgrading any of the default libraries of DBR 15.4 LTS? please share more details on your use-case and commands / settings being used.
- 0 kudos
- 2063 Views
- 1 replies
- 0 kudos
Job Stuck with single user access mode
Heya So I'm working on a new workflow. I've started by writing a notebook and running it on an interactive cluster with "Single User" access mode, and everything worked fine.I created a workflow for this task with the same interactive cluster, and ev...
- 2063 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey!You cannot access an Instance Profile (IAM Role) in “Shared” mode, so discard this option if your job relies on AWS credentials via an instance profile. If your workflow depends on accessing S3 or other AWS resources using an IAM Role, you must u...
- 0 kudos
- 3385 Views
- 4 replies
- 1 kudos
Linux ODBC driver Unknown error
Hi,I'm trying to debug an issue connecting to Azure Databricks from a CentOS 7 machine. Testing on my own machine only required unixODBC, the databricks-odbc driver and the connection string which all worked fine. When I test from the CentOS 7 machin...
- 3385 Views
- 4 replies
- 1 kudos
- 1 kudos
@TomBrick Can you use the latest ODBC driver. Below is the link to download it. https://www.databricks.com/spark/odbc-drivers-download Also, can you add the parameters below in the simba.sparkodbc.ini file? This will give you more information about t...
- 1 kudos
- 963 Views
- 1 replies
- 0 kudos
Streamsets to Databricks Integration failure
Hi Team,While trying to ingest data from ADLS gen 2 to Databricks through Streamsets. I am getting below errorConfiguration fs.azure.account.key.xxx.dfs.core.windows.net is not available.Able to make connection from Streamsets to databricks using sql...
- 963 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Subbu_G, Ensure that the Spark configuration for the ADLS Gen 2 account key is correctly set. The configuration should follow the format: spark.hadoop.fs.azure.account.key.<storage-account-name>.dfs.core.windows.net <your-storage-account-key>...
- 0 kudos
- 1715 Views
- 1 replies
- 0 kudos
Error handling - SQL states
Dear all,Few questions please - 1. Has anyone successfully used the below way of dealing with error handling in PySpark (example: that contains data frames) as well as SQL code based notebooks - from pyspark.errors import PySparkException try: spa...
- 1715 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @noorbasha534, The approach you mentioned for error handling in PySpark using PySparkException is a valid method. It allows you to catch specific exceptions related to PySpark operations and handle them accordingly. Logging errors into tables ...
- 0 kudos
- 2790 Views
- 1 replies
- 0 kudos
Distinguishing stream workload from batch work load
Is it possible the same data source of batch data as well as stream data. Please find the following code that I have got from internet. The following code handles both stream and batch workload. Please find attached the corresponding pdf file. I am f...
- 2790 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @subhas_hati, Thanks for your question: Batch Workload: The availableNow trigger is used for batch processing. When you set the trigger to availableNow, it processes all available data as a single batch and then stops. This is useful for scenarios...
- 0 kudos
- 2375 Views
- 1 replies
- 0 kudos
Best Strategy for Ingesting PostgreSQL Data into Bronze Layer in Databricks
I am designing a data ingestion strategy for ingesting 10 tables from a PostgreSQL 10 database into the Bronze layer using Databricks only (without ADF or other external tools).Full Load: 7 tables will be fully loaded in each run.Incremental Load: 3 ...
- 2375 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ijaza0489, Here are key points to keep in mind: Tracking and Implementing Incremental Loads:Delta Lake: Utilize Delta Lake for managing incremental loads. Delta Lake supports ACID transactions and allows you to perform upserts and merges eff...
- 0 kudos
- 1698 Views
- 1 replies
- 0 kudos
Limit access to certain pages of a dashboard
Hello,I would like to know if it is possible to restrict / limit access to certain pages of a multipage dashboard based on the user's group membership. In other words, the dashboard itself is able to be accessed by all, with some pages visible to all...
- 1698 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mgallagher, Databricks does not natively support page-level access control within a single dashboard, you can create separate dashboards for different user groups and control access at the dashboard level. This means creating a main dashboard acc...
- 0 kudos
- 31315 Views
- 10 replies
- 5 kudos
Retrieve job-level parameters in Python
Parameters can be passed to Tasks and the values can be retrieved with:dbutils.widgets.get("parameter_name")More recently, we have been given the ability to add parameters to Jobs.However, the parameters cannot be retrieved like Task parameters.Quest...
- 31315 Views
- 10 replies
- 5 kudos
- 5 kudos
The only thing that has worked for me consistently in python is params = dbutils.widgets.getAll() where an empty dictionary is returned if I'm in interactive mode and the job/task params are returned if they are present.
- 5 kudos
- 2205 Views
- 2 replies
- 0 kudos
Cant remove file on ADLS using dbutils.fs.rm because url contains illeagal character
The URL contains a "[" within, and I've tried to encode the path from "[" to "%5B%27", but it didn't work: from urllib.parse import quotepath = ""encoded_path = quote(path)
- 2205 Views
- 2 replies
- 0 kudos
- 0 kudos
Try https://community.databricks.com/t5/data-engineering/how-can-i-delete-a-file-in-dbfs-with-illegal-character/td-p/9755 https://community.databricks.com/t5/data-engineering/using-dbutils-fs-ls-on-uri-with-square-brackets-results-in-error/td-p/6928
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
