- 13931 Views
- 5 replies
- 1 kudos
Job parameters to get date and time
I'm trying to set up a workflow in databricks and I need my job parameter to get the date and time. I see in the documentation there's some options for dynamic values.I'm trying to use this one: {{job.start_time.[argument]}}For the "argument" there, ...
- 13931 Views
- 5 replies
- 1 kudos
- 1 kudos
Then please change the code to:```iso_datetime = dbutils.widgets.get("LoadID")```
- 1 kudos
- 1132 Views
- 0 replies
- 0 kudos
Archive file support in Jar Type application
In my spark application, I am using set of python libraries. I am submitting spark application as Jar Task. But I am not able to find any option provide Archive Files.So, in order to handle python dependencies, I am using approach:Create archive file...
- 1132 Views
- 0 replies
- 0 kudos
- 1747 Views
- 0 replies
- 0 kudos
Getting databricks-connect com.fasterxml.jackson.databind.exc.MismatchedInputException parse warning
Hi community, I am getting below warning when I try using pyspark code for some of my use-cases using databricks-connect. Is this a critical warning, and any idea what does it mean?Logs: WARN DatabricksConnectConf: Could not parse /root/.databricks-c...
- 1747 Views
- 0 replies
- 0 kudos
- 13691 Views
- 1 replies
- 0 kudos
Getting python version errors when using pyspark rdd using databricks connect
Hi community, When I use pyspark rdd related functions in my environment using databricks connect, I get below error: Databricks cluster version: 12.2. `RuntimeError: Python in worker has different version 3.9 than that in driver 3.10, PySpark cannot...
- 13691 Views
- 1 replies
- 0 kudos
- 0 kudos
Got it. As a side note, I tried above methods, but the error persisted, hence upon reading docs again, there was this statement: You must install Python 3 on your development machine, and the minor version of your client Python installation must be t...
- 0 kudos
- 2957 Views
- 0 replies
- 1 kudos
Native Slack Integration
Hi,Are there any plans to build native slack integration? I'm envisioning a one-time connector to Slack that would automatically populate all channels and users to select to use for example when configuring an alert notification. It is does not seem ...
- 2957 Views
- 0 replies
- 1 kudos
- 3061 Views
- 0 replies
- 1 kudos
connection from databricks to snowflake using OKTA
Hi team,This is how I connect to Snowflake from Jupyter Notebook:import snowflake.connector snowflake_connection = snowflake.connector.connect( authenticator='externalbrowser', user='U1', account='company1.us-east-1', database='db1',...
- 3061 Views
- 0 replies
- 1 kudos
- 6244 Views
- 2 replies
- 1 kudos
Workflows: Running dependent task despite earlier task fail
I have a scheduled task running in workflow.Task 1 computes some parameters then these are picked up by a dependent reporting task: Task 2.I want Task 2 to report "Failure" if Task 1 fails. Yet creating a dependency in workflows means that Task 2 wil...
- 6244 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @sharpbetty , Any suggestions how I can keep the parameter sharing and dependency from Task 1 to Task 2, yet also allow Task 2 to fire even on failure of Task 1?Setup:Task 2 dependent on Task1 Challenge: To Fire Task 2 even on Task 1 FailureSoluti...
- 1 kudos
- 3487 Views
- 1 replies
- 0 kudos
Issue with Python Package Management in Spark application
In a pyspark application, I am using set of python libraries. In order to handle python dependencies while running pyspark application, I am using the approach provided by spark : Create archive file of Python virtual environment using required set o...
- 3487 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I have not tried it but based on the doc you have to go by this approach. ./environment/bin/pythonmust be replaced with the correct path. import os from pyspark.sql import SparkSession os.environ['PYSPARK_PYTHON'] = "./environment/bin/python" sp...
- 0 kudos
- 3966 Views
- 3 replies
- 1 kudos
File not found error when trying to read json file from aws s3 using with open.
I am trying to reading json from aws s3 using with open in databricks notebook using shared cluster.Error message:No such file or directory:'/dbfs/mnt/datalake/input_json_schema.json'In single instance cluster the above error is not found.
- 3966 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Nagarathna , I just tried it on a shared cluster and did not face any issue. What is the exact error that you are facing? Complete stacktrace might help. Just to confirm are you accessing the "/dbfs/mnt/datalake/input.json" from the same workspac...
- 1 kudos
- 2404 Views
- 2 replies
- 0 kudos
Can we customize job run name when running azure data bricks notebook jobs from azure data factory
Hi All,we are executing databricks notebook activity inside the child pipeline thru ADF. we are getting child pipeline name in job name while executing databricks job. Is it possible to get master pipeline name as job name or customize job name thr...
- 2404 Views
- 2 replies
- 0 kudos
- 0 kudos
I think we should raise a Request/Product Feedback. Not sure if it would be Databricks that would own it or Microsoft but you may submit feedback for Databricks here - https://docs.databricks.com/en/resources/ideas.html
- 0 kudos
- 2920 Views
- 2 replies
- 0 kudos
FileReadException Error
Hi,I am getting FilereadException Error while reading JSON file using REST API Connector.It comes when data is huge in Json File and it's not able to handle more than 1 Lac records.Error details:org.apache.spark.SparkException: Job aborted due to sta...
- 2920 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @DataBricks_Use1 , It would great if you could add the entire stack trace, as Jose mentioned. But there should be a "Caused by:" section below which would give you an idea of what's the reason for this failure and then you can work on that. fo...
- 0 kudos
- 3834 Views
- 1 replies
- 0 kudos
temporary tables or dataframes,
We have to generate over 70 intermediate tables. Should we use temporary tables or dataframes, or should we create delta tables and truncate and reload? Having too many temporary tables could lead to memory problems. In this situation, what is the mo...
- 3834 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Phani1, It would be a use case specific answer, so if it is possible I would suggest to work with the Solution Architect on this or share some more insights for a better guidance. When I say that, I just would want to understand would we really ne...
- 0 kudos
- 1215 Views
- 0 replies
- 0 kudos
Databricks bundles - good practice for multiprocessing envs
I'm seeking advice regarding Databricks bundles. In my scenario, I have multiple production environments where I aim to execute the same DLT. To simplify, let's assume the DLT reads data from 'eventhub-region-name,' with this being the only differing...
- 1215 Views
- 0 replies
- 0 kudos
- 4598 Views
- 1 replies
- 1 kudos
Databricks Users Access Control via Azure AAD?
Hi All,Looking for suggestions to see if it is possible to control users via Azure AD (outside of Azure Databricks). As i want to create a new users in Azure and then I want to give RBAC to individual users and rather than control their permissions f...
- 4598 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you Kaniz, let me try some of the options as my Databricks is integrated with AAD. Let me try Option 1 as thats my primary requirement.
- 1 kudos
- 2541 Views
- 2 replies
- 1 kudos
Databricks sql API <- Delta tables
We have an integration flow where we want to expose databricks data for querying through odata(webapp). For this piecedatabricks sql API <- Delta tables :2 questions here:1. can you share link/documentation on how we can integrate databricks <-delta ...
- 2541 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Ruby8376 - can you please review the similar posts where the resolution is provided https://community.databricks.com/t5/warehousing-analytics/databricks-sql-restful-api-to-query-delta-table/td-p/8617 https://www.databricks.com/blog/2023/03/07/da...
- 1 kudos
