- 663 Views
- 1 replies
- 0 kudos
Native Slack Integration
Hi,Are there any plans to build native slack integration? I'm envisioning a one-time connector to Slack that would automatically populate all channels and users to select to use for example when configuring an alert notification. It is does not seem ...
- 663 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Hubcap7700, If you have any further details or specific requirements, feel free to share, and I’ll be happy to assist!
- 0 kudos
- 1381 Views
- 1 replies
- 0 kudos
requirements.txt with cluster libraries
Cluster libraries are supported from version 15.0 - Databricks Runtime 15.0 | Databricks on AWS.How can I specify requirements.txt file path in the libraries in a job cluster in my workflow? Can I use relative path? Is it relative from the root of th...
- 1381 Views
- 1 replies
- 0 kudos
- 0 kudos
To specify the requirements.txt file path for libraries in a job cluster workflow in Databricks, you have a few options. Let’s break it down: Upload the requirements.txt File: First, upload your requirements.txt file to your Databricks workspace....
- 0 kudos
- 396 Views
- 1 replies
- 0 kudos
Archive file support in Jar Type application
In my spark application, I am using set of python libraries. I am submitting spark application as Jar Task. But I am not able to find any option provide Archive Files.So, in order to handle python dependencies, I am using approach:Create archive file...
- 396 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Abhay_1002, Using --py-files Argument: When submitting a Spark application, you can use the --py-files argument to add Python files (including .zip or .egg archives) to be distributed with your application1. However, this approach is typical...
- 0 kudos
- 478 Views
- 1 replies
- 0 kudos
Databricks bundles - good practice for multiprocessing envs
I'm seeking advice regarding Databricks bundles. In my scenario, I have multiple production environments where I aim to execute the same DLT. To simplify, let's assume the DLT reads data from 'eventhub-region-name,' with this being the only differing...
- 478 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mderela, When dealing with Databricks bundles in a multi-environment setup, there are some best practices you can follow to ensure smooth execution and maintainable code. Let’s explore a couple of recommendations: Parameterization and Configu...
- 0 kudos
- 447 Views
- 1 replies
- 0 kudos
Issue with Python Package Management in Spark application
In a pyspark application, I am using set of python libraries. In order to handle python dependencies while running pyspark application, I am using the approach provided by spark : Create archive file of Python virtual environment using required set o...
- 447 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I have not tried it but based on the doc you have to go by this approach. ./environment/bin/pythonmust be replaced with the correct path. import os from pyspark.sql import SparkSession os.environ['PYSPARK_PYTHON'] = "./environment/bin/python" sp...
- 0 kudos
- 1388 Views
- 3 replies
- 1 kudos
File not found error when trying to read json file from aws s3 using with open.
I am trying to reading json from aws s3 using with open in databricks notebook using shared cluster.Error message:No such file or directory:'/dbfs/mnt/datalake/input_json_schema.json'In single instance cluster the above error is not found.
- 1388 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Nagarathna , I just tried it on a shared cluster and did not face any issue. What is the exact error that you are facing? Complete stacktrace might help. Just to confirm are you accessing the "/dbfs/mnt/datalake/input.json" from the same workspac...
- 1 kudos
- 751 Views
- 2 replies
- 0 kudos
Can we customize job run name when running azure data bricks notebook jobs from azure data factory
Hi All,we are executing databricks notebook activity inside the child pipeline thru ADF. we are getting child pipeline name in job name while executing databricks job. Is it possible to get master pipeline name as job name or customize job name thr...
- 751 Views
- 2 replies
- 0 kudos
- 0 kudos
I think we should raise a Request/Product Feedback. Not sure if it would be Databricks that would own it or Microsoft but you may submit feedback for Databricks here - https://docs.databricks.com/en/resources/ideas.html
- 0 kudos
- 3345 Views
- 2 replies
- 1 kudos
Insufficient Permissions Issue on Databricks
I have encountered a technical issue on Databricks.While executing commands both in Spark and SQL within the Databricks environment, I’ve run into permission-related errors from selecting files from DBFS. "org.apache.spark.SparkSecurityException: [IN...
- 3345 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @MOUNIKASIMHADRI , Workspace admins get ANY FILE granted by default. They can explicitly grant it to non-admin users. Hence as suggested in the kb, GRANT SELECT ON ANY FILE TO `<user@domain-name>`
- 1 kudos
- 1032 Views
- 2 replies
- 0 kudos
Impersonating a user
How do I impersonate a user? I can't find any documentation that explains how to do this or even hint that it's possible.Use case: I perform administrative tasks like assign grants and roles to catalogs, schemas, and tables for the benefit of busines...
- 1032 Views
- 2 replies
- 0 kudos
- 0 kudos
Hidbx_687_3__1b3Q, Actually, I have seen impersonation, is this something that you are looking for? https://docs.gcp.databricks.com/en/dev-tools/google-id-auth.html#step-5-impersonate-the-google-cloud-service-account
- 0 kudos
- 1467 Views
- 3 replies
- 1 kudos
Query results in csv file include 'null' string for blank cell
After running a sql script, when downloading the results to a csv file, the file includes a null string for blank cells (see screenshot). Is ther a setting I can change to simply get empty cells instead?
- 1467 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi AlexG, I tested with the table content containing null and with empty data and it works as expected in the download option too. Here is an eg: CREATE TABLE my_table_null_test1 ( id INT, name STRING ); INSERT INTO my_table_null_test1 (id, name)...
- 1 kudos
- 675 Views
- 2 replies
- 0 kudos
FileReadException Error
Hi,I am getting FilereadException Error while reading JSON file using REST API Connector.It comes when data is huge in Json File and it's not able to handle more than 1 Lac records.Error details:org.apache.spark.SparkException: Job aborted due to sta...
- 675 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @DataBricks_Use1 , It would great if you could add the entire stack trace, as Jose mentioned. But there should be a "Caused by:" section below which would give you an idea of what's the reason for this failure and then you can work on that. fo...
- 0 kudos
- 644 Views
- 1 replies
- 0 kudos
temporary tables or dataframes,
We have to generate over 70 intermediate tables. Should we use temporary tables or dataframes, or should we create delta tables and truncate and reload? Having too many temporary tables could lead to memory problems. In this situation, what is the mo...
- 644 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Phani1, It would be a use case specific answer, so if it is possible I would suggest to work with the Solution Architect on this or share some more insights for a better guidance. When I say that, I just would want to understand would we really ne...
- 0 kudos
- 452 Views
- 1 replies
- 0 kudos
Databrick Dashboard state not cleared when login as other user.
Hi all, I am using Databricks and created a notebook and would like to run in Dashboard. It works correctly. I share the Dashboard with another user UserA with "Can Run" permission When I login as a UserA and login and accesses Dashboard then does a...
- 452 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Koa, You’ve encountered a security concern related to Databricks and handling JWT tokens within notebooks. Dashboard State Persistence: When you share a dashboard with another user (in this case, UserA), any updates made by that user will re...
- 0 kudos
- 699 Views
- 2 replies
- 1 kudos
Databricks sql API <- Delta tables
We have an integration flow where we want to expose databricks data for querying through odata(webapp). For this piecedatabricks sql API <- Delta tables :2 questions here:1. can you share link/documentation on how we can integrate databricks <-delta ...
- 699 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Ruby8376 - can you please review the similar posts where the resolution is provided https://community.databricks.com/t5/warehousing-analytics/databricks-sql-restful-api-to-query-delta-table/td-p/8617 https://www.databricks.com/blog/2023/03/07/da...
- 1 kudos
- 644 Views
- 1 replies
- 0 kudos
Can we share Delta table data with Salesforce using OData?
Hello!I'm seeking recommendations for streaming on-demand data from Databricks Delta tables to Salesforce. Is OData a viable choice?Thanks.
- 644 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ChristopherQ1, OData (Open Data Protocol) is a standard for building and consuming RESTful APIs. It provides a consistent way to expose and consume data over the web.While OData can be used for data integration, it’s essential to evaluate whethe...
- 0 kudos
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group-
AI Summit
4 -
Azure
3 -
Azure databricks
3 -
Bi
1 -
Certification
1 -
Certification Voucher
2 -
Chatgpt
1 -
Community
7 -
Community Edition
3 -
Community Members
2 -
Community Social
1 -
Contest
1 -
Data + AI Summit
1 -
Data Engineering
1 -
Data Processing
1 -
Databricks Certification
1 -
Databricks Cluster
1 -
Databricks Community
11 -
Databricks community edition
3 -
Databricks Community Rewards Store
3 -
Databricks Lakehouse Platform
5 -
Databricks notebook
1 -
Databricks Office Hours
1 -
Databricks Runtime
1 -
Databricks SQL
4 -
Databricks-connect
1 -
DBFS
1 -
Dear Community
3 -
Delta
10 -
Delta Live Tables
1 -
Documentation
1 -
Exam
1 -
Featured Member Interview
1 -
HIPAA
1 -
Integration
1 -
LLM
1 -
Machine Learning
1 -
Notebook
1 -
Onboarding Trainings
1 -
Python
2 -
Rest API
11 -
Rewards Store
2 -
Serverless
1 -
Social Group
1 -
Spark
1 -
SQL
8 -
Summit22
1 -
Summit23
5 -
Training
1 -
Unity Catalog
4 -
Version
1 -
VOUCHER
1 -
WAVICLE
1 -
Weekly Release Notes
2 -
weeklyreleasenotesrecap
2 -
Workspace
1
- « Previous
- Next »