cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Abhay_1002
by New Contributor
  • 507 Views
  • 0 replies
  • 0 kudos

Archive file support in Jar Type application

In my spark application, I am using set of python libraries. I am submitting spark application as Jar Task. But I am not able to find any option provide Archive Files.So, in order to handle python dependencies, I am using approach:Create archive file...

  • 507 Views
  • 0 replies
  • 0 kudos
Hubcap7700
by New Contributor
  • 887 Views
  • 0 replies
  • 0 kudos

Native Slack Integration

Hi,Are there any plans to build native slack integration? I'm envisioning a one-time connector to Slack that would automatically populate all channels and users to select to use for example when configuring an alert notification. It is does not seem ...

  • 887 Views
  • 0 replies
  • 0 kudos
Abhay_1002
by New Contributor
  • 586 Views
  • 1 replies
  • 0 kudos

Issue with Python Package Management in Spark application

In a pyspark application, I am using set of python libraries. In order to handle python dependencies while running pyspark application, I am using the approach provided by spark : Create archive file of Python virtual environment using required set o...

  • 586 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi, I have not tried it but based on the doc you have to go by this approach. ./environment/bin/pythonmust be replaced with the correct path. import os from pyspark.sql import SparkSession os.environ['PYSPARK_PYTHON'] = "./environment/bin/python" sp...

  • 0 kudos
Nagarathna
by New Contributor II
  • 1878 Views
  • 3 replies
  • 1 kudos

File not found error when trying to read json file from aws s3 using with open.

I am trying to reading json from aws s3 using with open in databricks notebook using shared cluster.Error message:No such file or directory:'/dbfs/mnt/datalake/input_json_schema.json'In single instance cluster the above error is not found.  

  • 1878 Views
  • 3 replies
  • 1 kudos
Latest Reply
NandiniN
Databricks Employee
  • 1 kudos

Hi @Nagarathna , I just tried it on a shared cluster and did not face any issue. What is the exact error that you are facing? Complete stacktrace might help. Just to confirm are you accessing the "/dbfs/mnt/datalake/input.json" from the same workspac...

  • 1 kudos
2 More Replies
databricksdev
by New Contributor II
  • 939 Views
  • 2 replies
  • 0 kudos

Can we customize job run name when running azure data bricks notebook jobs from azure data factory

Hi All,we are executing databricks notebook activity  inside the child pipeline thru ADF. we are getting child pipeline name in job name while executing databricks job.  Is it possible to get master pipeline name as job name or customize job name thr...

  • 939 Views
  • 2 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

I think we should raise a Request/Product Feedback. Not sure if it would be Databricks that would own it or Microsoft but you may submit feedback for Databricks here - https://docs.databricks.com/en/resources/ideas.html  

  • 0 kudos
1 More Replies
MOUNIKASIMHADRI
by New Contributor
  • 4287 Views
  • 2 replies
  • 1 kudos

Insufficient Permissions Issue on Databricks

I have encountered a technical issue on Databricks.While executing commands both in Spark and SQL within the Databricks environment, I’ve run into permission-related errors from selecting files from DBFS. "org.apache.spark.SparkSecurityException: [IN...

  • 4287 Views
  • 2 replies
  • 1 kudos
Latest Reply
NandiniN
Databricks Employee
  • 1 kudos

Hi @MOUNIKASIMHADRI ,   Workspace admins get ANY FILE granted by default. They can explicitly grant it to non-admin users. Hence as suggested in the kb,  GRANT SELECT ON ANY FILE TO `<user@domain-name>`  

  • 1 kudos
1 More Replies
AlexG
by New Contributor II
  • 1825 Views
  • 3 replies
  • 1 kudos

Query results in csv file include 'null' string for blank cell

After running a sql script, when downloading the results to a csv file, the file includes a null string for blank cells (see screenshot). Is ther a setting I can change to simply get empty cells instead? 

AlexG_1-1702927614092.png
  • 1825 Views
  • 3 replies
  • 1 kudos
Latest Reply
NandiniN
Databricks Employee
  • 1 kudos

Hi AlexG, I tested with the table content containing null and with empty data and it works as expected in the download option too. Here is an eg: CREATE TABLE my_table_null_test1 ( id INT, name STRING ); INSERT INTO my_table_null_test1 (id, name)...

  • 1 kudos
2 More Replies
DataBricks_Use1
by New Contributor
  • 860 Views
  • 2 replies
  • 0 kudos

FileReadException Error

Hi,I am getting FilereadException Error while reading JSON file using REST API Connector.It comes when data is huge in Json File and it's not able to handle more than 1 Lac records.Error details:org.apache.spark.SparkException: Job aborted due to sta...

  • 860 Views
  • 2 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hello @DataBricks_Use1 , It would great if you could add the entire stack trace, as Jose mentioned. But there should be a "Caused by:" section below which would give you an idea of what's the reason for this failure and then you can work on that.  fo...

  • 0 kudos
1 More Replies
Phani1
by Valued Contributor II
  • 1201 Views
  • 1 replies
  • 0 kudos

temporary tables or dataframes,

We have to generate over 70 intermediate tables. Should we use temporary tables or dataframes, or should we create delta tables and truncate and reload? Having too many temporary tables could lead to memory problems. In this situation, what is the mo...

  • 1201 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi Phani1, It would be a use case specific answer, so if it is possible I would suggest to work with the Solution Architect on this or share some more insights for a better guidance. When I say that, I just would want to understand would we really ne...

  • 0 kudos
Ruby8376
by Valued Contributor
  • 918 Views
  • 2 replies
  • 1 kudos

Databricks sql API <- Delta tables

We have an integration flow where we want to expose databricks data for querying through odata(webapp). For this piecedatabricks sql API <- Delta tables :2 questions here:1. can you share link/documentation on how we can integrate databricks <-delta ...

  • 918 Views
  • 2 replies
  • 1 kudos
Latest Reply
shan_chandra
Databricks Employee
  • 1 kudos

Hi @Ruby8376 - can you please review the similar posts where the resolution is provided  https://community.databricks.com/t5/warehousing-analytics/databricks-sql-restful-api-to-query-delta-table/td-p/8617 https://www.databricks.com/blog/2023/03/07/da...

  • 1 kudos
1 More Replies
dvmentalmadess
by Valued Contributor
  • 28912 Views
  • 7 replies
  • 3 kudos

Resolved! Private PyPI repos on DBR 13+

We use a private PyPI repo (AWS CodeArtifact) to publish custom python libraries. We make the private repo available to DBR 12.2 clusters using an init-script as prescribed here in the Databricks KB.  When we tried to upgrade to 13.2 this stopped wor...

  • 28912 Views
  • 7 replies
  • 3 kudos
Latest Reply
dvmentalmadess
Valued Contributor
  • 3 kudos

I'm coming back to provide an updated solution that doesn't rely on the implementation detail of the user name (e.g., libraries) - which is not considered a contract and could potentially change and break in the future.The key is to use the --global ...

  • 3 kudos
6 More Replies
_raman_
by New Contributor II
  • 581 Views
  • 1 replies
  • 0 kudos

Internal Error : report

I'm getting tis error while running any cell in notebook. On the top middle it is coming like this. "Uncaught TypeError: Cannot redefine property: googletagReload the page and try again. If the error persists, contact support. Reference error code: 7...

  • 581 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi @_raman_ , Which DBR are you facing this issue? Most likely the issue is related to this: https://github.com/shadcn-ui/ui/issues/2837 If you are having this issue might be because of some browserextension. A quick test to confirm this theory is to...

  • 0 kudos
amit_jbs
by New Contributor II
  • 1247 Views
  • 2 replies
  • 1 kudos

In databricks deployment .py files getting converted to notebooks

A critical issue has arisen that is impacting our deployment planning for our client. We have encountered a challenge with our Azure CI/CD pipeline integration, specifically concerning the deployment of Python files (.py). Despite our best efforts, w...

  • 1247 Views
  • 2 replies
  • 1 kudos
Latest Reply
Dazza
New Contributor II
  • 1 kudos

Experiencing a similar issue that we are looking to resolve except the files are .sql. We have a process that has 1 orchestration notebook , calling multiple .sql files. These .sql files are being converted to regular databricks notebooks when deploy...

  • 1 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors