cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

oleprince
by New Contributor
  • 5206 Views
  • 1 replies
  • 0 kudos

Delta table definition - Identity column

Hello,Would anyone know if it is possible to create a delta table using Python that includes a column that is generated by default as identity (identity column for which the value inserted can be manually overriden)?There seems to be a way to create ...

  • 5206 Views
  • 1 replies
  • 0 kudos
Latest Reply
gmiguel
Contributor
  • 0 kudos

Hi @oleprince ,As far as I know, it's not possible yet to create tables with Identity columns using pyspark (DeltaTable api). You can create generated columns, but Identity columns are not allowed.The only way to achieve this is through Spark Sql.  

  • 0 kudos
fiverrpromotion
by New Contributor
  • 502 Views
  • 1 replies
  • 0 kudos

Addressing Memory Constraints in Scaling XGBoost and LGBM: A Comprehensive Approach for High-Volume

Scaling XGBoost and LightGBM models to handle exceptionally large datasets—those comprising billions to tens of billions of rows—presents a formidable computational challenge, particularly when constrained by the limitations of in-memory processing o...

  • 502 Views
  • 1 replies
  • 0 kudos
Latest Reply
earntodiessaz
New Contributor II
  • 0 kudos

Well, that's a superb article! Thank you for this great information, you write very well which I like very much. I am really impressed by your post. run 3

  • 0 kudos
Chris_Shehu
by Valued Contributor III
  • 1258 Views
  • 1 replies
  • 1 kudos

Feature Request: GUI: Additional Collapse options

When you're using a very large notebook sometimes it gets frustrating scrolling through all the code blocks. It would be nice to have a few additional options to make this easier. 1) Add a collapse all code cells button to the top.2) Add a collapse a...

Community Platform Discussions
Enhancement
Feature
GUI
Request
  • 1258 Views
  • 1 replies
  • 1 kudos
Latest Reply
fdawoud
New Contributor II
  • 1 kudos

this feature please

  • 1 kudos
qwerty3
by Contributor
  • 646 Views
  • 1 replies
  • 0 kudos

Resolved! Does a queued databricks job incur cost?

Does a queued databricks job incur cost?

  • 646 Views
  • 1 replies
  • 0 kudos
Latest Reply
filipniziol
Esteemed Contributor
  • 0 kudos

Hi @qwerty3 ,No, it does not. When a job is queued (waiting for an available cluster or resources), there is no compute usage, so no charges are incurred for Databricks units (DBUs) or cloud infrastructure (VMs). The queued state is essentially a wai...

  • 0 kudos
tejaswi24
by New Contributor III
  • 1951 Views
  • 11 replies
  • 1 kudos

Resolved! databricks Asset Bundle

i have come accross a documentation on asset bundles long back whcih states that when you typedatabricks bundle initit gives us option to choose a project type. But i see the below error when i do that i see the below erroris there a way, i can take ...

tejaswi24_0-1727696851777.png
Community Platform Discussions
Databricks Asset Bundle
databricks bundle
  • 1951 Views
  • 11 replies
  • 1 kudos
Latest Reply
gchandra
Databricks Employee
  • 1 kudos

Bash

  • 1 kudos
10 More Replies
DW
by New Contributor II
  • 1092 Views
  • 1 replies
  • 2 kudos

column mask on <tinyint>Y columns gives error

My table breaks when I try to mask a column with a name like `<tinyint>Y` -- Create a table with a masked column> CREATE FUNCTION mask_int_col(col_val INTEGER) RETURN CASE WHEN is_member('HumanResourceDept') THEN col_val ELSE CAST(NULL as INTEGER) EN...

  • 1092 Views
  • 1 replies
  • 2 kudos
Latest Reply
filipniziol
Esteemed Contributor
  • 2 kudos

Hi @DW ,I have replicated your scenario and encountered the same error when applying a column mask to a column named 1Y in Databricks SQL.In short, it makes sense simply to follow Databricks documentation and use the SQL naming conventions, so that c...

  • 2 kudos
abueno
by New Contributor III
  • 479 Views
  • 1 replies
  • 0 kudos

Databricks Pyspark filter several columns with similar criteria

I am querying a table from the Databricks Catalog which I have to filter several columns with the same criteria.  below is what I have created so far.  I have 10 columns that I have filter with a set of criteria from (dx_list1) and another 10 that I ...

  • 479 Views
  • 1 replies
  • 0 kudos
Latest Reply
filipniziol
Esteemed Contributor
  • 0 kudos

Hi @abueno ,As I understand the logic you want to implement is:1. For every pair of columns:First Column (DX_i): Must be in dx_list1Second Column (DX_{i+1}): Must be in dx_list22. The condition for each pair is:col('DX_i').isin(dx_list1) OR col('DX_{...

  • 0 kudos
ayush19
by New Contributor III
  • 1616 Views
  • 3 replies
  • 1 kudos

How to retrieve Spark Session inside java jar library installed on Cluster

I have a java app in form of jar package. This jar is installed on a Databricks cluster. This jar package reads and writes to few tables in databricks. In order to achieve that, I need SparkSession available in the code. Given that spark session is a...

  • 1616 Views
  • 3 replies
  • 1 kudos
Latest Reply
IslaGray
New Contributor II
  • 1 kudos

Thanks for the update, I will try it too.

  • 1 kudos
2 More Replies
harshshah
by New Contributor
  • 397 Views
  • 0 replies
  • 0 kudos

Course registration issue

Hi All,I'm trying to register for "Spark Developer course" which is free course but I don't see this course under the course catalogue and if I try to directly access using a link I get 403 forbidden error. Can anyone help me figure out the issue.

  • 397 Views
  • 0 replies
  • 0 kudos
rseshadri
by New Contributor III
  • 1467 Views
  • 7 replies
  • 0 kudos

Unable to install the Databricks in AWS

I have followed the instructions given in the following link https://docs.databricks.com/en/admin/workspace/quick-start.html#use-aws-quickstart-to-create-a-databricks-workspace%22There was an error in CFT while deploying the workspace. Please help[ER...

  • 1467 Views
  • 7 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @rseshadri ,I'm more familiar with Azure, but this error is most commonly caused by entering the wrong credentials in the Cloudformation template.Try to launch the Quickstart again from the Databricks Account Console and double check that you ente...

  • 0 kudos
6 More Replies
reddybhargava
by New Contributor II
  • 440 Views
  • 2 replies
  • 1 kudos

How to delete object permissions using REST API

We have REST API methods to set,update,Get object permissions documented here https://docs.databricks.com/api/workspace/permissionsIs there a way to revoke the permissions which are granted through REST API

  • 440 Views
  • 2 replies
  • 1 kudos
Latest Reply
reddybhargava
New Contributor II
  • 1 kudos

Thank you for your response!We have a situation where we assigned CAN_BIND permissions to a group for a service principal using the following:$ vim grant-service-principal.json{"access_control_list": [{"user_name": "{username}","permission_level": "C...

  • 1 kudos
1 More Replies
devendra_tomar
by New Contributor
  • 1974 Views
  • 3 replies
  • 0 kudos

How to Read Data from Databricks Worker Nodes in Unity Catalog Volume

I am currently working on a similarity search use case where we need to extract text from PDF files and create a vector index. We have stored our PDF files in a Unity Catalog Volume, and I can successfully read these files from the driver node.Here's...

  • 1974 Views
  • 3 replies
  • 0 kudos
Latest Reply
ayush007
New Contributor II
  • 0 kudos

I am also looking for it in databricks runtime version 15.4

  • 0 kudos
2 More Replies
dbsuersu
by New Contributor II
  • 4118 Views
  • 2 replies
  • 0 kudos

ArcGIS Connection

Hi,I am trying to connect to an ArcGIS instance using Data bricks. Is this possible? After connecting, I am trying to read the data into a Data fame.Please help me with this request. If its not possible to connect , please provide an alternative.Than...

  • 4118 Views
  • 2 replies
  • 0 kudos
Latest Reply
garridolecca
New Contributor II
  • 0 kudos

Hello, please. Feel free to reach out if you need support deploying ArcGIS within Databricks. Nowadays, there are multiple ways for users to leverage the power of Databricks while using Esri's advanced tools and the app creators once the analysis is ...

  • 0 kudos
1 More Replies
HemanthG
by New Contributor
  • 853 Views
  • 3 replies
  • 0 kudos

Unable to install databricks free trial on AWS

I was trying to install databricks on AWS using these instructions "https://docs.databricks.com/en/admin/workspace/quick-start.html#use-aws-quickstart-to-create-a-databricks-workspace". But when CloudFormation Template stack executed "createCredentia...

  • 853 Views
  • 3 replies
  • 0 kudos
Latest Reply
rseshadri
New Contributor III
  • 0 kudos

I am facing the same error when installing the Databricks in AWS, Please find attached the error below and help [ERROR] 2024-10-08T12:11:56.458Z ad8dd24a-67ec-4941-8096-3f8d883994b2 Exception: Expecting value: line 1 column 1 (char 0)Traceback (most ...

  • 0 kudos
2 More Replies
shubbansal27
by New Contributor II
  • 12910 Views
  • 9 replies
  • 3 kudos

How to disable task in databricks job workflow

I don't see any option to disable the task. I have created one workflow which contains multiple tasks.sometime i need to disable some task for my experiment.  but there is no straightforward way for this.  I feel "disable" button is must have option ...

  • 12910 Views
  • 9 replies
  • 3 kudos
Latest Reply
VickiT
New Contributor II
  • 3 kudos

I know this post was a while ago, but I've hit this issue, and come up with using if/else conditions between the tasks with a Job parameter of "IsActiveTask2" with a True or False, and then I can have the Task2 dependant on the success of this condit...

  • 3 kudos
8 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors