cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
SDas1
by • New Contributor
  • 8162 Views
  • 2 replies
  • 2 kudos

Identity column value of Databricks delta table is not started with 0 and increaed by 1. It always started with something like 1 or 2 and increased by 2. Below is the sample code and any logical input here is appreciated

spark.sql("CREATE TABLE integrated.TrailingWeeks(ID bigint GENERATED BY DEFAULT AS IDENTITY (START WITH 0 increment by 1) ,Week_ID int NOT NULL) USING delta OPTIONS (path 'dbfs:/<Path in Azure datalake>/delta')")

Data Engineering
Reply
  • 8162 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
agallard
Contributor
  • 2 kudos

Hi,When you define an identity column in  Databricks with GENERATED BY DEFAULT AS IDENTITY (START WITH 0 INCREMENT BY 1), it is expected to start at 0 and increment by 1. However, due to Databricks' distributed architecture, the values may not be str...

  • 2 kudos
1 More Replies
gazzyjuruj
by • Contributor II
  • 10959 Views
  • 5 replies
  • 11 kudos

Cluster start is currently disabled ?

Hi, i'm trying to run the notebooks but it doesn't do any activity.I had to create a cluster in order to start my code.pressing the play button inside of notebook does nothing at all.and the 'compute' , pressing play there on the clusters gives the e...

Data Engineering
Reply
  • 10959 Views
  • 5 replies
  • 11 kudos
View Replies
Latest Reply
mrp12
New Contributor III
  • 11 kudos

This is very common issue I see with community edition. I suppose the only work around is to create new cluster each time. More info on stackoverflow:https://stackoverflow.com/questions/69072694/databricks-community-edition-cluster-wont-start

  • 11 kudos
4 More Replies
JJ_LVS1
by • New Contributor III
  • 3180 Views
  • 4 replies
  • 1 kudos

FiscalYear Start Period Is not Correct

Hi, I'm trying to create a calendar dimension including a fiscal year with a fiscal start of April 1. I'm using the fiscalyear library and am setting the start to month 4 but it insists on setting April to month 7.runtime 12.1My code snipet is:start_...

Data Engineering
Reply
  • 3180 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
DataEnginner
New Contributor II
  • 1 kudos

 import fiscalyear import datetime def get_fiscal_date(year,month,day): fiscalyear.setup_fiscal_calendar(start_month=4) v_fiscal_month=fiscalyear.FiscalDateTime(year, month, day).fiscal_month #To get the Fiscal Month v_fiscal_quarter=fiscalyea...

  • 1 kudos
3 More Replies
Himanshu_90
by • New Contributor III
  • 6138 Views
  • 8 replies
  • 7 kudos

Databricks sql not able to evaluate expression current_user

Hi,I have a table as below:create table default.test_user(ID bigint NOT NULL GENERATED BY DEFAULT AS IDENTITY (START WITH 1 INCREMENT BY 1),usr1 varchar(255) NOT NULL,ts1 timestamp NOT NULL,usr2 varchar(255) NOT NULL,ts2 timestamp NOT NULL) USING Del...

Data Engineering
Reply
  • 6138 Views
  • 8 replies
  • 7 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @Himanshu Agrawal​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...

  • 7 kudos
7 More Replies
lzha174
by • Contributor
  • 3104 Views
  • 4 replies
  • 16 kudos

ipywidget not able to start running another notebook

I am trying to add a button in a notebook to trigger an execution of another notebook, but it does not respond to the command. Any idea why? The run command works if I run in a separate cell

image
Data Engineering
Reply
  • 3104 Views
  • 4 replies
  • 16 kudos
View Replies
Latest Reply
mbejarano89
New Contributor III
  • 16 kudos

Did you end up finding an answer to this? I am running into the same problem

  • 16 kudos
3 More Replies
Ajay-Pandey
by • Esteemed Contributor III
  • 3847 Views
  • 9 replies
  • 11 kudos

Databricks start support to run selected text in a cell this will help us a lot during debugging of the code.In windows just select the line of code w...

Databricks start support to run selected text in a cell this will help us a lot during debugging of the code.In windows just select the line of code which you want to execute and press Ctrl+Shift+Enter

sele
Data Engineering
Reply
  • 3847 Views
  • 9 replies
  • 11 kudos
View Replies
Latest Reply
Nhan_Nguyen
Valued Contributor
  • 11 kudos

Thanks @Ajay Pandey​ nice sharing

  • 11 kudos
8 More Replies
ty2
by • New Contributor II
  • 2415 Views
  • 3 replies
  • 1 kudos

Resolved! How to start my cluster

​I try to stop my_cluster from compute from admin role. BTW, using same account, I could not restart my_cluster. The information is as followings. How should I do?

20230121-my_cluster_not_start
Data Engineering
Reply
  • 2415 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 1 kudos

rit seems this is community edition so in CE this feature is disabled , delete this one and create new cluster

  • 1 kudos
2 More Replies
Mado
by • Valued Contributor II
  • 1837 Views
  • 2 replies
  • 3 kudos

When should I use ".start()" with writeStream?

Hi,I am practicing with Databricks. In sample notebooks,I have seen different use of writeStream with or without ".start()" method. Samples are below:Without .start() spark.readStream   .format("cloudFiles")   .option("cloudFiles.f...

Data Engineering
Reply
  • 1837 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Mohammad Saber​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks

  • 3 kudos
1 More Replies
PabloSeoaneF
by • New Contributor III
  • 4891 Views
  • 5 replies
  • 15 kudos

Databricks cluster's problems at start

Hello,I've spent some time trying to initialize a couple of clusters into an Azure Databricks private environment.Apparently, without reason they've been causing these two failures: With the advice of retrying in some minutes. Looking for information...

image.png image image
Data Engineering
Reply
  • 4891 Views
  • 5 replies
  • 15 kudos
View Replies
Latest Reply
PabloSeoaneF
New Contributor III
  • 15 kudos

Hello both! I attach here the 2 error details:Bootstrap timeout:HelpBootstrap Timeout. Please try again later.Instance bootstrap failed command: BootstrapTimeoutFailure message: Bootstrap script took too long and timeout. Please try again later.VM ex...

  • 15 kudos
4 More Replies
lizou
by • Contributor II
  • 3520 Views
  • 3 replies
  • 5 kudos

Resolved! Identity column definition lost using save as table

I found an issue:For a table with an identity column defined.when the table column is renamed using this method, the identity definition will be removed. That means using an identity column in a table requires extra attention to check whether the ide...

Data Engineering
Reply
  • 3520 Views
  • 3 replies
  • 5 kudos
View Replies
Latest Reply
lizou
Contributor II
  • 5 kudos

try to avoid reload table, I found we can upgrade table version, and use rename column commandALTER TABLE test_id2 SET TBLPROPERTIES (  'delta.columnMapping.mode' = 'name',  'delta.minReaderVersion' = '2',  'delta.minWriterVersion' = '6')ALTER TABLE ...

  • 5 kudos
2 More Replies
learnerbricks
by • New Contributor II
  • 1378 Views
  • 2 replies
  • 0 kudos

how should I start databricks ?

Hello Guys,I am new to databricks. I have try to read the documentation as much I can. Now I want to jump in. What I Want : I have store my parquet file in Databricks storage system. I want to load this file into Data Lake Table. And then want to do ...

Data Engineering
Reply
  • 1378 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Learner bricks​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
1 More Replies
Venkat1
by • New Contributor II
  • 4076 Views
  • 3 replies
  • 1 kudos

Resolved! Error: "Backend service unavailable" when trying to start a cluster in community edition.

I am using community edition of databricks for learning and hands-on projects. However, when I try to create a cluster today, I am getting an error popup- "Backend service unavailable". I would like to know if it is a problem with my account or a bac...

Data Engineering
Reply
  • 4076 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hey there @Venkat K​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 1 kudos
2 More Replies
auser85
by • New Contributor III
  • 2464 Views
  • 2 replies
  • 1 kudos

How to reset the IDENTITY column count?

After accumulating many updates to a delta table,like,keyExample bigint GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1),my identity column values are in the hundreds of millions. Is there any way that I can reset this value through vacuumi...

Data Engineering
Reply
  • 2464 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey there @Andrew Fogarty​ Does @Werner Stinckens​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help. Thanks!

  • 1 kudos
1 More Replies
MatheusData
by • New Contributor II
  • 1688 Views
  • 1 replies
  • 0 kudos

Failed to start cluster.

Hello,i'm getting this message today everytime i try to run a notebook since i logged in:When i click 'ok' nothing happens.I tried refreshing and switching browsers already. Tried run this cell, run below cells, run all cells, etc. I also tried to cr...

image
Data Engineering
Reply
  • 1688 Views
  • 1 replies
  • 0 kudos
View Replies
flachboard
by • New Contributor
  • 4346 Views
  • 3 replies
  • 1 kudos

How do you install R packages?

I've tried this, but it doesn't appear to be working: https://community.databricks.com/s/question/0D53f00001GHVX1CAP/unable-to-install-sf-and-rgeos-r-packages-on-the-clusterWhen I run the following after that init script, I receive an error.library(r...

Data Engineering
Reply
  • 4346 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey there @Christopher Flach​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...

  • 1 kudos
2 More Replies
Labels