cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

joselita
by New Contributor III
  • 22701 Views
  • 4 replies
  • 8 kudos

AnalysisException: is not a Delta table.

Hello, I changed the DBR from 7.2 to 10.4 and I receive the following error: AnalysisException: is not a Delta table. The table is create , using DELTA. so for sure is a Delta table, even though, I read that I read that from vers. 8 all tables are De...

STG_DATA_LOAD
  • 22701 Views
  • 4 replies
  • 8 kudos
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 8 kudos

Hi @JOSELITA MOLTISANTI​ can you run the following commands and share the output? table_name = "stg_data_load" path = spark.sql(f"describe detail {table_name}").select("location").collect()[0][0].replace('dbfs:', '') dbutils.fs.ls(path)

  • 8 kudos
3 More Replies
deisou
by New Contributor
  • 1506 Views
  • 1 replies
  • 0 kudos

How to change Databricks Academy username(Email)?

Is there a way to change my Databricks Academy username(email)? It is greyed out in my profile and I cannot update it. How do I go about getting it updated?

  • 1506 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Please go through the Databricks Academy FAQs here: https://files.training.databricks.com/lms/docebo/databricks-academy-faq.pdfAlso, please go through the post here: https://community.databricks.com/s/feed/0D53f00001dq6W6CAI.

  • 0 kudos
LearningDatabri
by Contributor II
  • 1913 Views
  • 2 replies
  • 1 kudos

Resolved! why this change in UI

The change in the UI is really confusing on what to use where. Earlier i had HC clusters and now I cant find it in new UI. It says HC clusters are not available. I want to use the HC cluster functions. where can I get that?

  • 1913 Views
  • 2 replies
  • 1 kudos
Latest Reply
Prabakar
Databricks Employee
  • 1 kudos

Hi @Databricks learner​ see if this post helps you.

  • 1 kudos
1 More Replies
yatharthmahesh
by New Contributor III
  • 2680 Views
  • 3 replies
  • 6 kudos

ENABLE CHANGE DATA FEED FOR EXISTING DELTA-TABLE

I have a delta table already created, now I want to enable the change data feed. I read that I have to set delta.enableChangeDataFeed property to true. But however, this cannot be done using the Scala API. I tried using this but it didn't work. I am ...

  • 2680 Views
  • 3 replies
  • 6 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

'delta.enableChangeDataFeed' have to be without quotes. spark.sql("ALTER TABLE delta_training.onaudience_dpm SET TBLPROPERTIES (delta.enableChangeDataFeed = true)").show()

  • 6 kudos
2 More Replies
_Orc
by New Contributor
  • 16586 Views
  • 5 replies
  • 3 kudos

Resolved! Precision and scale is getting changed in the dataframe while casting to decimal

When i run the below query in databricks sql the Precision and scale of the decimal column is getting changed.Select typeof(COALESCE(Cast(3.45 as decimal(15,6)),0));o/p: decimal(16,6)expected o/p: decimal(15,6)Any reason why the Precision and scale i...

  • 16586 Views
  • 5 replies
  • 3 kudos
Latest Reply
berserkersap
Contributor
  • 3 kudos

You can use typeof(COALESCE(Cast(3.45 as decimal(15,6)),0.0)); (instead of 0)

  • 3 kudos
4 More Replies
niels
by New Contributor III
  • 3654 Views
  • 3 replies
  • 10 kudos

Resolved! Change cluster mid-pipeline

I have a notebook functioning as a pipeline, where multiple notebooks are chained together. The issue I'm facing is that some of the notebooks are spark-optimized, others aren't, and what I want is to use 1 cluster for the former and another for the ...

  • 3654 Views
  • 3 replies
  • 10 kudos
Latest Reply
Prabakar
Databricks Employee
  • 10 kudos

Yes, you can achieve this by setting two different job clusters. In the screenshot, you can see I have used 2 job clusters PipelineTest and pipelinetest2. You can refer the doc https://docs.databricks.com/jobs.html#cluster-config-tips

  • 10 kudos
2 More Replies
petilodie
by New Contributor III
  • 1483 Views
  • 1 replies
  • 3 kudos

Resolved! How can I change the admin settings by Terraform?

I need to update most of the settings that are visible on the Admin Console UI by using Terraform. In another post in this forum I saw that I can use `custom_config` in a `databricks_workspace_conf` resource to achieve that but the options seem limit...

  • 1483 Views
  • 1 replies
  • 3 kudos
Latest Reply
petilodie
New Contributor III
  • 3 kudos

Ok, looks like I can inspect the network and see which flags are sent to the endpoint. Tried that and it worked.

  • 3 kudos
Hola1801
by New Contributor
  • 1784 Views
  • 3 replies
  • 3 kudos

Resolved! Float Value change when Load with spark? Full Path?

Hello,I have created my table in Databricks, at this point everything is perfect i got the same value than in my CSV. for my column "Exposure" I have :0 0,00 1 0,00 2 0,00 3 0,00 4 0,00 ...But when I load my fi...

  • 1784 Views
  • 3 replies
  • 3 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 3 kudos

Hi @Anis Ben Salem​ ,How do you read your CSV file? do you use Pandas or Pyspark APIs? also, how do you created your table?could you share more details on the code you are trying to run?

  • 3 kudos
2 More Replies
MarcLimotte
by New Contributor II
  • 23432 Views
  • 12 replies
  • 0 kudos

Why do I get 'java.io.IOException: File already exists' for saveAsTable with Overwrite mode?

I have a fairly small, simple DataFrame, month:month.schema org.apache.spark.sql.types.StructType = StructType(StructField(month,DateType,true), StructField(real_month,TimestampType,true), StructField(month_millis,LongType,true))The month Dataframe i...

  • 23432 Views
  • 12 replies
  • 0 kudos
Latest Reply
ReKa
New Contributor III
  • 0 kudos

Your schema is tight, but make sure that the conversion to it does not throw an exception. Try with Memory Optimized Nodes, you may be fine. My problem was parsing a lot of data from sequence files containing 10K xml files and saving them as a table...

  • 0 kudos
11 More Replies
Labels