cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

tariq
by New Contributor III
  • 5049 Views
  • 4 replies
  • 0 kudos

Importing python module

I'm not sure how a simple thing like importing a module in python can be so broken in such a product. First, I was able to make it work using the following:import sys sys.path.append("/Workspace/Repos/Github Repo/sparkling-to-databricks/src") from ut...

  • 5049 Views
  • 4 replies
  • 0 kudos
Latest Reply
KrishZ
Contributor
  • 0 kudos

I too wonder the same thing. How can importing a python module be so difficult and not even documented lol.No need for libraries..Here's what worked for me..Step1: Upload the module by first opening a notebook >> File >> Upload Data >> drag and drop ...

  • 0 kudos
3 More Replies
DineshNO
by New Contributor II
  • 1847 Views
  • 1 replies
  • 1 kudos

How to execute a spark-submit on a databricks job for a spring-boot jar built using maven. Failing with error : Error: Failed to load class com.****.settlement.jobs.EntryPoint.

I have setup a spring boot application which works as expected as a standalone spring boot app.When i build the jar and try to set it up as a databricks job, i am facing these issues.i am getting same error in local as well.I have tried using maven-s...

  • 1847 Views
  • 1 replies
  • 1 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 1 kudos

could you please try with python terminal and see how that behaves?I am not 100% sure if this is relates to your use case.@Dinesh L​ 

  • 1 kudos
F29
by New Contributor
  • 1639 Views
  • 3 replies
  • 0 kudos

is possible duplicated a new job in another stage Databricks? Devops or some way?

I need duplicated a new job create in stage A in another stage, automatically. is posible?

  • 1639 Views
  • 3 replies
  • 0 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 0 kudos

you may try to get the job details from our job api https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsGet and get the response to duplicate it.,

  • 0 kudos
2 More Replies
pantelis_mare
by Contributor III
  • 1757 Views
  • 3 replies
  • 0 kudos

Spark 3 AQE and cache

Hello everybody,I recently discovered (the hard way) that when a query plan uses cached data, the AQE does not kick-in. Result is that you loose the super cool feature of dynamic partition coalesce (no more custom shuffle readers in the DAG). Is ther...

  • 1757 Views
  • 3 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Hi @Pantelis Maroudis​,Did you check the physical query plan? did you check the SQL sub tab with in Spark UI? it will help you to undertand better what is happening.

  • 0 kudos
2 More Replies
pantelis_mare
by Contributor III
  • 2435 Views
  • 6 replies
  • 1 kudos

Too long hive type string

Hello community!I have a table with a column that is an array of a struct that has a very very long schema.When the table is written, all works well. Though, when I create a view based on this table and I try to access the view I get the error:rg.apa...

  • 2435 Views
  • 6 replies
  • 1 kudos
Latest Reply
Afzal
New Contributor II
  • 1 kudos

@Pantelis Maroudis​ , were you able to solve this issue? Please advise if you got any tip. Thanks in advance

  • 1 kudos
5 More Replies
BradSheridan
by Valued Contributor
  • 1023 Views
  • 1 replies
  • 0 kudos

Resolved! Using a UDF in %sql?

Afternoon everyone! I logged in hoping to see some suggestions but think maybe I need to reword the question a little How can I create a UDF that converts '30000101' to timestamp and then use it in a query like below?%sqlselectfield1,field2,nvl(some...

  • 1023 Views
  • 1 replies
  • 0 kudos
Latest Reply
BradSheridan
Valued Contributor
  • 0 kudos

Got it working (but going to post a new question momentarily): I needed to use timestamp(date '3000-01-01) instead of to_timestamp

  • 0 kudos
113775
by New Contributor
  • 3412 Views
  • 1 replies
  • 0 kudos

Running query on INFORMATION_SCHEMA.COLUMNS

Hi,I am trying to run following query:SELECT table_schema, table_name, COUNT(column_name) FROM {db_name}.INFORMATION_SCHEMA.COLUMNS GROUP BY table_schema, table_nameand I am getting following error:Error in SQL statement: AnalysisException: Catalog n...

  • 3412 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16741082858
Contributor III
  • 0 kudos

Hi @Suman Karki​, can you check if UC is enabled in the advanced settings of the endpoint? Also, what DBR is your DE cluster, and what Security Mode did you choose?

  • 0 kudos
elmohan
by New Contributor
  • 652 Views
  • 1 replies
  • 0 kudos

New Databricks Large Tables

At one of the Databricks presentations in August a new type of table was introduced, I kind of remember these were large tables which did not change much. I think I heard these new table typed were in Beta. Unfortunately I lost my notes for that pres...

  • 652 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Thanks for reaching out to community.databricks.com. Are you talking about data objects in Databricks lakehouse? https://docs.databricks.com/lakehouse/data-objects.html

  • 0 kudos
JohnMartensDB
by New Contributor II
  • 1470 Views
  • 4 replies
  • 1 kudos

Error with live connection SQL Datawarehouse

Error type: System call error. Cannot make named pipe /tmp/EvtnUM74TA574813952 due to error: No space left on device..))I can't find anything recent online related to Databricks or the SQL warehouses specifically. I don't seem to have enough monitori...

  • 1470 Views
  • 4 replies
  • 1 kudos
Latest Reply
JohnMartensDB
New Contributor II
  • 1 kudos

Microstrategy connects to a small SQL Datawarehouse (SQL endpoint), which I guess is serverless by default. I don't believe the Databricks UI gives many options other than size and cost efficient or reliable setting. May be a temporary issue since it...

  • 1 kudos
3 More Replies
TMNGB
by New Contributor II
  • 1260 Views
  • 0 replies
  • 2 kudos

Databricks to Azure Synapse SQL Server: error converting between Spark and Parquet column types

When writing data from Pyspark to Azure SQL Server (official databricks tutorial here) I am getting an error in the conversion between Spark and Parquet types.I believe this is caused in the temporary storage location mandatory when writing data from...

  • 1260 Views
  • 0 replies
  • 2 kudos
PriyaAnanthram
by Contributor III
  • 783 Views
  • 2 replies
  • 0 kudos

Past and Present sessions

Hi DatabricksCould we please regarding information about upcoming sessions and recorded videos of past sessions .This would be v helpful for users like meKind RegardsPriya

  • 783 Views
  • 2 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

the upcoming events are available on the community homepage.

  • 0 kudos
1 More Replies
jasperputs
by New Contributor III
  • 4969 Views
  • 6 replies
  • 3 kudos

Resolved! Add Identity Column to Existing Table

Hello everyone. I am working with tables that need an identity column. I currently have a view in which I cast the different columns to the data type that I want. Now I want the result of this view to be inserted or merged into a table. The schema of...

image
  • 4969 Views
  • 6 replies
  • 3 kudos
Latest Reply
ramankr48
Contributor II
  • 3 kudos

Hello @Jasper Puts​ how did you solve this issue of creating a identity column to existing table.I'm also getting the same error as you got.

  • 3 kudos
5 More Replies
Ravikumashi
by Contributor
  • 950 Views
  • 3 replies
  • 0 kudos

compatibility issue between 7.3 LTS and 10.4 LTS

We have views created with mixed case column names in 7.3 LTS and when we do select * from view everything looks OK I mean column names display in mixed case and problem started as we recently migrated to 10.4 LTS.Same views created with create or re...

  • 950 Views
  • 3 replies
  • 0 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 0 kudos

@Ravikumar Kumashi​ other than the DBR version, do you have any other configuration on the clusters? Also, is it possible for you to share a sample table and code so we could reproduce this?

  • 0 kudos
2 More Replies
merca
by Valued Contributor II
  • 6255 Views
  • 12 replies
  • 4 kudos

Value array {{QUERY_RESULT_ROWS}} in Databricks SQL alerts custom template

Please include in documentation an example how to incorporate the `QUERY_RESULT_ROWS` variable in the custom template.

  • 6255 Views
  • 12 replies
  • 4 kudos
Latest Reply
jose_gonzalez
Moderator
  • 4 kudos

Hi @Merca Ovnerud​ ,Here is the docs link https://docs.databricks.com/sql/user/alerts/index.html please let me know if this helps or you still have more follow questions.

  • 4 kudos
11 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels