cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Anonymous
by Not applicable
  • 1923 Views
  • 0 replies
  • 0 kudos

Data + AI World Tour �� ✈️ Data + AI World Tour brings the data lakehouse to the global datacommunity. With content, customers and speakers tai...

Data + AI World Tour Data + AI World Tour brings the data lakehouse to the global datacommunity. With content, customers and speakers tailored to eachregion, the tour showcases how and why the data lakehouse is quicklybecoming the cloud data archite...

  • 1923 Views
  • 0 replies
  • 0 kudos
ahuarte
by New Contributor III
  • 27392 Views
  • 17 replies
  • 3 kudos

Resolved! Getting Spark & Scala version in Cluster node initialization script

Hi there, I am developing a Cluster node initialization script (https://docs.gcp.databricks.com/clusters/init-scripts.html#environment-variables) in order to install some custom libraries.Reading the docs of Databricks we can get some environment var...

  • 27392 Views
  • 17 replies
  • 3 kudos
Latest Reply
Lingesh
Databricks Employee
  • 3 kudos

We can infer the cluster DBR version using the env $DATABRICKS_RUNTIME_VERSION. (For the exact spark/scala version mapping, you can refer to the specific DBR release notes)Sample usage inside a init script, DBR_10_4_VERSION="10.4" if [[ "$DATABRICKS_...

  • 3 kudos
16 More Replies
nickagel
by New Contributor III
  • 7071 Views
  • 5 replies
  • 4 kudos

AWS Glue Catalog w/ Delta Tables Connected to Databricks SQL Engine - Incompatible format detected.

I've posted the same question on stack overflow to try to maximize reach here & potentially raise this issue to Databricks.I am trying to query delta tables from my AWS Glue Catalog on Databricks SQL Engine. They are stored in Delta Lake format. I ha...

  • 7071 Views
  • 5 replies
  • 4 kudos
Latest Reply
Vidula
Honored Contributor
  • 4 kudos

Hi @Nick Agel​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 4 kudos
4 More Replies
tariq
by New Contributor III
  • 11641 Views
  • 4 replies
  • 0 kudos

Importing python module

I'm not sure how a simple thing like importing a module in python can be so broken in such a product. First, I was able to make it work using the following:import sys sys.path.append("/Workspace/Repos/Github Repo/sparkling-to-databricks/src") from ut...

  • 11641 Views
  • 4 replies
  • 0 kudos
Latest Reply
KrishZ
Contributor
  • 0 kudos

I too wonder the same thing. How can importing a python module be so difficult and not even documented lol.No need for libraries..Here's what worked for me..Step1: Upload the module by first opening a notebook >> File >> Upload Data >> drag and drop ...

  • 0 kudos
3 More Replies
DineshNO
by New Contributor II
  • 3133 Views
  • 1 replies
  • 1 kudos

How to execute a spark-submit on a databricks job for a spring-boot jar built using maven. Failing with error : Error: Failed to load class com.****.settlement.jobs.EntryPoint.

I have setup a spring boot application which works as expected as a standalone spring boot app.When i build the jar and try to set it up as a databricks job, i am facing these issues.i am getting same error in local as well.I have tried using maven-s...

  • 3133 Views
  • 1 replies
  • 1 kudos
Latest Reply
Atanu
Databricks Employee
  • 1 kudos

could you please try with python terminal and see how that behaves?I am not 100% sure if this is relates to your use case.@Dinesh L​ 

  • 1 kudos
F29
by New Contributor
  • 3816 Views
  • 3 replies
  • 0 kudos

is possible duplicated a new job in another stage Databricks? Devops or some way?

I need duplicated a new job create in stage A in another stage, automatically. is posible?

  • 3816 Views
  • 3 replies
  • 0 kudos
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

you may try to get the job details from our job api https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsGet and get the response to duplicate it.,

  • 0 kudos
2 More Replies
pantelis_mare
by Contributor III
  • 4380 Views
  • 3 replies
  • 0 kudos

Spark 3 AQE and cache

Hello everybody,I recently discovered (the hard way) that when a query plan uses cached data, the AQE does not kick-in. Result is that you loose the super cool feature of dynamic partition coalesce (no more custom shuffle readers in the DAG). Is ther...

  • 4380 Views
  • 3 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Pantelis Maroudis​,Did you check the physical query plan? did you check the SQL sub tab with in Spark UI? it will help you to undertand better what is happening.

  • 0 kudos
2 More Replies
pantelis_mare
by Contributor III
  • 5728 Views
  • 6 replies
  • 1 kudos

Too long hive type string

Hello community!I have a table with a column that is an array of a struct that has a very very long schema.When the table is written, all works well. Though, when I create a view based on this table and I try to access the view I get the error:rg.apa...

  • 5728 Views
  • 6 replies
  • 1 kudos
Latest Reply
Afzal
New Contributor II
  • 1 kudos

@Pantelis Maroudis​ , were you able to solve this issue? Please advise if you got any tip. Thanks in advance

  • 1 kudos
5 More Replies
BradSheridan
by Valued Contributor
  • 2150 Views
  • 1 replies
  • 0 kudos

Resolved! Using a UDF in %sql?

Afternoon everyone! I logged in hoping to see some suggestions but think maybe I need to reword the question a little How can I create a UDF that converts '30000101' to timestamp and then use it in a query like below?%sqlselectfield1,field2,nvl(some...

  • 2150 Views
  • 1 replies
  • 0 kudos
Latest Reply
BradSheridan
Valued Contributor
  • 0 kudos

Got it working (but going to post a new question momentarily): I needed to use timestamp(date '3000-01-01) instead of to_timestamp

  • 0 kudos
113775
by New Contributor
  • 4883 Views
  • 1 replies
  • 0 kudos

Running query on INFORMATION_SCHEMA.COLUMNS

Hi,I am trying to run following query:SELECT table_schema, table_name, COUNT(column_name) FROM {db_name}.INFORMATION_SCHEMA.COLUMNS GROUP BY table_schema, table_nameand I am getting following error:Error in SQL statement: AnalysisException: Catalog n...

  • 4883 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16741082858
Databricks Employee
  • 0 kudos

Hi @Suman Karki​, can you check if UC is enabled in the advanced settings of the endpoint? Also, what DBR is your DE cluster, and what Security Mode did you choose?

  • 0 kudos
elmohan
by New Contributor
  • 1380 Views
  • 1 replies
  • 0 kudos

New Databricks Large Tables

At one of the Databricks presentations in August a new type of table was introduced, I kind of remember these were large tables which did not change much. I think I heard these new table typed were in Beta. Unfortunately I lost my notes for that pres...

  • 1380 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Thanks for reaching out to community.databricks.com. Are you talking about data objects in Databricks lakehouse? https://docs.databricks.com/lakehouse/data-objects.html

  • 0 kudos
JohnMartensDB
by New Contributor II
  • 3070 Views
  • 4 replies
  • 1 kudos

Error with live connection SQL Datawarehouse

Error type: System call error. Cannot make named pipe /tmp/EvtnUM74TA574813952 due to error: No space left on device..))I can't find anything recent online related to Databricks or the SQL warehouses specifically. I don't seem to have enough monitori...

  • 3070 Views
  • 4 replies
  • 1 kudos
Latest Reply
JohnMartensDB
New Contributor II
  • 1 kudos

Microstrategy connects to a small SQL Datawarehouse (SQL endpoint), which I guess is serverless by default. I don't believe the Databricks UI gives many options other than size and cost efficient or reliable setting. May be a temporary issue since it...

  • 1 kudos
3 More Replies
TMNGB
by New Contributor II
  • 2716 Views
  • 0 replies
  • 2 kudos

Databricks to Azure Synapse SQL Server: error converting between Spark and Parquet column types

When writing data from Pyspark to Azure SQL Server (official databricks tutorial here) I am getting an error in the conversion between Spark and Parquet types.I believe this is caused in the temporary storage location mandatory when writing data from...

  • 2716 Views
  • 0 replies
  • 2 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels