cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Anonymous
by • Not applicable
  • 15236 Views
  • 2 replies
  • 3 kudos

Resolved! Is there a way to list packages / version installed on a cluster ?

Data Engineering
Reply
  • 15236 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
zerasmus
Contributor
  • 3 kudos

On newer Databricks Runtime versions, %conda commands are not supported. You can use %pip commands instead:%pip list I have tested this on Databricks Runtime 15.4 LTS Beta.

  • 3 kudos
1 More Replies
Student185
by • New Contributor III
  • 6903 Views
  • 7 replies
  • 5 kudos

Resolved! Is that long-term free version for students still available now?

Dear sir/madam,I've tried lots of methods in order to access the long-term Databricks' free version - community version for students.Also, I followed the instructions - Introduction to Databricks - in Coursera step by step: https://www.coursera.org/l...

Data Engineering
Reply
  • 6903 Views
  • 7 replies
  • 5 kudos
View Replies
Latest Reply
shreeves
New Contributor II
  • 5 kudos

Look for the "Community Edition" in small print below the button

  • 5 kudos
6 More Replies
ALIDI
by • New Contributor II
  • 1767 Views
  • 3 replies
  • 3 kudos

training_set.load_df().toPandas() fails with the new pandas version (2.0.0)

pandas 2.0.0 was released on 4.3.2023 and was pushed to my cluster on the same day. The day after I tried using training_set.load_df().toPandas() and it failed. Reverting to pandas 1.5.3. fixed the problem.

Data Engineering
Reply
  • 1767 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Al IDI​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your q...

  • 3 kudos
2 More Replies
William_Scardua
by • Valued Contributor
  • 1545 Views
  • 2 replies
  • 1 kudos

Cosmos DB Connector for 12.1 Cluster and above

Hi guys,​You know what`s the version suporte the databricks cluster version above 12.1 ?my cluster:error: Thank you

databricks-cluster cosmosdb-connector-erro
Data Engineering
Reply
  • 1545 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 1 kudos
1 More Replies
Mahesh_789
by • New Contributor II
  • 676 Views
  • 0 replies
  • 1 kudos

While accessing the data on recipient side using delta_sharing.load_table_changes_as_spark(), it shows data of all versions.

When I tried to access specific version data and set the arguments value to the specific number, I get all version data.data1 = delta_sharing.load_table_changes_as_spark(table_url, starting_version=1, ending_version=1)data2 = delta_sharing.load_table...

Data Engineering
Reply
  • 676 Views
  • 0 replies
  • 1 kudos
nameziane
by • New Contributor III
  • 8084 Views
  • 4 replies
  • 2 kudos

Set version (VERSION AS OF) dynamically from return of a subquery

Hello,We have a business request to compare the evolution in a certain delta table.We would like to compare the latest version of the table with the previous one using Delta time travel.The main issue we are facing is to retrieve programmatically us...

Data Engineering
Reply
  • 8084 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
apingle
Contributor
  • 2 kudos

In the docs it says that "'Neither timestamp_expression nor version can be subqueries." So it does sound challenging. I also tried playing with widgets to see if it could be populated using SQL but didn't succeed. With python it's really easy to do.

  • 2 kudos
3 More Replies
Yaswanth
by • New Contributor III
  • 16922 Views
  • 5 replies
  • 18 kudos

Resolved! How can Delta table protocol version be downgraded from higher version to lower version the table properties minReader from 2 to 1 and MaxWriter from 5 to 3.

Is there a possibility to downgrade the Delta Table protocol versions minReader from 2 to 1 and maxWriter from 5 to 3? I have set the TBL properties to 2 and 5 and columnmapping mode to rename the columns in the DeltaTable but the other users are rea...

Data Engineering
Reply
  • 16922 Views
  • 5 replies
  • 18 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 18 kudos

Hi @Yaswanth velkur, We haven’t heard from you since the last response from @Youssef Mrini​ and me​​, and I was checking back to see if our suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...

  • 18 kudos
4 More Replies
vk217
by • Contributor
  • 4320 Views
  • 3 replies
  • 1 kudos

Resolved! ERROR: No matching distribution found for databricks-connect==7.3.34

Previously, our databricks-connect was using 7.3.34 and the builds in pipenv and the builds were successful. As of today the builds are failing with error that the version 7.3.34 no longer exists.Is there a reason this version is no longer supported....

Data Engineering
Reply
  • 4320 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Atanu
Esteemed Contributor
  • 1 kudos

Hello @Vikas B​ this is the release note -https://docs.databricks.com/release-notes/dbconnect/index.htmlalso,Only the following Databricks Runtime versions are supported:Databricks Runtime 10.4 LTS ML, Databricks Runtime 10.4 LTSDatabricks Runtime 9....

  • 1 kudos
2 More Replies
User16137833804
by • New Contributor III
  • 3172 Views
  • 3 replies
  • 1 kudos

How do I specify a version of a library different from the Databricks runtime?

Data Engineering
Reply
  • 3172 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Sebastian
Contributor
  • 1 kudos

the best solution is to store the .whl locally and do a pip install of the local whl while server boots up. this will freeze the library version. if you install from the pip it might impact your production work.

  • 1 kudos
2 More Replies
brickster_2018
by • Esteemed Contributor
  • 1262 Views
  • 1 replies
  • 0 kudos

Resolved! What is the trade-off of using an unsupported DBR version on my cluster?

I do not want to upgrade my cluster every one month. I am looking for stability over new features.

Data Engineering
Reply
  • 1262 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Esteemed Contributor
  • 0 kudos

The strong recommendation is not to use an unsupported version of DBR on your cluster. For production workloads where you don't welcome newer versions, then check the Databricks LTS DBR versions. if using an unsupported version then you don't receiv...

  • 0 kudos
User16783853906
by • Contributor III
  • 3995 Views
  • 1 replies
  • 0 kudos

Metaexception [Version information not found in metastore] during cluster [re]start

Trying to configure new external metastore and running into the following exception during cluster initialization - Caused by: MetaException(message:Version information not found in metastore. )   at org.apache.hadoop.hive.metastore.RetryingHMSHandl...

Data Engineering
Reply
  • 3995 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16783853906
Contributor III
  • 0 kudos

The above exception happens when the hive schema is not available in the metastore instance. Please check in your init scripts to make sure the following flag is enabled to create hive Schema and tables if not already present. datanucleus.autoCreateA...

  • 0 kudos
Labels