cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Anonymous
by • Not applicable
  • 17151 Views
  • 2 replies
  • 3 kudos

Resolved! Is there a way to list packages / version installed on a cluster ?

Data Engineering
Reply
  • 17151 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
zerasmus
Contributor
  • 3 kudos

On newer Databricks Runtime versions, %conda commands are not supported. You can use %pip commands instead:%pip list I have tested this on Databricks Runtime 15.4 LTS Beta.

  • 3 kudos
1 More Replies
Student185
by • New Contributor III
  • 8921 Views
  • 7 replies
  • 5 kudos

Resolved! Is that long-term free version for students still available now?

Dear sir/madam,I've tried lots of methods in order to access the long-term Databricks' free version - community version for students.Also, I followed the instructions - Introduction to Databricks - in Coursera step by step: https://www.coursera.org/l...

Data Engineering
Reply
  • 8921 Views
  • 7 replies
  • 5 kudos
View Replies
Latest Reply
shreeves
New Contributor II
  • 5 kudos

Look for the "Community Edition" in small print below the button

  • 5 kudos
6 More Replies
ALIDI
by • New Contributor II
  • 2021 Views
  • 3 replies
  • 3 kudos

training_set.load_df().toPandas() fails with the new pandas version (2.0.0)

pandas 2.0.0 was released on 4.3.2023 and was pushed to my cluster on the same day. The day after I tried using training_set.load_df().toPandas() and it failed. Reverting to pandas 1.5.3. fixed the problem.

Data Engineering
Reply
  • 2021 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Al IDI​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your q...

  • 3 kudos
2 More Replies
William_Scardua
by • Valued Contributor
  • 1893 Views
  • 2 replies
  • 1 kudos

Cosmos DB Connector for 12.1 Cluster and above

Hi guys,​You know what`s the version suporte the databricks cluster version above 12.1 ?my cluster:error: Thank you

databricks-cluster cosmosdb-connector-erro
Data Engineering
Reply
  • 1893 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 1 kudos
1 More Replies
Mahesh_789
by • New Contributor II
  • 797 Views
  • 0 replies
  • 1 kudos

While accessing the data on recipient side using delta_sharing.load_table_changes_as_spark(), it shows data of all versions.

When I tried to access specific version data and set the arguments value to the specific number, I get all version data.data1 = delta_sharing.load_table_changes_as_spark(table_url, starting_version=1, ending_version=1)data2 = delta_sharing.load_table...

Data Engineering
Reply
  • 797 Views
  • 0 replies
  • 1 kudos
nameziane
by • New Contributor III
  • 10798 Views
  • 4 replies
  • 2 kudos

Set version (VERSION AS OF) dynamically from return of a subquery

Hello,We have a business request to compare the evolution in a certain delta table.We would like to compare the latest version of the table with the previous one using Delta time travel.The main issue we are facing is to retrieve programmatically us...

Data Engineering
Reply
  • 10798 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
apingle
Contributor
  • 2 kudos

In the docs it says that "'Neither timestamp_expression nor version can be subqueries." So it does sound challenging. I also tried playing with widgets to see if it could be populated using SQL but didn't succeed. With python it's really easy to do.

  • 2 kudos
3 More Replies
Yaswanth
by • New Contributor III
  • 18113 Views
  • 2 replies
  • 12 kudos

Resolved! How can Delta table protocol version be downgraded from higher version to lower version the table properties minReader from 2 to 1 and MaxWriter from 5 to 3.

Is there a possibility to downgrade the Delta Table protocol versions minReader from 2 to 1 and maxWriter from 5 to 3? I have set the TBL properties to 2 and 5 and columnmapping mode to rename the columns in the DeltaTable but the other users are rea...

Data Engineering
Reply
  • 18113 Views
  • 2 replies
  • 12 kudos
View Replies
Latest Reply
youssefmrini
Databricks Employee
  • 12 kudos

Unfortunately You can't downgrade the version. it's an irreversible operation.

  • 12 kudos
1 More Replies
vk217
by • Contributor
  • 5101 Views
  • 3 replies
  • 1 kudos

Resolved! ERROR: No matching distribution found for databricks-connect==7.3.34

Previously, our databricks-connect was using 7.3.34 and the builds in pipenv and the builds were successful. As of today the builds are failing with error that the version 7.3.34 no longer exists.Is there a reason this version is no longer supported....

Data Engineering
Reply
  • 5101 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 1 kudos

Hello @Vikas B​ this is the release note -https://docs.databricks.com/release-notes/dbconnect/index.htmlalso,Only the following Databricks Runtime versions are supported:Databricks Runtime 10.4 LTS ML, Databricks Runtime 10.4 LTSDatabricks Runtime 9....

  • 1 kudos
2 More Replies
User16137833804
by • Databricks Employee
  • 3663 Views
  • 3 replies
  • 1 kudos

How do I specify a version of a library different from the Databricks runtime?

Data Engineering
Reply
  • 3663 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Sebastian
Contributor
  • 1 kudos

the best solution is to store the .whl locally and do a pip install of the local whl while server boots up. this will freeze the library version. if you install from the pip it might impact your production work.

  • 1 kudos
2 More Replies
brickster_2018
by • Databricks Employee
  • 1465 Views
  • 1 replies
  • 0 kudos

Resolved! What is the trade-off of using an unsupported DBR version on my cluster?

I do not want to upgrade my cluster every one month. I am looking for stability over new features.

Data Engineering
Reply
  • 1465 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

The strong recommendation is not to use an unsupported version of DBR on your cluster. For production workloads where you don't welcome newer versions, then check the Databricks LTS DBR versions. if using an unsupported version then you don't receiv...

  • 0 kudos
User16783853906
by • Contributor III
  • 4593 Views
  • 1 replies
  • 0 kudos

Metaexception [Version information not found in metastore] during cluster [re]start

Trying to configure new external metastore and running into the following exception during cluster initialization - Caused by: MetaException(message:Version information not found in metastore. )   at org.apache.hadoop.hive.metastore.RetryingHMSHandl...

Data Engineering
Reply
  • 4593 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16783853906
Contributor III
  • 0 kudos

The above exception happens when the hive schema is not available in the metastore instance. Please check in your init scripts to make sure the following flag is enabled to create hive Schema and tables if not already present. datanucleus.autoCreateA...

  • 0 kudos
Labels