cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Dataengineer_mm
by • New Contributor
  • 1373 Views
  • 2 replies
  • 0 kudos

Databricks workflow migration to higher environments

How do we migrate the databricks workflows to higher environment ? I do see an option for calling the tasks (notebooks,python) from the github repositories. But as such how do we migrate the entire workflow jobs to other environment ?

Data Engineering
Reply
  • 1373 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Hi @Menaka Murugesan​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 0 kudos
1 More Replies
MachuPichu
by • New Contributor III
  • 5132 Views
  • 9 replies
  • 4 kudos

Copying Delta table from QA to Prod

I am looking for a way to copy large managed Delta table like 4TB from one environment (QA) to other(Prod) . QA and Prod are in different subscription and in different region. I understand Databricks provides a way to clone table. But I am not sure i...

Data Engineering
Reply
  • 5132 Views
  • 9 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

I would use a data factory to copy 4TB files as it has gigantic throughput. After completing a copy of everything, I would register as a table in the new metastore.

  • 4 kudos
8 More Replies
rgb
by • New Contributor
  • 688 Views
  • 0 replies
  • 0 kudos

Migration_pipeline.py failing to get default credentials

cat ~/.databrickscfg looks like this (with the correct token/host values in place of xxxxxx)[DEFAULT]host = xxxxxxtoken = xxxxxxjobs-api-version = 2.0The command I run to start the pipeline with default configured credentials is :sudo python3 migrati...

databrickserror
Data Engineering
Reply
  • 688 Views
  • 0 replies
  • 0 kudos
vk217
by • Contributor
  • 706 Views
  • 1 replies
  • 0 kudos

Databricks migration from 7.3 LTS to X.x

We are currently on 7.3 LTS version with Python 3.7. I see that we are several versions off the latest 11.1 release https://docs.databricks.com/release-notes/runtime/releases.html.I see that the End of support for 11.1 is earlier than 10.4 LTS. What ...

Data Engineering
Reply
  • 706 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Vikas B​, The following release notes provide information about Databricks Runtime 11.1, powered by Apache Spark™ 3.3.0. Databricks released these images in July 2022.

  • 0 kudos
sawya
by • New Contributor II
  • 1941 Views
  • 3 replies
  • 0 kudos

Migrate workspaces to another AWS account

Hi everyone,I have a Databricks workspace in an AWS account that I have to migrate to a new AWS accountDo you know how I can do it ? Or it's better to recreate a new one and move all the workbooks and if I choose to create one new how can you export ...

Data Engineering
Reply
  • 1941 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Abishek
Valued Contributor
  • 0 kudos

@AMADOU THIOUNE​ Can you check the below link to export the run jobs? https://docs.databricks.com/jobs.html#export-job-runs. Try to reuse the same job_id with the /update and /reset endpoints, it should allow you much better access to previous run re...

  • 0 kudos
2 More Replies
Direo
by • Contributor
  • 7064 Views
  • 3 replies
  • 2 kudos

Default indentation for Python has changed after migration to the new workspace

In our old workspace default identation was 2 spaces. In our new one it has changed to 4 spaces. Of course you can manually change it back to 2 spaces as we used to have, but it does not work. Does anyone know how to solve this issue?

Data Engineering
Reply
  • 7064 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
ranged_coop
Valued Contributor II
  • 2 kudos

You do have that option of Settings --> User Settings (Admin Settings ? not sure - I don't have admin access) --> Notebook Settings --> Default indentation for Python cells (in spaces)This will change the indentation for newer cells, but existing one...

  • 2 kudos
2 More Replies
Lonnie
by • New Contributor
  • 1745 Views
  • 1 replies
  • 1 kudos

Recommended Redshift-2-Delta Migration Path

Hello All!My team is previewing Databricks and are contemplating the steps to take to perform one-time migrations of datasets from Redshift to Delta. Based on our understandings of the tool, here are our initial thoughts:Export data from Redshift-2-S...

Data Engineering
Reply
  • 1745 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Awesome!

  • 1 kudos
saniafatimi
by • New Contributor II
  • 2626 Views
  • 2 replies
  • 1 kudos

How to migrate power bi reports to databricks

I have a sample set of power bi(.pbix) reports with all dropdowns, tables, filters etc. Now I would like to migrate this reports to data bricks. whatever visuals are created in power bi, I would like to create same in data bricks from scratch. I wou...

Data Engineering
Reply
  • 2626 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Neeljy
New Contributor II
  • 1 kudos

We must ensure you are sure that the Databricks cluster is operational. These are the steps needed for integration between Azure Databricks into Power BI Desktop.1. Constructing the URL for the connectionConnect to the cluster, and click the Advanced...

  • 1 kudos
1 More Replies
Soma
by • Valued Contributor
  • 2399 Views
  • 7 replies
  • 0 kudos

Resolved! Queries regarding workspace Migration to Premium

We are planning to migrate from standard to premium workspaceWe need to know if below artifacts will be maintainedneed to check on streaming Job DowntimeAccess token DBFS Access Production Cluster /JobsCluster ID Job ID and other properties like URL ...

Data Engineering
Reply
  • 2399 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Soma
Valued Contributor
  • 0 kudos

hi @Kaniz Fatma​  then I can assume there wont be any impact on metastore and all the metadata(table definition,schema ) will be available post upgradation

  • 0 kudos
6 More Replies
User16789201666
by • Contributor II
  • 1250 Views
  • 2 replies
  • 0 kudos

What are some guidelines for migrating to DBR 7/Spark 3?

What are some guidelines for migrating to DBR 7/Spark 3?

Data Engineering
Reply
  • 1250 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

Please refer to the below reference for switching to DBR 7.xWe have extended our DBR 6.4 support until December 2021, DBR 6.4 extended support - Release notes: https://docs.databricks.com/release-notes/runtime/6.4x.htmlMigration guide to DBR 7.x: htt...

  • 0 kudos
1 More Replies
User16826994223
by • Honored Contributor III
  • 1148 Views
  • 1 replies
  • 0 kudos

What is databricks Sync

I am trying to migrate my workload to another workspace ( from ST to E2), I am planning to use data bricks sync, but still I am not sure, will it migrate everything like , currents, user , groups, job, notebook etc or has some limitations which I s...

Data Engineering
Reply
  • 1148 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

Here is the support matrix for import/export operations for databricks-syncAlso checkout https://github.com/databrickslabs/migrate

  • 0 kudos
Labels