cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

brickster_2018
by Esteemed Contributor
  • 1250 Views
  • 1 replies
  • 0 kudos
  • 1250 Views
  • 1 replies
  • 0 kudos
Latest Reply
aladda
Honored Contributor II
  • 0 kudos

You could potentially do this through a Global Init Script - https://docs.databricks.com/clusters/init-scripts.html

  • 0 kudos
User16790091296
by Contributor II
  • 2529 Views
  • 3 replies
  • 0 kudos
  • 2529 Views
  • 3 replies
  • 0 kudos
Latest Reply
Mooune_DBU
Valued Contributor
  • 0 kudos

Dy doing a `GET` call using the cluster idcurl --netrc -X GET \ https://dbc-a1b2345c-d6e7.cloud.databricks.com/api/2.0/clusters/get \ --data '{ "cluster_id": "1234-567890-myclustID" }' \ | jq .The response json will have a `state` tag which will look...

  • 0 kudos
2 More Replies
User16826992666
by Valued Contributor
  • 1267 Views
  • 1 replies
  • 0 kudos

Can I move some partitions of a Delta table to a different location?

I am partitioning my Delta table by date. Older data is rarely accessed, so I am wondering if I can move some of the files off to colder storage options. What would happen if I did this? Is this a supported pattern or would it break the table?

  • 1267 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

You could look at S3 Intelligent-Tiering - https://aws.amazon.com/about-aws/whats-new/2018/11/s3-intelligent-tiering/

  • 0 kudos
brickster_2018
by Esteemed Contributor
  • 1559 Views
  • 1 replies
  • 0 kudos
  • 1559 Views
  • 1 replies
  • 0 kudos
Latest Reply
aladda
Honored Contributor II
  • 0 kudos

Deleting the Delta log directory would cause you to lose the underlying transaction history on the delta table and other delta related optimizations. In effect the table would be converted to a Parquet table at that point

  • 0 kudos
brickster_2018
by Esteemed Contributor
  • 1235 Views
  • 1 replies
  • 0 kudos
  • 1235 Views
  • 1 replies
  • 0 kudos
Latest Reply
aladda
Honored Contributor II
  • 0 kudos

Global: run on every cluster in the workspace. They can help you to enforce consistent cluster configurations across your workspace. Use them carefully because they can cause unanticipated impacts, like library conflicts. Only admin users can create ...

  • 0 kudos
MoJaMa
by Valued Contributor II
  • 1046 Views
  • 1 replies
  • 0 kudos
  • 1046 Views
  • 1 replies
  • 0 kudos
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

Yes.Currently supported paths are:BYO VPC -> New BYO VPCDatabricks-created VPC -> New Databricks-created VPC

  • 0 kudos
brickster_2018
by Esteemed Contributor
  • 775 Views
  • 1 replies
  • 1 kudos
  • 775 Views
  • 1 replies
  • 1 kudos
Latest Reply
aladda
Honored Contributor II
  • 1 kudos

For Databricks Runtime 5.5 LTS, Spark jobs, Python notebook cells, and library installation all support both Python 2 and 3.The default Python version for clusters created using the UI is Python 3. In Databricks Runtime 5.5 LTS the default version fo...

  • 1 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels