cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Digan_Parikh
by Valued Contributor
  • 748 Views
  • 1 replies
  • 0 kudos

Resolved! Widgets - Way to validate config parameters

Can you use widgets to validate config parameters for notebooks?

  • 748 Views
  • 1 replies
  • 0 kudos
Latest Reply
Digan_Parikh
Valued Contributor
  • 0 kudos

For example:folder = dbutils.widgets.get("Folder") if folder == "": raise Exception("Folder missing")or to get spark settings you can use:spark.conf.get("my_property")Learn more about them here - https://docs.databricks.com/notebooks/widgets.html

  • 0 kudos
User16826992666
by Valued Contributor
  • 1050 Views
  • 1 replies
  • 0 kudos

Can you use external job scheduling tools to start and schedule Databricks jobs?

I am wondering if I have to use the Databricks jobs scheduler to kick off Databricks jobs. My company already uses another job scheduler for our workflows and it would be useful to add our Databricks jobs to that flow.

  • 1050 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

You could use external tools to schedule jobs in Databricks. Here is a blogpost explaining how Databricks could be used along with Azure Data factory . This blog explains how to use Airflow with DatabricksIt is worth noting that a lot Databricks's f...

  • 0 kudos
Anonymous
by Not applicable
  • 4096 Views
  • 1 replies
  • 0 kudos

Resolved! Scheduling cluster start and stop time

I want to schedule cluster to start in the morning and shut down by evening. How can I achieve that?

  • 4096 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

You can call the REST API to schedule cluster starts and stops from a scheduler.See https://docs.databricks.com/dev-tools/api/latest/clusters.htmlPRO Tip: Use code generation tools within Postman to generate scripts in the language of your choice.

  • 0 kudos
User16826992666
by Valued Contributor
  • 777 Views
  • 1 replies
  • 0 kudos
  • 777 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

There shouldn't be. Generally speaking, models will be serialized according to their 'native' format for well-known libraries like Tensorflow, xgboost, sklearn, etc. Custom model will be saved with pickle. The files exist on distributed storage as ar...

  • 0 kudos
User16826992666
by Valued Contributor
  • 831 Views
  • 1 replies
  • 0 kudos

Resolved! What is the point of the model staging and promotion functions in MLflow?

Why not just directly deploy the model where you need it in production?

  • 831 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

The Model Registry is mostly a workflow tool. It helps 'gate' the process, so that (for example) only authorized users can set a model to be the newest Production version of a model - that's not something just anyone should be able to do!The Registry...

  • 0 kudos
User16826992666
by Valued Contributor
  • 1310 Views
  • 1 replies
  • 0 kudos

Resolved! Should I use Z Ordering on my Delta table every time I run Optimize?

Wondering if it always makes sense or if there are some situations where you might only want to run optimize

  • 1310 Views
  • 1 replies
  • 0 kudos
Latest Reply
Srikanth_Gupta_
Valued Contributor
  • 0 kudos

Its good idea to optimize at end of each batch job to avoid any small files situation, Z order is optional and can be applied on few non partition columns which are used frequently in read operationsZORDER BY -> Colocate column information in the sam...

  • 0 kudos
Anonymous
by Not applicable
  • 871 Views
  • 1 replies
  • 0 kudos
  • 871 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ryan_Chynoweth
Honored Contributor III
  • 0 kudos

In this scenario, the best option would be to have a single readStream reading a source delta table. Since checkpoint logs are controlled when writing to delta tables you would be able to maintain separate logs for each of your writeStreams. I would...

  • 0 kudos
User16826992666
by Valued Contributor
  • 1269 Views
  • 1 replies
  • 1 kudos

Resolved! If someone saves a flat file in a Databricks notebook without specifying a location, where does it go?

I ran the code block below and now I can't find the file. Where would this get saved since no location was specified?

Cmd
  • 1269 Views
  • 1 replies
  • 1 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

In this case, you are just writing a file to the working directory of the driver process. It'll be under something like /home/[user] on the local file system. Anything you write to "/dbfs/..." however goes to distributed storage, even though it looks...

  • 1 kudos
User16826994223
by Honored Contributor III
  • 676 Views
  • 1 replies
  • 0 kudos

Major changes in spark 3.0

What are the major changes released in spark 3.0

  • 676 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

Check out https://spark.apache.org/docs/latest/sql-migration-guide.html if you're looking for potentially breaking changes you need to be aware of, for any version.For a general overview of the new features, see https://databricks.com/blog/2020/06/18...

  • 0 kudos
User16857281869
by New Contributor II
  • 780 Views
  • 1 replies
  • 0 kudos

How do I benefit from parallelisation when doing machine learning?

There are in principle four distinct ways of using parallelisation when doing machine learning. Any combination of these can speed up the whole pipeline significantly.1) Using spark distributed processing in feature engineering 2) When the data set...

  • 780 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

Good summary! yes those are the main strategies I can think of.

  • 0 kudos
User16826992666
by Valued Contributor
  • 1006 Views
  • 2 replies
  • 0 kudos
  • 1006 Views
  • 2 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

You do not have to cache anything to make it work. You would decide that based on whether you want to spend memory/storage to avoid recomputing the DataFrame, like when you may use it in multiple operations afterwards.

  • 0 kudos
1 More Replies
User16826992666
by Valued Contributor
  • 2095 Views
  • 1 replies
  • 0 kudos

Resolved! What's the difference between SparkML and Spark MLlib?

I have heard people talk about SparkML but when reading documentation it talks about MLlib. I don't understand the difference, could anyone help me understand this?

  • 2095 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones ...

  • 0 kudos
sajith_appukutt
by Honored Contributor II
  • 2248 Views
  • 1 replies
  • 1 kudos
  • 2248 Views
  • 1 replies
  • 1 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

You could set up dnsmasq to configure  routing between your Databricks workspace and your on-premise network. More details here

  • 1 kudos
Labels