cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
User16790091296
by • Databricks Employee
  • 3875 Views
  • 1 replies
  • 0 kudos

Resolved! How can I use a Python function defined in my git-repo module within the DB notebook?

I have a function within a module in my git-repo. I want to import that to my DB notebook - how can I do that?

Data Engineering
Reply
  • 3875 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aladda
Databricks Employee
  • 0 kudos

Databricks Repos allows you to sync your work in Databricks with a remote Git repository. This makes it easier to implement development best practices. Databricks supports integrations with GitHub, Bitbucket, and GitLab. Using Repos you can bring you...

  • 0 kudos
Anonymous
by • Not applicable
  • 2194 Views
  • 0 replies
  • 0 kudos

Seeing all columns

I have a dataframe with a lot of columns (20 or so) and 8 rows. Part of the output is being cutoff and I can scroll to the right to see the rest of the columns but I was just wondering if it was possible to somehow "zoom out" of the table so I can se...

Data Engineering
Reply
  • 2194 Views
  • 0 replies
  • 0 kudos
MallikSunkara
by • New Contributor II
  • 10976 Views
  • 4 replies
  • 0 kudos

how to pass arguments and variables to databricks python activity from azure data factory

how to pass arguments and variables to databricks python activity from azure data factory

Data Engineering
Reply
  • 10976 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
CristianIspan
Databricks Partner
  • 0 kudos

try importing argv from sys. Then if you have the parameter added correctly in DataFactory you could get it in your python script typing argv[1] (index 0 is the file path).

  • 0 kudos
3 More Replies
User16790091296
by • Databricks Employee
  • 2152 Views
  • 0 replies
  • 0 kudos

docs.databricks.com

What is Databricks Database?A Databricks database is a collection of tables. A Databricks table is a collection of structured data. You can cache, filter, and perform any operations supported by Apache Spark DataFrames on Databricks tables. You can q...

Data Engineering
Reply
  • 2152 Views
  • 0 replies
  • 0 kudos
Joseph_B
by • Databricks Employee
  • 3956 Views
  • 1 replies
  • 0 kudos

How can I implement a custom graph algorithm on top of Apache Spark? Can I use GraphX or GraphFrames?

Data Engineering
Reply
  • 3956 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Joseph_B
Databricks Employee
  • 0 kudos

You can implement custom algorithms for GraphFrames using either Scala/Java or Python APIs. GraphFrames provides some structures to simplify writing graph algorithms; the three primary options are as follow, with the best options first:Pregel: This i...

  • 0 kudos
User15725630784
by • Databricks Employee
  • 3405 Views
  • 1 replies
  • 0 kudos

Spark JDBC query isn't working for Oracle Databases

I am trying to read with the following syntaxval df = spark.read .format("jdbc") .option("url", "<url>") .option("query", "SELECT * FROM oracle_test_table)") .option("user", "<user>") .option("password", "<password>") .option("driver", "oracle...

Data Engineering
Reply
  • 3405 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User15725630784
Databricks Employee
  • 0 kudos

https://kb.databricks.com/data-sources/query-option-not-work-oracle.html#problem-apache-spark-jdbc-datasource-query-option-doesnt-work-for-oracle-database

  • 0 kudos
User16790091296
by • Databricks Employee
  • 4612 Views
  • 1 replies
  • 0 kudos

When do I start getting charged for a DBU? Am I charged for cluster spin up time?

Data Engineering
Reply
  • 4612 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16790091296
Databricks Employee
  • 0 kudos

Databricks starts to charge for DBUs once the virtual machine is up and the Spark context is initialized, which may include a portion of start up costs, but not all. Init scripts are loaded before the Spark context is initialized, which therefore wou...

  • 0 kudos
User16790091296
by • Databricks Employee
  • 2178 Views
  • 1 replies
  • 0 kudos

Does the price increase as I attach more notebooks to the same cluster?

Databricks pricing related question - Do I consume more DBUs when I attach more notebooks to the same cluster?

Data Engineering
Reply
  • 2178 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16790091296
Databricks Employee
  • 0 kudos

Hey PJ, short answer is - No, attaching more notebooks does not increase the price of the cluster, which is solely based on compute power. Attaching more notebooks to the cluster is a value-add of the platform.If you're interested, you can find some ...

  • 0 kudos
saninanda
by • New Contributor II
  • 20443 Views
  • 7 replies
  • 0 kudos

how to read schema from text file stored in cloud storage

I have file a.csv or a.parquet while creating data frame reading we can explictly define schema with struct type. instead of write the schema in the notebook want to create schema lets say for all my csv i have one schema like csv_schema and stored ...

Data Engineering
Reply
  • 20443 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Nakeman
New Contributor II
  • 0 kudos

@shyampsr big thanks, was searching for the solution almost 3 hours _https://luckycanadian.com/

  • 0 kudos
6 More Replies
Labels