cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Anonymous
by • Not applicable
  • 1717 Views
  • 0 replies
  • 0 kudos

Seeing all columns

I have a dataframe with a lot of columns (20 or so) and 8 rows. Part of the output is being cutoff and I can scroll to the right to see the rest of the columns but I was just wondering if it was possible to somehow "zoom out" of the table so I can se...

Data Engineering
Reply
  • 1717 Views
  • 0 replies
  • 0 kudos
MallikSunkara
by • New Contributor II
  • 9092 Views
  • 4 replies
  • 0 kudos

how to pass arguments and variables to databricks python activity from azure data factory

how to pass arguments and variables to databricks python activity from azure data factory

Data Engineering
Reply
  • 9092 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
CristianIspan
New Contributor II
  • 0 kudos

try importing argv from sys. Then if you have the parameter added correctly in DataFactory you could get it in your python script typing argv[1] (index 0 is the file path).

  • 0 kudos
3 More Replies
User16790091296
by • Contributor II
  • 1210 Views
  • 0 replies
  • 0 kudos

docs.databricks.com

What is Databricks Database?A Databricks database is a collection of tables. A Databricks table is a collection of structured data. You can cache, filter, and perform any operations supported by Apache Spark DataFrames on Databricks tables. You can q...

Data Engineering
Reply
  • 1210 Views
  • 0 replies
  • 0 kudos
Joseph_B
by • Databricks Employee
  • 2431 Views
  • 1 replies
  • 0 kudos

How can I implement a custom graph algorithm on top of Apache Spark? Can I use GraphX or GraphFrames?

Data Engineering
Reply
  • 2431 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Joseph_B
Databricks Employee
  • 0 kudos

You can implement custom algorithms for GraphFrames using either Scala/Java or Python APIs. GraphFrames provides some structures to simplify writing graph algorithms; the three primary options are as follow, with the best options first:Pregel: This i...

  • 0 kudos
User15725630784
by • Databricks Employee
  • 2125 Views
  • 1 replies
  • 0 kudos

Spark JDBC query isn't working for Oracle Databases

I am trying to read with the following syntaxval df = spark.read .format("jdbc") .option("url", "<url>") .option("query", "SELECT * FROM oracle_test_table)") .option("user", "<user>") .option("password", "<password>") .option("driver", "oracle...

Data Engineering
Reply
  • 2125 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User15725630784
Databricks Employee
  • 0 kudos

https://kb.databricks.com/data-sources/query-option-not-work-oracle.html#problem-apache-spark-jdbc-datasource-query-option-doesnt-work-for-oracle-database

  • 0 kudos
User16790091296
by • Contributor II
  • 2917 Views
  • 1 replies
  • 0 kudos

When do I start getting charged for a DBU? Am I charged for cluster spin up time?

Data Engineering
Reply
  • 2917 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16790091296
Contributor II
  • 0 kudos

Databricks starts to charge for DBUs once the virtual machine is up and the Spark context is initialized, which may include a portion of start up costs, but not all. Init scripts are loaded before the Spark context is initialized, which therefore wou...

  • 0 kudos
User16790091296
by • Contributor II
  • 1319 Views
  • 1 replies
  • 0 kudos

Does the price increase as I attach more notebooks to the same cluster?

Databricks pricing related question - Do I consume more DBUs when I attach more notebooks to the same cluster?

Data Engineering
Reply
  • 1319 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16790091296
Contributor II
  • 0 kudos

Hey PJ, short answer is - No, attaching more notebooks does not increase the price of the cluster, which is solely based on compute power. Attaching more notebooks to the cluster is a value-add of the platform.If you're interested, you can find some ...

  • 0 kudos
saninanda
by • New Contributor II
  • 13542 Views
  • 7 replies
  • 0 kudos

how to read schema from text file stored in cloud storage

I have file a.csv or a.parquet while creating data frame reading we can explictly define schema with struct type. instead of write the schema in the notebook want to create schema lets say for all my csv i have one schema like csv_schema and stored ...

Data Engineering
Reply
  • 13542 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Nakeman
New Contributor II
  • 0 kudos

@shyampsr big thanks, was searching for the solution almost 3 hours _https://luckycanadian.com/

  • 0 kudos
6 More Replies
Anonymous
by • Not applicable
  • 1414 Views
  • 0 replies
  • 0 kudos

foreachBatch in pyspark throwing OSError: [WinError 10022] An invalid argument was supplied

Hello Team,Since last 3 weeks I am trying to move my project from batch to structure streaming.But every time I am running my code I am getting below error:Traceback (most recent call last): File "C:\Users\avisriva1\git_4May2021\comint-ml-scores\src\...

Data Engineering
Reply
  • 1414 Views
  • 0 replies
  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels