cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Ram443
by • New Contributor III
  • 25653 Views
  • 9 replies
  • 5 kudos

Resolved! I created a data frame but was not able to see the data

Code to create a data frame:from pyspark.sql import SparkSessionspark=SparkSession.builder.appName("oracle_queries").master("local[4]")\  .config("spark.sql.warehouse.dir", "C:\\softwares\\git\\pyspark\\hive").getOrCreate()from pyspark.sql.functions ...

Data Engineering
Reply
  • 25653 Views
  • 9 replies
  • 5 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 5 kudos

@ramanjaneyulu kancharla​  can you please select my answer as best answer

  • 5 kudos
8 More Replies
Lazloo
by • New Contributor III
  • 901 Views
  • 0 replies
  • 2 kudos

Cannot load spark-avro jars with databricksversion 10.4

Currently, I am facing an issue since the `databricks-connect` runtime on our cluster was updated to 10.4. Since then, I cannot load the jars for spark-avro anymore. By Running the following code from pyspark.sql import SparkSession   spark = SparkSe...

Data Engineering
Reply
  • 901 Views
  • 0 replies
  • 2 kudos
Chris_Shehu
by • Valued Contributor III
  • 5333 Views
  • 2 replies
  • 10 kudos

Resolved! When trying to use pyodbc connector to write files to SQL server receiving error. java.lang.ClassNotFoundException Any alternatives or ways to fix this?

jdbcUsername = ******** jdbcPassword = *************** server_name = "jdbc:sqlserver://***********:******" database_name = "********" url = server_name + ";" + "databaseName=" + database_name + ";"   table_name = "PatientTEST"   try: df.write \ ...

Data Engineering
Reply
  • 5333 Views
  • 2 replies
  • 10 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 10 kudos

please check following code:df.write.jdbc( url="jdbc:sqlserver://<host>:1433;database=<db>;user=<user>;password=<password>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;driver=com.microsof...

  • 10 kudos
1 More Replies
TJS
by • New Contributor II
  • 15404 Views
  • 6 replies
  • 5 kudos

Resolved! Can you help with this error please? Issue when using a new high concurrency cluster

Hello, I am trying to use MLFlow on a new high concurrency cluster but I get the error below. Does anyone have any suggestions? It was working before on a standard cluster. Thanks.py4j.security.Py4JSecurityException: Method public int org.apache.spar...

Data Engineering
Reply
  • 15404 Views
  • 6 replies
  • 5 kudos
View Replies
Latest Reply
User16753724828
New Contributor III
  • 5 kudos

@Tom Soto​ We have a workaround for this. This cluster spark configuration setting will disable py4jSecurity while still enabling passthrough spark.databricks.pyspark.enablePy4JSecurity false

  • 5 kudos
5 More Replies
Labels