- 447 Views
- 1 replies
- 0 kudos
NoClassDefFoundError: org/apache/spark/sql/SparkSession$ at com.microsoft.nebula.common.ConfigProvider.<init>(configProvider.scala:17) at $linef37a348949c145718a08f6b29642317b35.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$...
- 447 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @bhanuteja_1 ,
Where are you running this from? Based on the short output, it looks like from a Databricks Notebook, but it would be a weird error unless you're having some classpath overrides or jar conflicts, leading to this error; it is simply ...
- 6072 Views
- 7 replies
- 19 kudos
I see that spark fully supports Scala 2.13. I wonder why is there no databricks runtime with Scala 2.13 yet. Any plans on making this available? It would be super useful.
- 6072 Views
- 7 replies
- 19 kudos
Latest Reply
I agree with @777. As Scala 3 is getting mature and there are more real use cases with Scala 3 on Spark now, support for Scala 2.13 will be valuable to users including us.I think the recent upgrade of Databricks runtime from JDK 8 to 17 was one of a ...
6 More Replies
- 1907 Views
- 3 replies
- 0 kudos
Hello everyone,I have a very basic question wrt Databricks spark parallelism.I have a python function within a for loop, so I believe this is running sequentially.Databricks cluster is enabled with Photon and with Spark 15x, does that mean the driver...
- 1907 Views
- 3 replies
- 0 kudos
- 6975 Views
- 3 replies
- 2 kudos
I am teaching a class for BYU Idaho and every table in every database has been imploded for my class. We keep getting this error:com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: com.databricks.rpc.UnknownRemoteException: ...
- 6975 Views
- 3 replies
- 2 kudos
Latest Reply
i am facing the issue before i trying to execute a code error message:com.databricks.rpc.UnknownRemoteException: Remote exception occurred:
2 More Replies
- 1707 Views
- 1 replies
- 2 kudos
Free Training: Databricks Lakehouse FundamentalsThe demand for technology roles is only growing – it's projected that over 150 million jobs will be added in the next five years. Across industries and regions, this is translating to increased demand f...
- 1707 Views
- 1 replies
- 2 kudos
Latest Reply
I watched all 4 videos but getting an error when I try to take the test. How do I complete the test and get my badge?
- 1679 Views
- 7 replies
- 3 kudos
We are trying to connect to AWS RDS MySQL instance from DBX with PySpark using JDBC: jdbc_df = (spark.read.format("jdbc").options(url=f"jdbc:mysql://{creds['host']}:{creds['port']}/{creds['database']}", driver="com.mysql.cj.jdbc.Driver", dbtable="(SE...
- 1679 Views
- 7 replies
- 3 kudos
Latest Reply
@Gaurav_Lokhande With Spark JDBC usage, connectivity happens between your Databricks VPC (in your AWS account) and RDS VPC, assuming you are using non-serverless clusters. You may need to ensure this connectivity works (like by peering).
6 More Replies
- 414 Views
- 1 replies
- 0 kudos
I have deployed a new databricks environment for development. I've copied a workflow from production to this environment with exactly the same compute configuration. Four notebooks that complete within minutes do not complete after 2 hours in develop...
- 414 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @trentlglover,
It sounds like you're experiencing a significant performance issue with your notebooks in the new development environment. Here are a few potential areas to investigate:
Cluster Configuration: Even though you mentioned that the comp...
- 341 Views
- 0 replies
- 0 kudos
Hello, I am new in databricks and related to data engineering. I am running a POC to sync data between a Salesforce sandbox and Databricks using LakeFlow connect.I already make the connection and i successfully sync data between salesforce and databr...
- 341 Views
- 0 replies
- 0 kudos
- 12638 Views
- 7 replies
- 3 kudos
Hi Team, I can see logs in Databricks console by navigating workflow -> job name -> logs. These logs are very generic like stdout, stderr and log4-avtive.log. How to download event, driver, and executor logs at once for a job? Regards,Rajesh.
- 12638 Views
- 7 replies
- 3 kudos
Latest Reply
@Kaniz Fatma @John Lourdu @Vidula Khanna Hi Team,I managed to download logs using the Databricks command line as below: Installed the Databricks command line on my Desktop (pip install databricks-cli)Configured the Databricks cluster URL and perso...
6 More Replies
by
hkmodi
• New Contributor II
- 1187 Views
- 3 replies
- 0 kudos
I have created an autoloader job that reads data from S3 (files with no extension) having json using (cloudFiles.format, text). Now this job is suppose to run every 4 hours and read all the new data that arrived. But before writing into a delta table...
- 1187 Views
- 3 replies
- 0 kudos
Latest Reply
HI @hkmodi ,Basically, as @daniel_sahal said, bronze layer should reflect the source system. The silver layer is dedicated for deduplication/cleaning/enrichment of dataset. If you still need to deduplicate at bronze layer you have 2 options:- use me...
2 More Replies
- 5027 Views
- 3 replies
- 1 kudos
py4j.security.Py4JSecurityException: Method public com.databricks.backend.daemon.dbutils.DBUtilsCore$Result com.databricks.backend.daemon.dbutils.DBUtilsCore.mount(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.util.Map) is ...
- 5027 Views
- 3 replies
- 1 kudos
Latest Reply
Hi , have you sorted this issue ? can you please let me know the solution?
2 More Replies
- 1369 Views
- 6 replies
- 0 kudos
I have a dataframe containing several columns among which 1 contains, for one specific record, just a comma, nothing else.When displaying the dataframe with the commanddisplay(df_input.where(col("erp_vendor_cd") == 'B6SA-VEN0008838')) The data is dis...
- 1369 Views
- 6 replies
- 0 kudos
- 360 Views
- 1 replies
- 0 kudos
Hi!I am struggling to understand how to properly manage my table to make queries effective. My table has columns date_time_utc, car_id, car_owner etc. date_time_utc, car_id and position is usually the ZORDER or Liquid Clustering-columns.Selecting max...
- 360 Views
- 1 replies
- 0 kudos
Latest Reply
1. According to the databricks yes But as always, I recommend to perform benchamarks yourself. There a lot of blog posts, that are saying that it's not alway the case. Yesterday, I was at data community event and presenter did several benchmark and ...
- 811 Views
- 2 replies
- 0 kudos
Hello,I'm creating a DLT pipeline with Databricks on AWS. After creating an external location for my bucket, I encountered the following error:DataPlaneException: [DLT ERROR CODE: CLUSTER_LAUNCH_FAILURE.CLIENT_ERROR] Failed to launch pipeline cluster...
- 811 Views
- 2 replies
- 0 kudos
Latest Reply
Hi!The error was related to the roles and permissions created when the workspace was set up. I reloaded the setup script in a new workspace, and it worked without problems.Hope it helps anyone in the future.Thanks!
1 More Replies
- 670 Views
- 1 replies
- 1 kudos
Hi!We have a lot use cases were we need to load data from Snowflake into Databricks, where users are using both R and Python for further analysis and machine learning. For this we have been using Lakehouse Federation combined with basic auth, but are...
- 670 Views
- 1 replies
- 1 kudos
Latest Reply
For anyone interested: We solved this by building an OAuth integration to Snowflake ourselfs using Entra ID: https://community.snowflake.com/s/article/External-oAuth-Token-Generation-using-Azure-ADWe also created some simple Python and R-packages tha...
