I have Azure notebook which take widget input parameter and performs necessary action. But this note book should be called within another notebook using dbutils.notebook.run , How do I pass parameter to the widget?
I was mounting the Datalake Gen1 to Databricks for accessing and processing files, The below code was working great for the past 1 year and all of a sudden I'm getting an errorconfigs = {"df.adl.oauth2.access.token.provider.type": "ClientCredential"...
When using display, more than 1 spaces in strings are ignored. Can we change that behaviour? Are there any options for display functions? code example: display( spark.createDataFrame( [ ( 'a a' , 'a a' ) ], [ 'string_column', 'string_column_2' ] )...
Guys, could you help me? I'm reading 5 kafka threads through a list and saving the data in a Delta table The execution will be 1x a day, it seems that everything is working but I noticed that when I read the topic and it has no message, it still gen...
I'm operating on some data that looks like the image attached. the command that I'm performing is :<code>library(magrittr) subsetting the data for MAC-OS & sorting by event-timestamp. acDF <- eventsDF %>% SparkR::select("device", "event_timestamp...
I am trying to connect PostgreSQL from Azure Databricks. I am using the below code to connect. jdbcHostname = "Test" jdbcPort = 1234 jdbcDatabase = "Test1" jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jdbcPort, jdbcDatabase) Conn...
@Javier De La Torre do you really need two-way SSL (verify-full)? In most cases one way SSL (sslmode=require) should be enough. @akj2784​ When you say "Connection was successful", where do you mean you established a successful connection? You might...
I am trying to obtain the month and year in the format of "MM-YYY", then "YYY" to get a values such as 12-2012. I noticed an error where a timestamp of 2012-12-30T00:00:00.000+0000 results in both 12-2013 and 2013. This is an error, since 2012-12-30...
Say I have two notebooks A and B. Notebook A generates data for notebook B to process. However, I want multiple B to process the data concurrently. Is this possible?
For those of you who use databricks-connect probably know that it’s a great tool to use the power of spark/databricks, while executing/debugging code (and having proper git integration) from your favorite IDE. However, when you want to test somethin...
<br> <code>Time Message Cluster terminated.Reason:Network Configuration Failure The data plane network is misconfigured. Please verify that the network for your data plane is configured correctly. Instance ID: ............... Error mess...
Spark Standalone Cluster Configuration (Spark 3.0.0)- 1 Master2 Workers (4 cores each) I am using Airflow SparkSubmitOperator to submit the job to Spark Master in Cluster mode. There are multiple(~20) DAGs on airflow submitting jobs to Spark. These ...
I have created a custom transformer to be used in a ml pipeline. I was able to write the pipeline to storage by extending the transformer class with DefaultParamsWritable. Reading the pipeline back in however, does not seem possible in Scala. I have...
When trying to upload to DBFS from local machine getting error as "Error occurred when processing file ... : Server responded with 0 code" DBR 7.3 LTSSpark 3.0.1 Scala 2.12 Uploading the file using the "upload" in the Databricks cloud console, the c...
Even I am facing the same issue with GCP databricks. I am able to upload files with smaller size. When i tried with 3MB file, databricks chokes. I get the above error. I tried with aws databricks, it works good even for bigger size files.
Hi, I would like to provision a Databricks environment in Azure and looking at options to create a workspace, cluster, notebook using code. Could you please point me to the documentation around this. Thank you.
By "dbt" do you mean Database Transactions? If not, what do you mean?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group| User | Count |
|---|---|
| 1610 | |
| 763 | |
| 345 | |
| 286 | |
| 251 |
