Hello! I am trying to forward fill a column in a Pandas dataframe based on a keyword. I have come up with: pdf_df['EEName_TEST'] = pdf_df['EEName_TEST'].str.contains('Name:').ffill() This gives me a boolean result but I still can't figure out what ...
I have Azure notebook which take widget input parameter and performs necessary action. But this note book should be called within another notebook using dbutils.notebook.run , How do I pass parameter to the widget?
I was mounting the Datalake Gen1 to Databricks for accessing and processing files, The below code was working great for the past 1 year and all of a sudden I'm getting an errorconfigs = {"df.adl.oauth2.access.token.provider.type": "ClientCredential"...
When using display, more than 1 spaces in strings are ignored. Can we change that behaviour? Are there any options for display functions? code example: display( spark.createDataFrame( [ ( 'a a' , 'a a' ) ], [ 'string_column', 'string_column_2' ] )...
Guys, could you help me? I'm reading 5 kafka threads through a list and saving the data in a Delta table The execution will be 1x a day, it seems that everything is working but I noticed that when I read the topic and it has no message, it still gen...
I'm operating on some data that looks like the image attached. the command that I'm performing is :<code>library(magrittr) subsetting the data for MAC-OS & sorting by event-timestamp. acDF <- eventsDF %>% SparkR::select("device", "event_timestamp...
I am trying to connect PostgreSQL from Azure Databricks. I am using the below code to connect. jdbcHostname = "Test" jdbcPort = 1234 jdbcDatabase = "Test1" jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jdbcPort, jdbcDatabase) Conn...
@Javier De La Torre do you really need two-way SSL (verify-full)? In most cases one way SSL (sslmode=require) should be enough. @akj2784​ When you say "Connection was successful", where do you mean you established a successful connection? You might...
I am trying to obtain the month and year in the format of "MM-YYY", then "YYY" to get a values such as 12-2012. I noticed an error where a timestamp of 2012-12-30T00:00:00.000+0000 results in both 12-2013 and 2013. This is an error, since 2012-12-30...
Say I have two notebooks A and B. Notebook A generates data for notebook B to process. However, I want multiple B to process the data concurrently. Is this possible?
For those of you who use databricks-connect probably know that it’s a great tool to use the power of spark/databricks, while executing/debugging code (and having proper git integration) from your favorite IDE. However, when you want to test somethin...
<br> <code>Time Message Cluster terminated.Reason:Network Configuration Failure The data plane network is misconfigured. Please verify that the network for your data plane is configured correctly. Instance ID: ............... Error mess...
Spark Standalone Cluster Configuration (Spark 3.0.0)- 1 Master2 Workers (4 cores each) I am using Airflow SparkSubmitOperator to submit the job to Spark Master in Cluster mode. There are multiple(~20) DAGs on airflow submitting jobs to Spark. These ...
Hi, I would like to provision a Databricks environment in Azure and looking at options to create a workspace, cluster, notebook using code. Could you please point me to the documentation around this. Thank you.
By "dbt" do you mean Database Transactions? If not, what do you mean?
I am attempting to use Databricks Connect with a cluster in Azure Government with a port of 443 but get the following error when running databricks-connect test.The port you specified is either being used already or invalid. Port: The port that Data...
Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!
Sign Up Now| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
