I'm operating on some data that looks like the image attached. the command that I'm performing is :<code>library(magrittr) subsetting the data for MAC-OS & sorting by event-timestamp. acDF <- eventsDF %>% SparkR::select("device", "event_timestamp...
I am trying to connect PostgreSQL from Azure Databricks. I am using the below code to connect. jdbcHostname = "Test" jdbcPort = 1234 jdbcDatabase = "Test1" jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jdbcPort, jdbcDatabase) Conn...
@Javier De La Torre do you really need two-way SSL (verify-full)? In most cases one way SSL (sslmode=require) should be enough. @akj2784​ When you say "Connection was successful", where do you mean you established a successful connection? You might...
I am trying to obtain the month and year in the format of "MM-YYY", then "YYY" to get a values such as 12-2012. I noticed an error where a timestamp of 2012-12-30T00:00:00.000+0000 results in both 12-2013 and 2013. This is an error, since 2012-12-30...
Say I have two notebooks A and B. Notebook A generates data for notebook B to process. However, I want multiple B to process the data concurrently. Is this possible?
For those of you who use databricks-connect probably know that it’s a great tool to use the power of spark/databricks, while executing/debugging code (and having proper git integration) from your favorite IDE. However, when you want to test somethin...
<br> <code>Time Message Cluster terminated.Reason:Network Configuration Failure The data plane network is misconfigured. Please verify that the network for your data plane is configured correctly. Instance ID: ............... Error mess...
Spark Standalone Cluster Configuration (Spark 3.0.0)- 1 Master2 Workers (4 cores each) I am using Airflow SparkSubmitOperator to submit the job to Spark Master in Cluster mode. There are multiple(~20) DAGs on airflow submitting jobs to Spark. These ...
Hi, I would like to provision a Databricks environment in Azure and looking at options to create a workspace, cluster, notebook using code. Could you please point me to the documentation around this. Thank you.
By "dbt" do you mean Database Transactions? If not, what do you mean?
I am attempting to use Databricks Connect with a cluster in Azure Government with a port of 443 but get the following error when running databricks-connect test.The port you specified is either being used already or invalid. Port: The port that Data...
Hi everybody. Looks like EXISTS statement works incorrectly. If i execute the following statement in SQL Server it returns one row, as it should WITH a AS ( SELECT '1' AS id, 'Super Company' AS name UNION SELECT '2' AS id, 'SUPER COMPANY...
In newer versions of spark it's possible to use ANTI JOIN and SEMI JOIN It looks this way:WITH a AS ( SELECT '1' AS id, 'Super Company' AS name UNION SELECT '2' AS id, 'SUPER COMPANY' AS name ), b AS ( SELECT 'a@b.com' AS user_username, 'Super Co...
Hi team, New to Databricks and trying to understand if there is a "True" auto-start capability with Databricks. We are evaluating Databricks Delta lake as an alternative cloud based datawarehouse solution but the biggest problem I see is the inabili...
Just adding on to this. Using DBeaver as a client and using a singlenode cluster and a pool of idling VM, it was possible to get the autostart time of the cluster down to 35 seconds, + 17 seconds for the query time on top to show the first 200 rows ...
Hi there, Trying to decide if I am going to get started with ml and really enjoyed it so far. When going through the documentation, there was a blocker moment for me, as I feel the documentation doesn't mention much about the dataset used to train t...
I am working on pandas and python.After processing a particular dataframe in my program , I am appending that dataframe below an existing Excel file. Now problem is my excel has font size of 11 pt but dataframe has font size of 12 pt.I want to set f...
HiI'm loading df from redis using this code:df = (spark.read.format("org.apache.spark.sql.redis") .option("table", f"state_store_ready_to_sell") .option("key.column", "msid").option("infer.schema", "true").load()and then i'm running f...
Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!
Sign Up Now| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
