- 10161 Views
- 2 replies
- 0 kudos
Combine Python + R in data manipulation in Databricks Notebook
Want to combine Py + Rfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()# Create a sample DataFramedata = [("Alice", 25), ("Bob", 30), ("Charlie", 35), ("Oscar",36), ("Hiromi",41), ("Alejandro", ...
- 10161 Views
- 2 replies
- 0 kudos
- 0 kudos
@Oscar CENTENO MORA​ :To combine Py and R in a Databricks notebook, you can use the magics command %python and %rto switch between Python and R cells. Here's an example of how to create a Spark DataFrame in Python and then use it in R:from pyspark.sq...
- 0 kudos
- 1329 Views
- 3 replies
- 0 kudos
Performance issue while loading bulk data into Post Gress DB from data bricks.
We are facing a performance issue while loading bulk data into Postgress DB from data bricks. We are using spark JDBC connections to move the data. However, the rate of transfer is very low which is causing performance bottleneck. is there any better...
- 1329 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @Janga Reddy​ @Daniel Sahal​ and @Vidula Khanna​ ,To enhance performance in general we need to design for more parallelism, in Spark JDBC context this controlled by the number of partitions for the data to be writtenThe example here shows how t...
- 0 kudos
- 2509 Views
- 2 replies
- 2 kudos
Resolved! I am new to data bricks. setting up Workspace for NON-prod environment Separate workspaces for DEV, QA or Just one work space for NON-prod ?
What i learned based on learning materials, documents, etc.. For data bricks it is a good practice to set up 1 non-prod workspace but separate clusters for Dev, QA, SIT, etc.Is it best practice to set up only 1 NON-PROD Workspace instead of separate ...
- 2509 Views
- 2 replies
- 2 kudos
- 909 Views
- 1 replies
- 5 kudos
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and...
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and reliable platform for users to build and deploy their applications. As part of this release, the en...
- 909 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi @Hubert Dudek​,We express appreciation for the informative content you've contributed to our community.Your posts have sparked engaging discussions and proven invaluable resources for our members.You've truly made a difference in our community, an...
- 5 kudos
- 875 Views
- 1 replies
- 2 kudos
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Ma...
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to shar...
- 875 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Lu Wang​, Thank you for your outstanding efforts in sharing important announcements within our community. Your proactive approach to communication has helped maintain a well-informed and cohesive work environment for our team.Your dedication to k...
- 2 kudos
- 1199 Views
- 1 replies
- 6 kudos
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and...
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and reliable platform for users to build and deploy their applications. As part of this release, the en...
- 1199 Views
- 1 replies
- 6 kudos
- 6 kudos
Thank you for sharing @Hubert Dudek​ !!!
- 6 kudos
- 1277 Views
- 1 replies
- 7 kudos
Starting from #databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function ...
Starting from #databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function takes an array column as input and returns a new row for each element in the array, offering new pos...
- 1277 Views
- 1 replies
- 7 kudos
- 7 kudos
Thank you for sharing @Hubert Dudek​
- 7 kudos
- 3670 Views
- 1 replies
- 3 kudos
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:45-5:00 pm PT.We'll have Dave M...
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:45-5:00 pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to sha...
- 3670 Views
- 1 replies
- 3 kudos
- 3 kudos
Looking forward.
- 3 kudos
- 5054 Views
- 2 replies
- 2 kudos
Resolved! Connect GraphQL to Data Bricks
Hi I am new to Databricks however I need to expose data found in the delta lake directly to GraphQL to be queried by several applications. Is there a connector or something similar to GraphQL that works with Databricks?
- 5054 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Andrew Camilleri​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 2 kudos
- 884 Views
- 1 replies
- 5 kudos
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...
- 884 Views
- 1 replies
- 5 kudos
- 5 kudos
Thanks for sharing this with the Databricks community.
- 5 kudos
- 1258 Views
- 2 replies
- 5 kudos
"Hey everyone, it seems like there's some confusion about enhanced autoscaling in Databricks lately. If you're feeling lost or unsure abo...
"Hey everyone, it seems like there's some confusion about enhanced autoscaling in Databricks lately. If you're feeling lost or unsure about how it works, don't worry - you're not"Enhanced autoscaling is a feature in Databricks that enables dynamic sc...
- 1258 Views
- 2 replies
- 5 kudos
- 2838 Views
- 2 replies
- 0 kudos
Overwriting the existing table in Databricks; Mechanism and History?
Hi,Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table. (DF.write .format("delta") .mode("overwrite") .option("...
- 2838 Views
- 2 replies
- 0 kudos
- 0 kudos
the overwrite will add new files, keep the old ones and in a log keeps track of what is current data and what is old data.If the overwrite fails, you will get an error message in the spark program, and the data to be overwritten will still be the cur...
- 0 kudos
- 675 Views
- 1 replies
- 5 kudos
Exciting news for Databricks users! #databricks launched a new feature that allows users to run job workflows continuously. Setting up a continuous jo...
Exciting news for Databricks users! #databricks launched a new feature that allows users to run job workflows continuously. Setting up a continuous job workflow is straightforward: create a job and select the continuous trigger option in the scheduli...
- 675 Views
- 1 replies
- 5 kudos
- 5 kudos
Thank you for sharing!!!
- 5 kudos
- 1681 Views
- 3 replies
- 7 kudos
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...
- 1681 Views
- 3 replies
- 7 kudos
- 1812 Views
- 1 replies
- 6 kudos
Variable explorer in Databricks With Databricks Runtime 12.1 and above, you can directly observe current Python variables in the notebook UI.To open t...
Variable explorer in DatabricksWith Databricks Runtime 12.1 and above, you can directly observe current Python variables in the notebook UI.To open the variable explorer, click in the right sidebar. The variable explorer opens, showing the value and ...
- 1812 Views
- 1 replies
- 6 kudos
- 6 kudos
Thank you for sharing
- 6 kudos