- 5668 Views
- 11 replies
- 8 kudos
Resolved! Facing issues in running the converted code in spark sql framework with 5 to 10 percent volume of prod data. Need help in solving this and suggestions required.
Hi All.. Need your help in this issue what i am facing. Currently we are using data bricks as a platform to build pipeline and execute our talend ETL sqls converted into the spark sql framework as we were facing issues in loading the history data int...
- 5668 Views
- 11 replies
- 8 kudos
- 11932 Views
- 2 replies
- 0 kudos
Combine Python + R in data manipulation in Databricks Notebook
Want to combine Py + Rfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()# Create a sample DataFramedata = [("Alice", 25), ("Bob", 30), ("Charlie", 35), ("Oscar",36), ("Hiromi",41), ("Alejandro", ...
- 11932 Views
- 2 replies
- 0 kudos
- 0 kudos
@Oscar CENTENO MORA​ :To combine Py and R in a Databricks notebook, you can use the magics command %python and %rto switch between Python and R cells. Here's an example of how to create a Spark DataFrame in Python and then use it in R:from pyspark.sq...
- 0 kudos
- 2126 Views
- 3 replies
- 0 kudos
Performance issue while loading bulk data into Post Gress DB from data bricks.
We are facing a performance issue while loading bulk data into Postgress DB from data bricks. We are using spark JDBC connections to move the data. However, the rate of transfer is very low which is causing performance bottleneck. is there any better...
- 2126 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @Janga Reddy​ @Daniel Sahal​ and @Vidula Khanna​ ,To enhance performance in general we need to design for more parallelism, in Spark JDBC context this controlled by the number of partitions for the data to be writtenThe example here shows how t...
- 0 kudos
- 3623 Views
- 2 replies
- 2 kudos
Resolved! I am new to data bricks. setting up Workspace for NON-prod environment Separate workspaces for DEV, QA or Just one work space for NON-prod ?
What i learned based on learning materials, documents, etc.. For data bricks it is a good practice to set up 1 non-prod workspace but separate clusters for Dev, QA, SIT, etc.Is it best practice to set up only 1 NON-PROD Workspace instead of separate ...
- 3623 Views
- 2 replies
- 2 kudos
- 1763 Views
- 1 replies
- 6 kudos
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and...
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and reliable platform for users to build and deploy their applications. As part of this release, the en...
- 1763 Views
- 1 replies
- 6 kudos
- 6 kudos
Thank you for sharing @Hubert Dudek​ !!!
- 6 kudos
- 1904 Views
- 1 replies
- 7 kudos
Starting from #databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function ...
Starting from #databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function takes an array column as input and returns a new row for each element in the array, offering new pos...
- 1904 Views
- 1 replies
- 7 kudos
- 7 kudos
Thank you for sharing @Hubert Dudek​
- 7 kudos
- 7866 Views
- 1 replies
- 3 kudos
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:45-5:00 pm PT.We'll have Dave M...
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:45-5:00 pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to sha...
- 7866 Views
- 1 replies
- 3 kudos
- 3 kudos
Looking forward.
- 3 kudos
- 7865 Views
- 2 replies
- 2 kudos
Resolved! Connect GraphQL to Data Bricks
Hi I am new to Databricks however I need to expose data found in the delta lake directly to GraphQL to be queried by several applications. Is there a connector or something similar to GraphQL that works with Databricks?
- 7865 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Andrew Camilleri​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 2 kudos
- 1284 Views
- 1 replies
- 5 kudos
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...
- 1284 Views
- 1 replies
- 5 kudos
- 5 kudos
Thanks for sharing this with the Databricks community.
- 5 kudos
- 1783 Views
- 2 replies
- 5 kudos
"Hey everyone, it seems like there's some confusion about enhanced autoscaling in Databricks lately. If you're feeling lost or unsure abo...
"Hey everyone, it seems like there's some confusion about enhanced autoscaling in Databricks lately. If you're feeling lost or unsure about how it works, don't worry - you're not"Enhanced autoscaling is a feature in Databricks that enables dynamic sc...
- 1783 Views
- 2 replies
- 5 kudos
- 4674 Views
- 2 replies
- 0 kudos
Overwriting the existing table in Databricks; Mechanism and History?
Hi,Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table. (DF.write .format("delta") .mode("overwrite") .option("...
- 4674 Views
- 2 replies
- 0 kudos
- 0 kudos
the overwrite will add new files, keep the old ones and in a log keeps track of what is current data and what is old data.If the overwrite fails, you will get an error message in the spark program, and the data to be overwritten will still be the cur...
- 0 kudos
- 1232 Views
- 0 replies
- 2 kudos
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Ma...
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to shar...
- 1232 Views
- 0 replies
- 2 kudos
- 1063 Views
- 1 replies
- 5 kudos
Exciting news for Databricks users! #databricks launched a new feature that allows users to run job workflows continuously. Setting up a continuous jo...
Exciting news for Databricks users! #databricks launched a new feature that allows users to run job workflows continuously. Setting up a continuous job workflow is straightforward: create a job and select the continuous trigger option in the scheduli...
- 1063 Views
- 1 replies
- 5 kudos
- 5 kudos
Thank you for sharing!!!
- 5 kudos
- 1341 Views
- 0 replies
- 5 kudos
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and...
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and reliable platform for users to build and deploy their applications. As part of this release, the en...
- 1341 Views
- 0 replies
- 5 kudos
- 2408 Views
- 3 replies
- 7 kudos
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...
- 2408 Views
- 3 replies
- 7 kudos