- 42853 Views
- 2 replies
- 8 kudos
What's the difference between a Global view and a Temp view?
The difference between Global and Temp is how the lifetime of the view is tied to the application:http://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.createOrReplaceTempView.html?highlight=createorreplacetempview#pyspar...
- 42853 Views
- 2 replies
- 8 kudos
- 8 kudos
Correct A Temp View is scoped to the SparkSession and dropped when that session closes. Each notebook runs in its own SparkSession. The Global Temp View is scoped to the cluster and dropped when the cluster re-starts or you drop it. ---------------...
- 8 kudos
- 17077 Views
- 5 replies
- 4 kudos
Resolved! Pass date value as parameter in Databricks SQL notebook
I want to pass yesterday date (In the example 20230115*.csv) in the csv file. Don't know how to create parameter and use it here.CREATE OR REPLACE TEMPORARY VIEW abc_delivery_logUSING CSVOPTIONS ( header="true", delimiter=",", inferSchema="true", pat...
- 17077 Views
- 5 replies
- 4 kudos
- 4 kudos
@Retired_mod @sp1 @Chaitanya_Raju @daniel_sahal Hi Everyone,I need the same scenario on SQL code, because my DBR cluster not allowed me to run python codeError: Unsupported cell during execution. SQL warehouses only support executing SQL cells.I appr...
- 4 kudos
- 903 Views
- 0 replies
- 0 kudos
On the fly transformations on DLT tables
Hi, I am loading data from a kinesis data stream using DLT. CREATE STREAMING TABLE Consumers_kinesis_2 ( ..., unbase64(data) String, ... ) AS SELECT * FROM STREAM read_kinesis (...) Is it possible to directly cast, unbase64, and/or transform the resu...
- 903 Views
- 0 replies
- 0 kudos
- 17781 Views
- 9 replies
- 6 kudos
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resource...
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resources by triggering your Databricks jobs only when new files arrive in your cloud storage instead of mou...
- 17781 Views
- 9 replies
- 6 kudos
- 6 kudos
@daniel_sahal I get your point, but if for a scheduled trigger you can get all kind of attributes on the trigger time (arguably, this is available for all the triggers), then why wouldn't the most important attribute of a file event not be available ...
- 6 kudos
- 5435 Views
- 0 replies
- 0 kudos
Connecting to MuleSoft from Databricks
Hi, Is there any connectivity pipeline established already to access MuleSoft or AnyPoint exchange data using Databricks. I have seen many options to access databricks data in mulesoft but can we read the data from Mulesoft into databricks. Please gi...
- 5435 Views
- 0 replies
- 0 kudos
- 975 Views
- 0 replies
- 0 kudos
Using AAD Spn on AWS Databricks
I use AWS Databricks which has an SSO&Scim integration with AAD. I generated an SPN in AAD, synced it to Databricks, and want to use this SPN with using AAD client secrets to use Databricks SDK. But it doesnt work. I dont want to generate another tok...
- 975 Views
- 0 replies
- 0 kudos
- 1161 Views
- 1 replies
- 0 kudos
An error occurred when loading Jobs and Workflows App.
Hi,I was trying to open the Workflows but there is an error "An error occurred when loading Jobs and Workflows App." we need help to know why it happened and how we can resolve it please.
- 1161 Views
- 1 replies
- 0 kudos
- 0 kudos
Same...and the weirdest is that all of the services looks healthy in https://status.databricks.com/Region: eu-central-1Provider: AWSCould anyone provide some info here?
- 0 kudos
- 1315 Views
- 0 replies
- 0 kudos
Migrate a notebook that reside in workspace using Databricks Asset Bundle
Hello Community Folks -Did anyone implemented migration of notebooks that is in workspace to production databricks workspace using Databricks Asset Bundle? If so can you please help me with any documentation which I can refer? Thanks!!RegardsNiruban ...
- 1315 Views
- 0 replies
- 0 kudos
- 1391 Views
- 1 replies
- 0 kudos
Cached Views in MERGE INTO operation
Hi everyone!I want to use in-memory cached views in a merge into operation, but I am not entirely sure if the exactly saved in-memory view is used in this operation or not.So, suppose I have a table named table_1 and a cached view named cached_view_1...
- 1391 Views
- 1 replies
- 0 kudos
- 0 kudos
@deng_dev - Are you using external metastore by any chance. From the physical plan, we could see the catalog`.`db`.`table_1` is not cached. If it is glue catalog, then caching can be enabled based on the below configs in the article below https://do...
- 0 kudos
- 18641 Views
- 15 replies
- 8 kudos
Resolved! What are some best practices for CICD?
A number of people have questions on using Databricks in a productionalized environment. What are the best practices to enable CICD automation?
- 18641 Views
- 15 replies
- 8 kudos
- 8 kudos
Any leads/posts for Databricks CI/CD integration with Bitbucket pipeline. I am facing the below error while I creation my CICD pipeline pipelines:branches:master:- step:name: Deploy Databricks Changesimage: docker:19.03.12services:- dockerscript:# U...
- 8 kudos
- 1554 Views
- 0 replies
- 0 kudos
Error during deserializing protobuf data
I am receiving protobuf data in a json attribute and along with it I receive a descriptor file.I am using from_protobuf to deserialize the data as below,It works most of the time but giving error when there are some recursive fields within the protob...
- 1554 Views
- 0 replies
- 0 kudos
- 16825 Views
- 7 replies
- 0 kudos
Spark CSV file read option to read blank/empty value from file as empty value only instead Null
Hi,I am trying to read one file which having some blank value in column and we know spark convert blank value to null value during reading, how to read blank/empty value as empty value ?? tried DBR 13.2,14.3I have tried all possible way but its not w...
- 16825 Views
- 7 replies
- 0 kudos
- 0 kudos
OK, after some tests:The trick is in surrounding text in your csv with quotes. Like that spark can actually make a difference between a missing value and an empty value. Missing values are null and can only be converted to something else implicitel...
- 0 kudos
- 2199 Views
- 2 replies
- 0 kudos
Pushdown in Postgres
Hi team,In Databricks I need to query a postgres source likeselect * from postgres_tbl where id in (select id from df)the df is got from a hive table. If I use JDBC driver, and doquery = '(select * from postgres_tbl) as t' src_df = spark.read.format(...
- 2199 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for response. I cannot do that as we incrementally loading from source very frequently. We cannot read full data each time.
- 0 kudos
- 4482 Views
- 2 replies
- 1 kudos
Resolved! Best way to parse Google Analytics data in Databricks notebook
I managed to extract the Google Analytics data via lakehouse federation and the Big Query connection but the events table values are in a weird JSON format{"v":[{"v":{"f":[{"v":"ga_session_number"},{"v":{"f":[{"v":null},{"v":"2"},{"v":null},{"v":null...
- 4482 Views
- 2 replies
- 1 kudos
- 1 kudos
@AnaMocanu I was using this function, with a little modifications on my end:https://gist.github.com/shreyasms17/96f74e45d862f8f1dce0532442cc95b2Maybe this will be helpful for you
- 1 kudos
- 1889 Views
- 1 replies
- 0 kudos
Can I see queries sent to All Purpose Compute from Power BI?
I am brand new to Databricks and am working on connecting a power bi semantic model to our databricks instance. I have successfully connected it to an All Purpose Compute but was wondering if there was a way I could see the queries that power bi is ...
- 1889 Views
- 1 replies
- 0 kudos
- 0 kudos
For All purpose compute, best bet would be to use the system tables,specifically the system.access.audit table. https://docs.databricks.com/en/administration-guide/system-tables/index.html
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
