- 7113 Views
- 5 replies
- 1 kudos
Resolved! [TABLE_OR_VIEW_ALREADY_EXISTS] when running create or replace view
When I'm running `dbt run -s model` for different models in parallel on databricks general compute cluster. Im getting the error: TABLE_OR_VIEW_ALREADY_EXISTS. Also, at the same time, the view/table was not created at all.But I run them in sequence, ...
- 7113 Views
- 5 replies
- 1 kudos
- 1 kudos
@raphaelblg yes! Im the owner. However, these views are inside the hive_metastore catalog. Could it be the reason?Many thaaanks!
- 1 kudos
- 3120 Views
- 1 replies
- 0 kudos
Running JAR jobs in parallel on a cluster
Hi everyone, I'm trying to find out if databricks has support for clusters which can scale out with more drivers to run new jobs in parallel. If not, then is there a work around for this? I've noticed that all-purpose and job compute clusters both fe...
- 3120 Views
- 1 replies
- 0 kudos
- 0 kudos
@AchintyaSingh Databricks does not support clusters with multiple drivers to run new jobs in parallel. Each Databricks cluster has a single driver node, allowing only one job at a time.Workarounds for Achieving Parallel Job Execution: 1. Multiple Cl...
- 0 kudos
- 5351 Views
- 2 replies
- 0 kudos
InconsistentReadException: The file might have been updated during query - CSV backed table
I have some CSV files that I upload to DBFS storage several times a day. From these CSVs, I have created SQL tables: CREATE TABLE IF NOT EXISTS masterdata.lookup_host USING CSV OPTIONS (header "true", inferSchema "true") LOCATION '/mnt/masterdata/...
- 5351 Views
- 2 replies
- 0 kudos
- 0 kudos
One approach I'm testing (positive results so far, but still early).%sql # Prep and cleanup REFRESH TABLE masterdata.lookup_host; DROP TABLE IF EXISTS t_hosts; # Forcibly cache the needed columns before using the data in another query. CACHE TABLE...
- 0 kudos
- 2590 Views
- 1 replies
- 0 kudos
Add Raw Tabless to Unity Catalog?
Hello everyone,Hope someone here can help me out as I am a bit stuck I am currently loading some data from number of sources to parquet using ADF. This ends up in a ADLS2 storage account as .parquet. This is the staging / landing zone area.From there...
- 2590 Views
- 1 replies
- 0 kudos
- 16297 Views
- 0 replies
- 0 kudos
Change adls gen2 attached to Unity Catalog metastore from premium to standard
Hello,Our cloud platform engineer created Azure storage for the production Unity Catalog metastore in our environment, but mistakenly chose the Premium tier instead of the Standard tier.Unfortunately, this decision is impacting our costs on Azure, as...
- 16297 Views
- 0 replies
- 0 kudos
- 3978 Views
- 2 replies
- 0 kudos
Accessing ADLS Gen2 related Hadoop configuration in notebook
I have a cluster in which I have the required configuration to access an ADLS Gen2, and it works without any problems.I want to access this storage using the Hadoop filesystem APIs. To achieve this, I am trying to get the Hadoop configuration from th...
- 3978 Views
- 2 replies
- 0 kudos
- 0 kudos
By the way how do you achieve inline code highlighting in the editor I tried `` but it didn't work.
- 0 kudos
- 2232 Views
- 1 replies
- 2 kudos
Resolved! DAIS24 Attendee Badge - Oops
I wasn't at DAIS24, but I received one of these emails and it appears to have come from Databricks. I think I was given the badge in error. Can we have it removed?Just Wondering,Shawn
- 2232 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Shawn_Eary Thank you for bringing this to our attention. We were conducting a test on badges, and some of our community members may have received this email by accident. We have revoked the changes now, please ignore the message.
- 2 kudos
- 3515 Views
- 1 replies
- 0 kudos
Tableau Prep Save Output to Databricks
Has anyone run into use cases where your data scientist/data engineer end users build Tableau Prep Flows and steps in Tableau Prep Flow require saving output back into Databricks? There appears to be no native support for this in Tableau Prep if the ...
- 3515 Views
- 1 replies
- 0 kudos
- 0 kudos
These are awesome suggestions. To expand on our setup, we also have Informatica Cloud - IICS (CMI, CDI, etc.) connected to the entire setup generally used for bringing data from a source (PaaS, SaaS, On-prem SQL, Flat Files or streaming devices) to D...
- 0 kudos
- 4929 Views
- 4 replies
- 1 kudos
Resolved! Getting error while running the MLOPS demo
Hello,I am trying to run this demo in databricks community edition but i am facing error.MLOPS DEMO - https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/mlops-end-to-end-pipeline?itm_data=demo_centerSomeone else also faced the s...
- 4929 Views
- 4 replies
- 1 kudos
- 1 kudos
Hey @Kushal_Nagrani , thank you for sharing this. I am facing the same issue. Could you please help me where can I find these settings? I am new, just created a community workspace and I thought given I am the admin I will have all these by default -...
- 1 kudos
- 1355 Views
- 0 replies
- 0 kudos
Training link deprecated: How to ingest data for Databricks SQL
I am currently doing a course in Databricks academy: How to ingest data for Databricks SQL.To create a table in the external location I am provided with the link that is not working anymore. Below is the link:wasbs://courseware@dbacademy.blob.core.wi...
- 1355 Views
- 0 replies
- 0 kudos
- 2025 Views
- 1 replies
- 0 kudos
Materialized Views Still In Public Preview?
The title is self-explanatory, but I'm curious if anyone has any inclination as to when Materialized Views will enter General Availability? I'm surprised it's not already in GA considering (A) it's been a year or more since Databricks originally anno...
- 2025 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, if the documentation states it, it's still in preview.
- 0 kudos
- 8719 Views
- 2 replies
- 0 kudos
Seeking Tips: Ways to Master Databricks on Azure?
Hello everyone. I'm currently learning Databricks on Azure through a Udemy course. Recently, I was surprised by a charge of $86 from Azure, which has made me cautious about continuing in this manner. Is there a more cost-effective approach to learn D...
- 8719 Views
- 2 replies
- 0 kudos
- 3645 Views
- 1 replies
- 0 kudos
databricks asset bundles on azure devops
@herewe are able to work on DAB locally or Github actions, but looks from azure devops end lot of missing pieces in terms of pipeline and examples. can any one point me out github repo that is good to test azure devops pipeline for DAB.
- 3645 Views
- 1 replies
- 0 kudos
- 1596 Views
- 1 replies
- 0 kudos
Assigning a group as USER to service principal
How can we assigning a group as USER to service principal using databricks-sdk, this is not supported?
- 1596 Views
- 1 replies
- 0 kudos
- 0 kudos
I found this API, https://docs.databricks.com/api/account/accountaccesscontrol/updaterulesetbut its PUT and GET method both requires a parameter "etag", how can someone know this "etag"?
- 0 kudos
- 3263 Views
- 3 replies
- 2 kudos
DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute
Hi there, I would like to understand for DLT jobs if there is any way to get DLT jobs running in an *existing* (and currently running) All-purpose compute rather than spinning up an "ephemeral" (not yet initialized) Job Compute?
- 3263 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ChristianRRL , you cannot run a DLT pipeline in an All-purpose compute cluster. DLT Clusters are managed resources. Here are some details on how to configure a DLT cluster: https://docs.databricks.com/en/delta-live-tables/settings.html#configure-...
- 2 kudos
