- 1222 Views
- 1 replies
- 1 kudos
Multiple metastores in Unity Catalog
Hi All,Can we have more than one meta store in a region in Unity Catalog? Having a single meta store per region helps keep metadata management organized, but customers are asking for multiple meta stores for different needs. Is it possible to have se...
- 1222 Views
- 1 replies
- 1 kudos
- 1 kudos
@Phani1 When attempting to create multiple metastores in a specific region, you may encounter the following error: This region already contains a metastore. Only a single metastore per region is allowed.Databricks recommends having only one metastore...
- 1 kudos
- 982 Views
- 1 replies
- 0 kudos
Clarification on VACUUM LITE operation
Hi all, I wanted some insight and clarification on the VACUUM LITE command. VACUUM | Databricks on AWSSo I am aware that the VACUUM FULL command will deletes data files outside of the retention duration and all files in the table directory not refere...
- 982 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @dbxlearner, If you set your retention duration to 3 days for a VACUUM LITE operation, it means that the command will use the Delta transaction log to identify and remove files that are no longer referenced by any table versions within the last...
- 0 kudos
- 3784 Views
- 10 replies
- 4 kudos
Custom Tag Usage Reporting
Hi Team,Does anyone have a good SQL query that I can use for showing usage costs against custom tags for example on clusters. The Account Console usage report is good but I can only seem to query one custom tag at a time and ideally I want a dashboar...
- 3784 Views
- 10 replies
- 4 kudos
- 4898 Views
- 10 replies
- 2 kudos
Unable to login to Community Edition
Facing the below issueWe were not able to find a Community Edition workspace with this email. Please login to accounts.cloud.databricks.com to find the non-community-edition workspaces you may have access to. For help, please see Community Edition Lo...
- 4898 Views
- 10 replies
- 2 kudos
- 2 kudos
Hi All,Now neither I can sign up from my account not login. o response from them so good luck to everyone out there who's trying to learn this tool.
- 2 kudos
- 1232 Views
- 1 replies
- 0 kudos
Data difference between SQL warehouse and all-purpose compute
Hey everyone,Executing the following query on my sql warehouse does not return any data:select * from acc_bolt.eod.configurationhistory where netarea = '541454827900000139';However running the same query using an all-purpose compute does return the e...
- 1232 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Daan , Greetings! Can you please confirm which SQL warehouse you are using here? If you are using a serverless, then can you try to run the query with PRO/Classic warehouse? Kind Regards, Ayushi
- 0 kudos
- 10260 Views
- 4 replies
- 0 kudos
Pyspark cast error
Hi All,hive> create table UK ( a decimal(10,2)) ;hive> create table IN ( a decimal(10,5)) ;hive> create view T as select a from UK union all select a from IN ;above all statements executes successfully in Hive and return results when select statement...
- 10260 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Nandini,Thanks for sharing the above solution. To be sure my understanding is correct, could you confirm below please ?hive> create table test.UK ( a decimal(10,2)) ;hive> create table test.IN ( a decimal(10,5)) ;hive> create view test.T as select...
- 0 kudos
- 710 Views
- 1 replies
- 0 kudos
EMR cluster pyspark scripts to databricks
Hi All,The PySpark scripts currently operating on the EMR cluster need to be migrated to Databricks. Are there any tools available that can assist in minimizing the time required for code conversion? Your suggestions would be appreciated.Regards,Phan...
- 710 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Phani1, This guide can help you: https://www.databricks.com/resources/guide/emr-databricks-migration-guide
- 0 kudos
- 1038 Views
- 1 replies
- 0 kudos
Airflow jobs migration to Databricks Workflows
Hi All,We need to move our Airflow jobs over to Databricks Workflows. Are there any tools out there that can help with this migration and make the process quicker? If you have any sample code or documents that could assist, I would really appreciate ...
- 1038 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Phani1, Please see this post which can help you: https://community.databricks.com/t5/data-engineering/migrating-logic-from-airflow-dags-to-databricks-workflow/td-p/104501
- 0 kudos
- 2321 Views
- 4 replies
- 1 kudos
Different JSON Results when Running a Job vs Running a Notebook
I have a regularly scheduled job that runs a PySpark Notebook that GETs semi-structured JSON data from an external API, loads that data into dataframes, and saves those dataframes to delta tables in Databricks. I have the schema for the JSON defined ...
- 2321 Views
- 4 replies
- 1 kudos
- 1 kudos
@Alberto_Umana Sounds good, thank you for looking into it and let me know if there's any additional information I can provide in the meantime!
- 1 kudos
- 5875 Views
- 4 replies
- 9 kudos
Zero to Hero - Databricks
Hi all!In a nutshell, I want to go from zero to hero with Databricks. I'd like to pursue the Databricks Data Engineering pathway, I think that makes sense as I have a background with Alteryx.I'd really like to get hands on whilst learning. Are the le...
- 5875 Views
- 4 replies
- 9 kudos
- 9 kudos
@MariuszK thanks for the link to your medium article. There's some great stuff in there!Good point about the 30 day Azure free trial for Databricks.
- 9 kudos
- 1636 Views
- 4 replies
- 0 kudos
dbt error: Data too long for column at row 1
Hi there!We are experiencing a Databricks error we don’t recognise when we are running one of our event-based dbt models in dbt core (version 1.6.18). The dbt model uses the ‘insert_by_period’ materialisation that is still experimental for version 1....
- 1636 Views
- 4 replies
- 0 kudos
- 0 kudos
We are yet to upgrade dbt core to the latest version but will check again once we have done so.
- 0 kudos
- 3598 Views
- 4 replies
- 2 kudos
Resolved! Unity Catalog Migration: External AWS S3 Location Tables vs. Managed Tables in Databricks!
Hey Databricks enthusiasts!Migrating to Unity Catalog? Understanding the difference between External S3 Location Tables and Managed Tables is crucial for optimizing governance, security, and cost efficiency.External S3 Location TablesData remains in ...
- 3598 Views
- 4 replies
- 2 kudos
- 2 kudos
Hey!I hope I’m not too late, and I’d like to share my opinion. While it’s true that managed services offer certain advantages over external tables, you should keep in mind that Databricks services often come with an associated cost, such as Predictiv...
- 2 kudos
- 1073 Views
- 1 replies
- 0 kudos
Terminated cluster on free account
Hi,I mistakenly terminated my cluster. Could you please advise on how I can reactivate the same cluster?
- 1073 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Lupo123, To reactivate a terminated cluster on a free Databricks account, you will need to create a new cluster. Unfortunately, once a cluster is terminated, it cannot be reactivated
- 0 kudos
- 11917 Views
- 4 replies
- 2 kudos
Gathering Data Off Of A PDF File
Hello everyone,I am developing an application that accepts pdf files and inserts the data into my database. The company in question that distributes this data to us only offers PDF files, which you can see attached below (I hid personal info for priv...
- 11917 Views
- 4 replies
- 2 kudos
- 2 kudos
You can use PDF Data Source for read data from pdf files. Examples here: https://stabrise.com/blog/spark-pdf-on-databricks/And after that use Scale DP library for extract data from the text in declarative way using LLM. Here is example of extraction ...
- 2 kudos
- 2064 Views
- 1 replies
- 0 kudos
Speaker diarization on databricks with Nemo throwing error
The configuration of my compute is 15.4 LTS ML (includes Apache Spark 3.5.0, GPU, Scala 2.12)Standard_NC8as_T4_v3 on Azure Databricks
- 2064 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Nishat ,It looks like there's a problem with GPU compability. As mentioned in the error message, FlashAttention only supports Ampere GPUs or newer.According to following thread, GPU architecture you've chosen is not supportedRuntimeError: FlashAt...
- 0 kudos
