cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
flobib123
by • New Contributor III
  • 7484 Views
  • 5 replies
  • 3 kudos

Resolved! Syntax highlight support for python multiline SQL strings

I would like to know if a tool is available in Databricks to manage the SQL syntax highlight in text.Like this VSCode plugin:https://marketplace.visualstudio.com/items?itemName=ptweir.python-string-sqlThank you.

Data Engineering
Reply
  • 7484 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
flobib123
New Contributor III
  • 3 kudos

Hello @Vidula Khanna​ ,No, no one answered my question correctly, but I'm using PySpark now, so I'm not on this topic anymore.But thank you for taking the time to answer me.

  • 3 kudos
4 More Replies
4kb_nick
by • New Contributor III
  • 4836 Views
  • 3 replies
  • 5 kudos

302 Found when trying to run Unity Catalog Quickstart

Hi there,I'm helping a client of mine set up an Azure Databricks environment. The workspace is set up for private access only, and we are using Azure Firewall and Azure Private Link.We have the network environment successfully configured to the point...

Data Engineering
Reply
  • 4836 Views
  • 3 replies
  • 5 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Nick Barretta​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest ...

  • 5 kudos
2 More Replies
h_aloha
by • New Contributor III
  • 1783 Views
  • 2 replies
  • 2 kudos

Bugs in data-engineer-learning-path-v1-0-0-notebooks.dbc

Error message returned in DE 4.1 - DLT UI WalkthroughCmd 6:pipeline_language = "SQL"# pipeline_language = "Python"DA.print_pipeline_config(pipeline_language)Error:AttributeError: 'DBAcademyHelper' object has no attribute 'get_username_hash'----------...

Data Engineering
Reply
  • 1783 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Helen Morgen​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...

  • 2 kudos
1 More Replies
RaviMuna_33806
by • New Contributor II
  • 2722 Views
  • 8 replies
  • 1 kudos

users/administrators counts per workspace

How can i find number of users/admins per workspace - looking for notebook execution option?

Data Engineering
Reply
  • 2722 Views
  • 8 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Ravi M​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest providi...

  • 1 kudos
7 More Replies
robert37201
by • New Contributor II
  • 2053 Views
  • 3 replies
  • 4 kudos

Job aborted due to stage failure: Input buffer size 0 for bloom filter is not power of 2

Query works great in a notebook, fails in Classic SQL Warehouse (photon enabled) with that error. Tables are relatively small. Just don't know where to begin understanding that error, google wasn't much help and Query History doesn't give me anything...

Data Engineering
Reply
  • 2053 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Robert McCartney​ We haven't heard from you since the last response from @Lakshay Goel​ ​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to...

  • 4 kudos
2 More Replies
620139
by • New Contributor III
  • 2500 Views
  • 3 replies
  • 3 kudos

Error when running OPTIMIZE on a Delta table with generated columns

I am seeing an error when running OPTIMIZE on a Delta table with generated columns:com.databricks.sql.transaction.tahoe.schema.DeltaInvariantViolationException: CHECK constraint Generated Column (created <=> now()) violated by row with values: - crea...

Data Engineering
Reply
  • 2500 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jeff Erickson​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...

  • 3 kudos
2 More Replies
billaspiel
by • New Contributor II
  • 13986 Views
  • 3 replies
  • 0 kudos

Resolved! Python open function is unable to detect the file in dbfs

hi , Im a newbie learning spark using databricks , I did some investigation and searched if this questions was been asked earlier in community forum but unable to find anything so . 1. DBFS is unable to detect the file even though its present in it...

Data Engineering
Reply
  • 13986 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Dflo
New Contributor II
  • 0 kudos

I am having similar issues currently. I can read or access my storage account but when I attempted to read or access the container it told me path not found. I create the container and have full access as an owner.

  • 0 kudos
2 More Replies
pvignesh92
by • Honored Contributor
  • 16634 Views
  • 8 replies
  • 0 kudos

Resolved! Multi Statement Writes from Spark to Snowflake

Does Spark support multi statement writes to Snowflake in a single session? To elaborate, I have a requirement where I need to do A selective deletion of data from a Snowflake table and Insert records to Snowflake table ( Ranges from around 1 M rows)...

Data Engineering
Reply
  • 16634 Views
  • 8 replies
  • 0 kudos
View Replies
Latest Reply
pvignesh92
Honored Contributor
  • 0 kudos

In my analysis, I got the below understanding If your data is sitting in Snowflake and you have a set of DDL/DML queries that need to wrapped into a single transaction, you can use MULTI_STATEMENT option to 0 and use snowflake utils runQuery method t...

  • 0 kudos
7 More Replies
sanjay
by • Valued Contributor II
  • 18319 Views
  • 4 replies
  • 3 kudos

Resolved! How to backup databrick delta tables or workspace

Hi,I am trying to understand how to take backup of databrick delta tables/workspace and how to restore in case if any failure.Or suggest me any alternative solution to revert back if data is corrupted.Regards,Sanjay

Data Engineering
Reply
  • 18319 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
NandiniN
Databricks Employee
  • 3 kudos

Hi @Sanjay Jain​ ,Here are some of the ways Deep Clone: https://www.databricks.com/wp-content/uploads/notebooks/using-deep-clone-disaster-recovery-delta-lake-databricks.htmlRepos for notebooks and code: https://docs.databricks.com/repos/index.htmlht...

  • 3 kudos
3 More Replies
KVNARK
by • Honored Contributor II
  • 3944 Views
  • 7 replies
  • 7 kudos

Resolved! No certification vouchers in Rewards portal.

there are no certification vouchers in rewards portal. Could some one from data bricks team update whether you are planning to include them again and also very few data bricks merchandise.

Data Engineering
Reply
  • 3944 Views
  • 7 replies
  • 7 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @KVNARK .​ Thank you for reaching out to us!The reward store will be updated for the Spring and new rewards will be introduced in the coming weeks. Kindly, bear with us.Regards

  • 7 kudos
6 More Replies
mkd_140848
by • New Contributor II
  • 3663 Views
  • 5 replies
  • 0 kudos

Open notebook in new tab not working as of 2 March 2023. Web browser: Google Chrome 110.0.5481.178 Databricks version: Standard Cloud platform: Azure

Before Feb 2023, I often opened Notebook in new tabs by Ctrl (or Command for Mac) + Click the notebook names, in the Workspace explorer UI. Recently it stopped working (Ctrl + Click Notebook name, nothing happened). Is this a bug?

Data Engineering
Reply
  • 3663 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @David H​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
4 More Replies
Kayla
by • Valued Contributor II
  • 3623 Views
  • 2 replies
  • 0 kudos

BigQuery - Delete or update from Databricks?

I'm trying to sync a delta table in Databricks to a BigQuery table. For the most part, appending is sufficient, but occassionally we need to overwrite rows - which we've only been able to do by overwriting the entire table.Is there any way to do upda...

Data Engineering
Reply
  • 3623 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Kayla Pomakoy​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 0 kudos
1 More Replies
souvik
by • New Contributor
  • 1172 Views
  • 2 replies
  • 0 kudos

Unable to get or download fundamentals of databricks lake house platform badge download

Hi Team,I have completed the fundamentals of databricks lakehosue platform and received the certificate but unable to download or get the badge.When I login to website I am unbale to see the badge as well.Please help me on this.I haven't received any...

Data Engineering
Reply
  • 1172 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Souvik Banerjee​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 0 kudos
1 More Replies
RajeshRK
by • Contributor II
  • 4850 Views
  • 7 replies
  • 2 kudos

How to optimize jobs performance

Hi Team,We have a complex ETL job running in databricks for 6 hours. The cluster has the below configuration: Minworkers: 16Maxworkers: 24Worker and Driver Node Type: Standard_DS14_v2. (16 cores, 128 GB RAM)I have monitored the job progress in Spark...

Data Engineering
Reply
  • 4850 Views
  • 7 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rajesh Kannan R​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedb...

  • 2 kudos
6 More Replies
AyushModi038
by • New Contributor III
  • 5022 Views
  • 6 replies
  • 0 kudos

Library mismatch in same cluster different file

In continuation to the issues encountered in this discussion.https://community.databricks.com/s/feed/0D58Y00009tCiQTSA0 I have a bizzare issue.Here are the 2 screenshots taken few seconds apart1.2 . Same cluster, same command, executed 6 seconds apar...

image image
Data Engineering
Reply
  • 5022 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Ayush Modi​  I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest pr...

  • 0 kudos
5 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels
Top