cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
rpaschenko
by • New Contributor II
  • 688 Views
  • 1 replies
  • 0 kudos

Bootstrap Timeout during cluster start on AWS cloud

Hi!We had bunch of strange failures for our jobs during 28-29 of September.Some jobs` runs could not start for some time (30-50 mins) and then were failed with an error:Unexpected failure while waiting for the cluster (0929-002141-2zkekhdj) to be rea...

Community Discussions
Reply
  • 688 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @rpaschenko, The failures you experienced on September 28-29 could be due to various reasons.  For the jobs that could not start for some time and then failed with an error, there appears to be a timeout issue while initializing the instance. This...

  • 0 kudos
Phani1
by • Valued Contributor
  • 1732 Views
  • 1 replies
  • 0 kudos

Databricks on Virtualization

Hi Team,Can you please direct me to any content on Databricks on Virtualization?Regards,Phanindra   

Community Discussions
Databricks on Virtualization
Reply
  • 1732 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Phani1, Databricks provides the ability to deploy a Databricks workspace to your virtual network, also known as VNet injection. This option allows for network customization and offers a variety of features: * Connect Databricks to other Azure se...

  • 0 kudos
youssefmrini
by • Honored Contributor III
  • 3708 Views
  • 0 replies
  • 3 kudos

What's new in Databricks for September 2023

Platform You can now use Structured Streaming to Stream Data from Apache Pulsar on Databricks. For more information : https://docs.databricks.com/en/structured-streaming/pulsar.html (DBR 14.1 required)Databricks Runtime 14.1 and 14.1 ML are now avail...

Screenshot 2023-10-03 at 8.58.12 PM.png
Community Discussions
delta
Reply
  • 3708 Views
  • 0 replies
  • 3 kudos
manas_884
by • New Contributor II
  • 1638 Views
  • 3 replies
  • 0 kudos

om.microsoft.azure.storage.StorageException: The specifed resource name contains invalid characters.

Hi guys I'm relatively new to Databricks and struggling to implement an autoloader ( with trigger once = true ) in file notifications mode. I have CSV files in one container (landing zone). I would like the autoloader to pick up new and existing file...

Community Discussions
Reply
  • 1638 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
manas_884
New Contributor II
  • 0 kudos

Hi Kaniz, thank you for your reply. I initially made the mistake of using a capital letter in the queue as part of config files. I can now write, there is no error as a batch process. However, when I try to run the write stream, it says"Running Comma...

  • 0 kudos
2 More Replies
sirishavemula20
by • New Contributor III
  • 6680 Views
  • 9 replies
  • 1 kudos

My exam Datbricks Data Engineer Associate got suspended_need immediate help please (10/09/2023)

Hello Team,I encountered Pathetic experience while attempting my DataBricks Data engineer certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam.I want to file ...

Community Discussions
Reply
  • 6680 Views
  • 9 replies
  • 1 kudos
View Replies
Latest Reply
Cert-Team
Honored Contributor III
  • 1 kudos

yes, it was!

  • 1 kudos
8 More Replies
Ruby8376
by • Valued Contributor
  • 1449 Views
  • 2 replies
  • 2 kudos

Is Datbricks-Salesforce already available?

Reference: Salesforce and Databricks Announce Strategic Partnership to Bring Lakehouse Data Sharing and Shared AI Models to Businesses - Salesforce NewsI was going through this article and wanted to know if anyone in community is planning to use this...

Community Discussions
Reply
  • 1449 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Ruby8376, Will get back to you on this. 

  • 2 kudos
1 More Replies
6502
by • New Contributor III
  • 2112 Views
  • 3 replies
  • 3 kudos

Resolved! Terraform provider, problemi in creating dependant task!

Hola all.I have a serious problem, perhaps I missed something, but can't find the solution. I need to push a job description to Databricks using TERRAFORM. I wrote the code, but there is no way to get a task dependant from two different tasks.Conside...

Community Discussions
Reply
  • 2112 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 3 kudos

@6502 You need to make multiple depends_on blocks for each dependency, ex.depends_on { task_key = "ichi" } depends_on { task_key = "ni" }

  • 3 kudos
2 More Replies
Kaviana
by • New Contributor III
  • 1140 Views
  • 2 replies
  • 0 kudos

Connection VPC AWS in Databricks for extract data Oracle Onpremise

 Hi, help meHow can I consume a VPC when it is already anchored in the "network" of Databricks to extract information from a server that really made ping and works

Community Discussions
Reply
  • 1140 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaviana
New Contributor III
  • 0 kudos

Hello @Kaniz_Fatma I have already configured in "Cloud resources"/ "Network" the Private endpoint links and the linked VPC in ibabricks. How can I connect to Oracle using the EC2 virtual machine?Thank u

  • 0 kudos
1 More Replies
simran_27
by • New Contributor
  • 615 Views
  • 1 replies
  • 0 kudos

Error while attempt to give Lakehouse fundamentals exam

I would like to know why I am getting this error when I tried to earn badges for lake house fundamentals. I can't access the quiz page. Can you please help on this?I am getting 403: Forbidden error.

Community Discussions
Reply
  • 615 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
APadmanabhan
Moderator
  • 0 kudos

Hi @simran_27 could you please try using the link 

  • 0 kudos
dvmentalmadess
by • Valued Contributor
  • 7354 Views
  • 3 replies
  • 1 kudos

Structured Streaming of S3 source

I am trying to setup s3 as a structured streaming source. The bucket receives ~17K files/day and the original load to the bucket was ~54K files. The bucket was first loaded 3 months ago and we haven't started reading from it since. So let's say there...

Community Discussions
Reply
  • 7354 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
dvmentalmadess
Valued Contributor
  • 1 kudos

Thanks,We were able to make things work by increasing the driver instance size so it has more memory for the initial load. After initial load we scaled the instance down for subsequent runs. We're still testing, if we aren't able to make it work we'l...

  • 1 kudos
2 More Replies
rajib_bahar_ptg
by • New Contributor III
  • 2271 Views
  • 7 replies
  • 6 kudos

Databricks workspace unstable

Our company's databricks workspace is unstable lately. It can't launch any compute cluster. I have never seen this issue. In addition to this issues, I have seen storage credential error on main unity catalog. Why would this happen AWS Databricks ins...

Community Discussions
Reply
  • 2271 Views
  • 7 replies
  • 6 kudos
View Replies
Latest Reply
rajib_bahar_ptg
New Contributor III
  • 6 kudos

Hello @Kaniz_Fatma and @jose_gonzalez ,I couldn't locate the support ticket we opened. How can we track that ticket down? It came from the peopletech.com domain. If it is more efficient to create another ticket, please let me know. Let us know the UR...

  • 6 kudos
6 More Replies
gideont
by • New Contributor III
  • 2145 Views
  • 3 replies
  • 3 kudos

Application extracting Data from Unity Catalogue

Dear Databricks community,I'm seeking advice on the best method for applications to extract data from the Unity catalogue. One suggested approach is to use JDBC, but there seems to be a dilemma. Although using a job cluster has been recommended due t...

Community Discussions
Reply
  • 2145 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

What exacltly do you mean by 'extracting'?  If you want to load tables defined in Unity into a database, I would indeed do this using job clusters and a notebook.If you want to extract some data once in a while into a csv f.e., you could perfectly do...

  • 3 kudos
2 More Replies
JohnSmith2
by • New Contributor II
  • 1788 Views
  • 4 replies
  • 2 kudos

Resolved! Error on Workflow

Hi , I have some mysteries situation here My workflow (job) ran and got an error -> [INVALID_IDENTIFIER] The identifier transactions-catalog is invalid. Please, consider quoting it with back-quotes as `transactions-catalog`.(line 1, pos 12) == SQL ==...

Community Discussions
Reply
  • 1788 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

Jobs are just notebooks executed in background, so if the notebook is the same between interactive (manual) and job run, there should be no difference.So I don't see what is wrong.  Is the job using DLT perhaps?

  • 2 kudos
3 More Replies
DBEnthusiast
by • New Contributor III
  • 2655 Views
  • 2 replies
  • 2 kudos

Resolved! DataBricks Cluster

Hi All,I am curious to know the difference between a spark cluster and a DataBricks one.As per the info I have read Spark Cluster creates driver and Workers when the Application is submitted whereas in Databricks we can create cluster in advance in c...

Community Discussions
Reply
  • 2655 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @DBEnthusiast, In a Spark cluster, the SparkContext object in your main program (the driver program) connects to a cluster manager, which could be Sparkâs standalone cluster manager, Mesos, YARN, or Kubernetes. This cluster manager allocates resou...

  • 2 kudos
1 More Replies
eimis_pacheco
by • Contributor
  • 3019 Views
  • 4 replies
  • 2 kudos

Resolved! Is it not needed to preserve the data in its original format anymore with the usage of medallion?

Hi Community I have a doubt. The bronze layer always causes confusion for me. Someone mentioned, "File Format: Store data in Delta Lake format to leverage its performance, ACID transactions, and schema evolution capabilities" for bronze layers.Then, ...

Community Discussions
Reply
  • 3019 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @eimis_pacheco ,  You can store the data in its original format in the Bronze layer. The recommendation to use Delta Lake format for the Bronze layer is mainly for better. The purpose of the Bronze layer in the Lambda architecture is to store data...

  • 2 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all