cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
syedmuhammedmeh
by • New Contributor III
  • 2410 Views
  • 3 replies
  • 6 kudos

Resolved! Databricks Kafka Read Not connecting

I'm trying to read data from GCP kafka through azure databricks but getting below warning and notebook is simply not completing. Any suggestion please? WARN NetworkClient: Consumer groupId Bootstrap broker rack disconnectedPlease note I've properly c...

Data Engineering
Reply
  • 2410 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 6 kudos

Hi @Syed Mohammed Mehdi​, We haven’t heard from you since the last response from @Jose Gonzalez​ , and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to other...

  • 6 kudos
2 More Replies
Sneha1594
by • New Contributor III
  • 6428 Views
  • 11 replies
  • 2 kudos

Visualizations from notebook keeps on disappearing

I have created a notebook which filters on the main table and generates bar graph. The graphs keeps on disappearing and sometimes they dont generate at all. PFA. 'Visualization 1' has issues. Each time I run notebook manually, I have to build the gra...

image
Data Engineering
Reply
  • 6428 Views
  • 11 replies
  • 2 kudos
View Replies
Latest Reply
Abishek
Esteemed Contributor III
  • 2 kudos

@Andrew Freix​ @Rubel Ahamed​ @Sneha Mulrajani​ eha15Can you please share the workspace URL version output? so that I can check your workspace and let you know whether new Visualization is enabled or not. Please run the below URL in your web browser ...

  • 2 kudos
10 More Replies
michaelh
by • New Contributor III
  • 3363 Views
  • 4 replies
  • 4 kudos

Resolved! AWS Databricks Cluster terminated.Reason:Container launch failure

We're developing custom runtime for databricks cluster. We need to version and archive our clusters for client. We made it run successfully in our own environment but we're not able to make it work in client's environment. It's large corporation with...

Data Engineering
Reply
  • 3363 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
FerArribas
Contributor
  • 4 kudos

@michael henzl​ , did you manage to solve the problem?

  • 4 kudos
3 More Replies
Pat
by • Honored Contributor III
  • 5226 Views
  • 4 replies
  • 16 kudos

Resolved! Unity Catalog - External Table

I am not sure if I am missing something, but I just created External Table using External Location and I can still access both data through the table and directly access files from the storage: documentation:https://docs.databricks.com/data-governanc...

image.png image.png
Data Engineering
Reply
  • 5226 Views
  • 4 replies
  • 16 kudos
View Replies
Latest Reply
Pat
Honored Contributor III
  • 16 kudos

I got the answer from the Databricks Support on this.The point which has been mentioned in doc "Once a table is created in a path, users can no longer directly access the files in that path even if they have been given privileges on an external locat...

  • 16 kudos
3 More Replies
Meenu1
by • New Contributor II
  • 772 Views
  • 0 replies
  • 1 kudos

X axis range for visualization in sql dashboard

Unable to set x axis range for charts in sql dashboard. The automatic range and padding that is set doesn't match across all plot types. For line charts with date type data on x axis, there is a padding on both sides of the plot. But for bar charts, ...

Data Engineering
Reply
  • 772 Views
  • 0 replies
  • 1 kudos
jwilliam
by • Contributor
  • 5337 Views
  • 3 replies
  • 3 kudos

Resolved! Differences between Spark Cluster Manager and Databricks Cluster Manager?

I didn't found any documentation on Databricks Cluster Manager. Could anyone give me some resources on this topic?

Data Engineering
Reply
  • 5337 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
User16752242622
Valued Contributor
  • 3 kudos

Hi @John William​ Databricks clusters use Spark's Standalone cluster manager. Each Databricks cluster has its own standalone Master and Worker processes run inside of the LXC containers and share a lifecycle with the cluster. Each cluster has a singl...

  • 3 kudos
2 More Replies
anibose
by • New Contributor III
  • 3103 Views
  • 5 replies
  • 7 kudos

Resolved! Hands-On exercise material

Hi FriendsI am following Databricks Customer Academy training material, and created a Databricks service in Azure Trial account and was able to launched a single node cluster there. Could you please guide me on how to do all the hands-on?

Data Engineering
Reply
  • 3103 Views
  • 5 replies
  • 7 kudos
View Replies
Latest Reply
anibose
New Contributor III
  • 7 kudos

Thanks Doug, I was able to locate .dbc file, appreciate your response. Best RegardsAnindya

  • 7 kudos
4 More Replies
weldermartins
by • Honored Contributor
  • 2099 Views
  • 4 replies
  • 11 kudos

Resolved! Databricks pyspark - Find columns in xls file.

Hello everyone, every day I extract data into xls files but the column position changes every day. Is there any way to find these columns within the file?Here's a snippet of my code.df = spark.read.format("com.crealytics.spark.excel")\ .option("hea...

Data Engineering
Reply
  • 2099 Views
  • 4 replies
  • 11 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 11 kudos

Hi @welder martins​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 11 kudos
3 More Replies
kfoster
by • Contributor
  • 1390 Views
  • 2 replies
  • 3 kudos

DLT Event Log

I am trying to utilize the Event Log DLT is keeping updated, I noticed some of the fields are consistently empty/null.In the Event Log, located ".../storage/system/events", I see the field "origin" and there are nested fields within which are empty/n...

Data Engineering
Reply
  • 1390 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 3 kudos

Hi @Kristian Foster​,The following docs will provide more details on the event log schema. Please refer to this link https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-event-log.html#monitor-pipelines-with-the-delta-live-tables...

  • 3 kudos
1 More Replies
LadislavSulak
by • New Contributor II
  • 1435 Views
  • 3 replies
  • 2 kudos

What is the long-term plan for the official Databricks Docker Containers?

Hi, I'd like to ask you, how much resources do you plan to dedicate to the maintenance/development of the official Databricks Docker images, please? Do you have a view on the longer-term plan for these docker images? It seems to be maintained, but i...

Data Engineering
Reply
  • 1435 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

Curious too,but I have a feeling it is not a priority.

  • 2 kudos
2 More Replies
parthsalvi
by • Contributor
  • 5569 Views
  • 7 replies
  • 3 kudos

GRANT permission not working on Storage Credential using DBR 11.2 Shared mode

We were trying to update permissions of Storage credential using DBR 11.2 Shared mode but running into following issue Operation not allowed: GRANT(line 1, pos 0) PFA complete error in file.Is the above issue with cluster permission or changing permi...

image
Data Engineering
Reply
  • 5569 Views
  • 7 replies
  • 3 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 3 kudos

Hi @Parth Salvi​ , could you please paste the full error here, also, it is better to create a support case with Databricks with the whole error.

  • 3 kudos
6 More Replies
benydc
by • New Contributor II
  • 863 Views
  • 0 replies
  • 1 kudos

Is it possible to connect to IPython Kernel from local or client outside databricks cluster?

When looking in the standard output of a notebook run in a cluster, we get this message: "To connect another client to this kernel, use: /databricks/kernel-connections-dj8dj93d3d3.json"Is it possible to connect to the databricks ipython kernel and ma...

Data Engineering
Reply
  • 863 Views
  • 0 replies
  • 1 kudos
dataexplorer
by • New Contributor III
  • 6478 Views
  • 6 replies
  • 5 kudos

Resolved! COPY INTO generating duplicate rows in Delta table

Hello Everyone,I'm trying to bulk load tables from a SQL server database into ADLS as parquet files and then loading these files into Delta tables (raw/bronze). I had done a one off history/base load but my subsequent incremental loads (which had a d...

Data Engineering
Reply
  • 6478 Views
  • 6 replies
  • 5 kudos
View Replies
Latest Reply
dataexplorer
New Contributor III
  • 5 kudos

thanks for the guidance!

  • 5 kudos
5 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors
View all