cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
marco_almeida
by • New Contributor II
  • 1212 Views
  • 2 replies
  • 2 kudos

I can't import an library like the example

I read this article and I created a notebook to use like a library but when I tried to import it in other notebook I received this error: No module named 'lib.lib_test' No module named 'lib.lib_*****'

Data Engineering
Reply
  • 1212 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey @Marco Antônio de Almeida Fernandes​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love...

  • 2 kudos
1 More Replies
MaverickF14
by • New Contributor II
  • 2115 Views
  • 6 replies
  • 2 kudos

From a noob Databrickser... concerning Python programming in databricks

The following...We,ve got clients working with us in contracts. Per client several contracts of a certain type with start- and end dates. If I need aggregated info per client in one record like:how many different contract did the client haveof which ...

Data Engineering
Reply
  • 2115 Views
  • 6 replies
  • 2 kudos
View Replies
Latest Reply
MaverickF14
New Contributor II
  • 2 kudos

Yeah, Thanks for all the help!

  • 2 kudos
5 More Replies
alexgv12
by • New Contributor III
  • 2649 Views
  • 2 replies
  • 0 kudos

How can I somehow run spark.something in a worker? - rdd foreach spark.context

i am using rdd to parallelize a function, in this function i format the record i want to save, how can i store from this function the record with a dataframe? because every time i use spark..... an error is generated Caused by: org.apache.spark.api.p...

Data Engineering
Reply
  • 2649 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @alexander grajales vanegas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear ...

  • 0 kudos
1 More Replies
HariharaSam
by • Contributor
  • 2223 Views
  • 2 replies
  • 3 kudos

Enabling Fair Scheduler from Databricks Notebook

How to enable fair scheduler from Databricks notebook using python commands?

Data Engineering
Reply
  • 2223 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Hariharan Sambath​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 3 kudos
1 More Replies
Braxx
by • Contributor II
  • 1234 Views
  • 2 replies
  • 3 kudos

List mounts for specyfic scope

I have like 40 mounts and some of them were done based on the scopes which now I want to remove. How can I list the mounts for specyfic scope or even better, get the list of all the mounts alongside the scopes?

Data Engineering
Reply
  • 1234 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Bartosz Wachocki​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fro...

  • 3 kudos
1 More Replies
Pat
by • Honored Contributor III
  • 9546 Views
  • 7 replies
  • 18 kudos

Resolved! Cluster Modes - High Concurrency

It took me quite some time to find the option to create a cluster in High Concurrency mode. It was hidden in the new UI.What should be the way to access the data with TAC?What is the equivalent mode to work with TAC ?Does it mean that we are being pu...

image image.png image
Data Engineering
Reply
  • 9546 Views
  • 7 replies
  • 18 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 18 kudos

Thanks. Always happy to help.

  • 18 kudos
6 More Replies
RamaSantosh
by • New Contributor II
  • 3735 Views
  • 2 replies
  • 3 kudos

Data load from Azure databricks dataframe to cosmos db container

I am trying to load data from Azure databricks dataframe to cosmos db container using below commandcfg = { "spark.cosmos.accountEndpoint" : cosmosEndpoint, "spark.cosmos.accountKey" : cosmosMasterKey, "spark.cosmos.database" : cosmosDatabaseName, "sp...

Data Engineering
Reply
  • 3735 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey @Rama Santosh Ravada​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 3 kudos
1 More Replies
Praveen2609
by • New Contributor
  • 1813 Views
  • 2 replies
  • 0 kudos

dbfs access for job clusters and interactive cluster

Hi All,I am new to databricks need some understanding for my requirement .our requirement:a: we have zip file in azure blob storage and we are bringing that file to dbfs and unzip that file and executing our transformations in multiple steps (3 steps...

Data Engineering
Reply
  • 1813 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @praveen rajak​ Does @Debayan Mukherjee​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
Trey
by • New Contributor III
  • 1760 Views
  • 2 replies
  • 3 kudos

Where do you usually store and manage "JDBC credentials" to use on databricks notebook?

Hi all,I would like to improve the way I use JDBC credenditial information (ID/PW, host, port, etc)Where do you guys usually store and use the jdbc credentials?Thanks for your help in advance!

Data Engineering
Reply
  • 1760 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Kwangwon Yi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 3 kudos
1 More Replies
RohitKulkarni
by • Contributor II
  • 2026 Views
  • 3 replies
  • 4 kudos

TCP/IP Issue

Hello Team,I am trying to read the data from Synapse analytics from databricks.The below is the Query to read the table.%python### Read from Azure Synapse table via spark.read.load df = spark.read \ .format("com.databricks.spark.sqldw") \ .option("ur...

Data Engineering
Reply
  • 2026 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 4 kudos
2 More Replies
db-avengers2rul
by • Contributor II
  • 1140 Views
  • 1 replies
  • 1 kudos

Resolved! DBFS Rest Api is disabled

Dear Team,I have created a db account using gcp when i tried to create the token and configure databricks cli and tried to connect i get the below errordatabricks fs lserrorError: b'{"error_code":"FEATURE_DISABLED","message":"DBFS Rest Api is disable...

Data Engineering
Reply
  • 1140 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 1 kudos

Hi @Rakesh Reddy Gopidi​ This is a known limitation with DBFS API and GCP. We are planning to redesign the DBFS API and we wanted to not gain more users that we later might need to migrate to a new API. If this is really required for you, please pro...

  • 1 kudos
harri_renney
by • New Contributor
  • 442 Views
  • 0 replies
  • 0 kudos

New Note

The visualisation of workflow tasks doesn't handle particular layouts well. This is because the arrows can pass underneath task blocks, this makes what would be a nice visualisation for exporting into a confusing one.I have provided an image of a set...

Data Engineering
Reply
  • 442 Views
  • 0 replies
  • 0 kudos
HB
by • New Contributor III
  • 2097 Views
  • 4 replies
  • 3 kudos

Resolved! Still missing Badge for Apache Spark 3.0 Associate Dev certification

Hello,I have taken my exam 2 weeks ago and have passed it but I still did not received my badge. I have contacted the support team twice but still no response. Could you please help? Thank you!

Data Engineering
Reply
  • 2097 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
ashok_k_gupta12
New Contributor III
  • 3 kudos

Databricks should fix the certification platform ASAP, currently a user needs to login to multiple different sites to get a certification.Each site has its own login that makes it very difficult to remember. There is not integration or synergy among ...

  • 3 kudos
3 More Replies
rv1
by • New Contributor
  • 3686 Views
  • 0 replies
  • 0 kudos

NULL vs NaN in SQL Mode

In SQL Mode | SQL Editor there seems to be no distinction between NULL and NaN. In some cases it is very misleading as it makes the user to search mistake in the wrong place.DE/DS mode works as expected: UPDATE: a bit later I found this article: http...

image image
Data Engineering
Reply
  • 3686 Views
  • 0 replies
  • 0 kudos
elementalM
by • New Contributor III
  • 2580 Views
  • 5 replies
  • 0 kudos

Resolved! GCP auth time out in long running databricks job

I'm wondering if you can help me with a google auth issue related to structured streaming and long running databricks jobs in general. I will get this error after running for 8+ hours. Any tips on this? GCP auth issues for long running jobs?Caused by...

Data Engineering
Reply
  • 2580 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
elementalM
New Contributor III
  • 0 kudos

Thanks, yes this seems to be the best work around - the good ole retry on fail. Thanks for the help.

  • 0 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels