cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Trey
by • New Contributor III
  • 2069 Views
  • 2 replies
  • 3 kudos

Where do you usually store and manage "JDBC credentials" to use on databricks notebook?

Hi all,I would like to improve the way I use JDBC credenditial information (ID/PW, host, port, etc)Where do you guys usually store and use the jdbc credentials?Thanks for your help in advance!

Data Engineering
Reply
  • 2069 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Kwangwon Yi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 3 kudos
1 More Replies
RohitKulkarni
by • Contributor II
  • 2432 Views
  • 3 replies
  • 4 kudos

TCP/IP Issue

Hello Team,I am trying to read the data from Synapse analytics from databricks.The below is the Query to read the table.%python### Read from Azure Synapse table via spark.read.load df = spark.read \ .format("com.databricks.spark.sqldw") \ .option("ur...

Data Engineering
Reply
  • 2432 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 4 kudos
2 More Replies
db-avengers2rul
by • Contributor II
  • 1483 Views
  • 1 replies
  • 1 kudos

Resolved! DBFS Rest Api is disabled

Dear Team,I have created a db account using gcp when i tried to create the token and configure databricks cli and tried to connect i get the below errordatabricks fs lserrorError: b'{"error_code":"FEATURE_DISABLED","message":"DBFS Rest Api is disable...

Data Engineering
Reply
  • 1483 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 1 kudos

Hi @Rakesh Reddy Gopidi​ This is a known limitation with DBFS API and GCP. We are planning to redesign the DBFS API and we wanted to not gain more users that we later might need to migrate to a new API. If this is really required for you, please pro...

  • 1 kudos
harri_renney
by • New Contributor
  • 544 Views
  • 0 replies
  • 0 kudos

New Note

The visualisation of workflow tasks doesn't handle particular layouts well. This is because the arrows can pass underneath task blocks, this makes what would be a nice visualisation for exporting into a confusing one.I have provided an image of a set...

Data Engineering
Reply
  • 544 Views
  • 0 replies
  • 0 kudos
HB
by • New Contributor III
  • 2642 Views
  • 4 replies
  • 3 kudos

Resolved! Still missing Badge for Apache Spark 3.0 Associate Dev certification

Hello,   I have taken my exam 2 weeks ago and have passed it but I still did not received my badge. I have contacted the support team twice but still no response.   Could you please help?   Thank you!

Data Engineering
Reply
  • 2642 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
ashok_k_gupta12
New Contributor III
  • 3 kudos

Databricks should fix the certification platform ASAP, currently a user needs to login to multiple different sites to get a certification.Each site has its own login that makes it very difficult to remember. There is not integration or synergy among ...

  • 3 kudos
3 More Replies
rv1
by • New Contributor
  • 8387 Views
  • 0 replies
  • 0 kudos

NULL vs NaN in SQL Mode

In SQL Mode | SQL Editor there seems to be no distinction between NULL and NaN. In some cases it is very misleading as it makes the user to search mistake in the wrong place.DE/DS mode works as expected: UPDATE: a bit later I found this article: http...

image image
Data Engineering
Reply
  • 8387 Views
  • 0 replies
  • 0 kudos
elementalM
by • New Contributor III
  • 3165 Views
  • 5 replies
  • 0 kudos

Resolved! GCP auth time out in long running databricks job

I'm wondering if you can help me with a google auth issue related to structured streaming and long running databricks jobs in general. I will get this error after running for 8+ hours. Any tips on this? GCP auth issues for long running jobs?Caused by...

Data Engineering
Reply
  • 3165 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
elementalM
New Contributor III
  • 0 kudos

Thanks, yes this seems to be the best work around - the good ole retry on fail. Thanks for the help.

  • 0 kudos
4 More Replies
jwilliam
by • Contributor
  • 2058 Views
  • 2 replies
  • 2 kudos

Resolved! Does libraries installation happen on Data Plane or Control Plane?

Currently, when I install libraries on my clusters. This errors happens:WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol...

Data Engineering
Reply
  • 2058 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Sivaprasad1
Valued Contributor II
  • 2 kudos

@John William​ : Yeah that's true. All the clusters will be residing in the data plane.

  • 2 kudos
1 More Replies
Bency
by • New Contributor III
  • 12604 Views
  • 1 replies
  • 2 kudos

Queries with streaming sources must be executed with writeStream.start();

When I try to perform some transformations on a streaming data , I get Queries with streaming sources must be executed with writeStream.start(); error My aim is to do a lookup for every column in each rows in the streaming data . steaming_table=spark...

Data Engineering
Reply
  • 12604 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 2 kudos

Hi @Bency Mathew​ You can use forEachBatch to perform the custom logic on each microbatch. Please refer to below document:https://docs.databricks.com/structured-streaming/foreach.html#perform-streaming-writes-to-arbitrary-data-sinks-with-structured-s...

  • 2 kudos
dceman
by • New Contributor
  • 1847 Views
  • 0 replies
  • 0 kudos

Databricks with CloudWatch metrics without Instanceid dimension

I have jobs running on job clusters. And I want to send metrics to the CloudWatch. I set CW agent followed this guide.But issue is that I can't create useful metrics dashboard and alarms because I always have InstanceId dimension, and InstanceId is d...

image
Data Engineering
Reply
  • 1847 Views
  • 0 replies
  • 0 kudos
477061
by • Contributor
  • 3090 Views
  • 3 replies
  • 0 kudos

Resolved! Renamed table cannot be written to or deleted from

I have renamed a table, however on trying to write to it (or delete from it) I get the following error: `java.io.FileNotFoundException: No such file or directory: s3a://.../hive/warehouse/testing.db/renamed_table_name/_delta_log/00000000000000000002....

Data Engineering
Reply
  • 3090 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 0 kudos

Hi @477061 Could you please try to test it in DBR 11.1 and see if the issue persists for you?

  • 0 kudos
2 More Replies
Taha_Hussain
by • Databricks Employee
  • 2355 Views
  • 2 replies
  • 6 kudos

Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMT Databricks Office Hours connects you directly with exper...

Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMTDatabricks Office Hours connects you directly with experts to answer your Databricks questions.Join us to:• Troubleshoot your technical questions• Learn the ...

Data Engineering
Reply
  • 2355 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
Taha_Hussain
Databricks Employee
  • 6 kudos

Cont...Q: Do generated columns in Delta Live Tables include IDENTITY columns?A: My understanding is that generated columns in Delta Live Tables do not contain IDENTITY columns. Here is more on generated columns in DLT.Q: We store raw data for each cu...

  • 6 kudos
1 More Replies
Invincible
by • New Contributor
  • 1942 Views
  • 2 replies
  • 2 kudos

Can we run multiple jobs in parallel on one cluster?

Data Engineering
Reply
  • 1942 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 2 kudos

Hi @Pankaj Sharma​ Yes, you can run multiple jobs on one cluster if you choose an all-purpose cluster to run your jobs in Databricks.You can understand more about the clusters in the below document:https://docs.databricks.com/clusters/index.html

  • 2 kudos
1 More Replies
databricksuser2
by • New Contributor II
  • 1439 Views
  • 1 replies
  • 2 kudos

Structured streaming job sees throughput being capped after running normally for a few days

The job (written in PySpark) uses azure eventhub as source and use Databricks delta table as sink. The job is hosted in Azure Databricks.Transformation part is simple, the message body is converted from bytes to json string, the json string is then a...

figure 1
Data Engineering
Reply
  • 1439 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 2 kudos

Hi @Databricks User10293847​ You can try using auto-inflate and let the TU increase automatically. The feature then scales automatically to the maximum limit of TUs you need, depending on the increase in your traffic. You can check the below doc: htt...

  • 2 kudos
ef-zee
by • New Contributor III
  • 15470 Views
  • 3 replies
  • 7 kudos

How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?

I am trying to execute a DLT pipeline, but I am getting an error which says - "INVALID_PARAMETER_VALUE: The field 'node_type_id' cannot be supplied when an instance pool ID is provided."I am using my company's Azure Databricks platform with premium b...

Data Engineering
Reply
  • 15470 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 7 kudos

Do you have cluster ACL enabled?

  • 7 kudos
2 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels