cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
MelMazz
by • New Contributor
  • 2715 Views
  • 2 replies
  • 0 kudos

Data in transit security when using jdbc connections.

When using spark.read.jdbc() for a direct connection, how is data in transit secured?

Data Engineering
Reply
  • 2715 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Melody Mazaiwana​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 0 kudos
1 More Replies
167559
by • New Contributor III
  • 2761 Views
  • 3 replies
  • 4 kudos

Internal error message: Failed to launch spark container on instance i-xxxxxxxxxxxxx. Exception: Unexpected internal error, please contact Databricks support

I have a docker image for debezium in my ECR repo and added IAM roles from Databricks to pull this image for my cluster and seeing this error when cluster is created

Data Engineering
Reply
  • 2761 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
167559
New Contributor III
  • 4 kudos

I have been trying for more than a week now..... where is the logs for these pulls,....error message is not helpful

  • 4 kudos
2 More Replies
vinaykumar
by • New Contributor III
  • 2951 Views
  • 1 replies
  • 0 kudos

Resolved! Pros and cons - running SQL query in databricks notebook and serverless warehouse sql editor

Can you please help technical pros and cons for running SQL query in databricks notebook (data engineering workspace) and serverless warehouse SQL editor

Data Engineering
Reply
  • 2951 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

NotebookPROS: More traditional cluster, git integration, choose DBR versionCONS: Cluster startup time, photon not automatically part of the clusterServerlessPROS: Faster, almost immediate startup time, less expensive for a single query, photon enable...

  • 0 kudos
petehart92
by • New Contributor II
  • 4516 Views
  • 5 replies
  • 6 kudos

Error While Rendering Visualization -- Map (Markers)

I have a table with latitude and longitude for a few addresses (no more than 10 at the moment) but when I select the appropriate columns in the visualization editor for Map (Markers) I get an message that states "error while rendering visualization"....

Not a lot of detail...
Data Engineering
Reply
  • 4516 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
SawyKoop_2865
New Contributor II
  • 6 kudos

Seeing the same issue

  • 6 kudos
4 More Replies
ABVectr
by • New Contributor III
  • 3737 Views
  • 6 replies
  • 1 kudos

Resolved! Maven Package install failing on DBR 11.3 LTS

Hi Databricks Community,I ran into the following issue when setting up a new cluster with the latest LTS Databricks runtime (11.3). When trying to install the package with the coordinates com.microsoft.azure.kusto:kusto-spark_3.0_2.12:3.1.4 from Mave...

Data Engineering
Reply
  • 3737 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Andrei Bondarenko​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 1 kudos
5 More Replies
User16270906190
by • New Contributor III
  • 5481 Views
  • 8 replies
  • 5 kudos

customer is trying to generate a Databricks token for a service principal (SP). They’ve created the SP in Azure AD and have used the Databricks rest ...

customer is trying to generate a Databricks token for a service principal (SP). They’ve created the SP in Azure AD and have used the Databricks rest api to add it as an admin. When using the Databricks rest api /api/2.0/token-management/on-behalf-of...

Data Engineering
Reply
  • 5481 Views
  • 8 replies
  • 5 kudos
View Replies
Latest Reply
Buxert
New Contributor II
  • 5 kudos

Having the same problem here. @Nitisha Nigam​ did you solve it?

  • 5 kudos
7 More Replies
cmilligan
by • Contributor II
  • 1982 Views
  • 2 replies
  • 0 kudos

Delay in retrieving task variables from a notebook utilizing %run

I have a databricks workflow where the first task is to set up task parameters for other notebooks to use later in the process. Since these are variables that are used in all of my notebooks, I have opted to assign them in a shared notebook and call ...

Data Engineering
Reply
  • 1982 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Please refer: https://docs.databricks.com/notebooks/notebook-workflows.htmlCould you please provide the error received?

  • 0 kudos
1 More Replies
Jayesh
by • New Contributor III
  • 14958 Views
  • 13 replies
  • 13 kudos

How to deploy Azure Databricks Sql (Table, Views, etc) into higher environments?

We have build multiple tables and Views under Databricks Sql and unable to figure out how can we take this code and deploy at our higher environments? Need some guidance as we're unable to get any information from searched documentations.

Data Engineering
Reply
  • 14958 Views
  • 13 replies
  • 13 kudos
View Replies
Latest Reply
mmlime
New Contributor III
  • 13 kudos

Hi, I agree with @Josef Prakljacic​ . If Databricks would like to compete SQL DWH/Synapse or Snowflake and target DWH users, It should prepare some guidelines how to manage "database" objects. Yea, @Werner Stinckens​ with Engineering workspace and py...

  • 13 kudos
12 More Replies
theSoyf
by • New Contributor II
  • 4361 Views
  • 2 replies
  • 1 kudos

How to write to Salesforce object using Spark Salesforce Library

Hi I'm facing an issue when writing to a salesforce object. I'm using the springml/spark-salesforce library. I have the above libraries installed as recommended based on my research.I try to write like this:(_sqldf .write .format("com.springml.spar...

Screen Shot 2022-12-14 at 8.18.07 AM
Data Engineering
Reply
  • 4361 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Gauthy
New Contributor II
  • 1 kudos

Im facing the same issue while trying to write to Salesforce, if you have found a resolution could you please share it ?

  • 1 kudos
1 More Replies
knawara
by • Contributor
  • 3801 Views
  • 4 replies
  • 1 kudos

Delta Live Tables: reading from output

I'm trying to implement an incremental ingestion logic in the following way:database tables have DbUpdatedDate columnDuring initial load I perform a full copy of the database tableDuring incremental load I:scan the data already in the DLT to see what...

Data Engineering
Reply
  • 3801 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
fecavalc08
New Contributor III
  • 1 kudos

Hi @Chris Nawara​, I had the same issue you had. I was trying to avoid the apply_changes but we in the end I implemented it and I'm happier that I expected heheand if you have any additional standardization columns that you need to implement, you can...

  • 1 kudos
3 More Replies
Bie1234
by • New Contributor III
  • 2531 Views
  • 2 replies
  • 3 kudos

Resolved! accidently delete paquet file in dbfs

I accidently  delete manual paquet file in dbfs how can I recovery this recovery this file

Data Engineering
Reply
  • 2531 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 3 kudos

Hi @pansiri panaudom​ ,There is no option restore deleted files in databricks .

  • 3 kudos
1 More Replies
Mado
by • Valued Contributor II
  • 2377 Views
  • 1 replies
  • 1 kudos

Resolved! How to query Databricks audit logs?

Hi,I would like to ask where the Databricks Audit Log files are stored on the DBFS.And is there any way that I can query log files?Thanks.

Data Engineering
Reply
  • 2377 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 1 kudos

Hi @Mohammad Saber​ ,I think first you need to configure audit log in databricks then you use it.Please refer below blog that will help you in this.Configure audit logging | Databricks on AWS

  • 1 kudos
Vinayak_s
by • New Contributor II
  • 1895 Views
  • 4 replies
  • 1 kudos

Need help to understand Databricks workspace service principle token expire calculate

Hi Team, Need assistance to understand Databricks workspace service principle token expire calculation. Issue : when I am creating a token I have given lifetime =3600, but when I doing get token I am getting unexpected expiry number and even when I ...

Data Engineering
Reply
  • 1895 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Vinayak_s
New Contributor II
  • 1 kudos

Hi Team, Please help on my issue,Is there any way to find expiry of token, i mean still how much time have token to expiry. creation_time - expiry_time  is not giving me exact output.Kindly let me know if there is any way to find as soon as possibleT...

  • 1 kudos
3 More Replies
vinaykumar
by • New Contributor III
  • 4684 Views
  • 3 replies
  • 1 kudos

Resolved! Run databricks job instantly without waiting job cluster get active

when we run databricks job it take some time to get job cluster active . I created pool also and attached with job cluster but still it take time to attached the cluster and job cluster get active to start the job run. is there any way - we can run d...

Data Engineering
Reply
  • 4684 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

If you want instant processing, you will have to have a cluster running all the time.As mentioned above, Databricks is testing serverless compute for data engineering workloads (comparable to serverless SQL). This fires up a cluster in a few seconds...

  • 1 kudos
2 More Replies
tinendra
by • New Contributor III
  • 4316 Views
  • 7 replies
  • 8 kudos

Can we run pandas dataframe inside databricks?

Hi, I want to run df=pd.read_csv('/dbfs/FileStore/airlines1.csv') while trying to run getting error likeFileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/airlines1.csv'Could you please help me out how to run pandas dataframe in...

Data Engineering
Reply
  • 4316 Views
  • 7 replies
  • 8 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 8 kudos

Hi @Tinendra Kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 8 kudos
6 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels