- 6557 Views
- 5 replies
- 5 kudos
How to reduce time while loading data into the azure synapse table?
Hi All,I just wanted to know if is there any option to reduce time while loading Pyspark Dataframe into the Azure synapse table using Databricks.like..I have a pyspark dataframe that has around 40k records and I am trying to load data into the azure ...
- 6557 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @Tinendra Kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 5 kudos
- 10377 Views
- 3 replies
- 3 kudos
Resolved! Impacts of running multiple jobs in parallel that refers the same notebook
Can I run multiple jobs(for example: 100+) in parallel that refers the same notebook? I supply each job with a different parameter. If we can do this, what would be the impact? (for example: reliability, performance, troubleshooting etc. )Example: N...
- 10377 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Murthy Ramalingam​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 3 kudos
- 4642 Views
- 4 replies
- 3 kudos
Resolved! Databricks Design Help
Hello Community,I am currently working on populating gold layer tables. Source for these gold layer tables are silver layer tables. A query is going to run on silver layer tables, spark sql query contains joins between multiple tables.ex:select colum...
- 4642 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @bodempudi venkat​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 3 kudos
- 4864 Views
- 2 replies
- 0 kudos
Data in transit security when using jdbc connections.
When using spark.read.jdbc() for a direct connection, how is data in transit secured?
- 4864 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Melody Mazaiwana​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 0 kudos
- 4501 Views
- 3 replies
- 4 kudos
Internal error message: Failed to launch spark container on instance i-xxxxxxxxxxxxx. Exception: Unexpected internal error, please contact Databricks support
I have a docker image for debezium in my ECR repo and added IAM roles from Databricks to pull this image for my cluster and seeing this error when cluster is created
- 4501 Views
- 3 replies
- 4 kudos
- 4 kudos
I have been trying for more than a week now..... where is the logs for these pulls,....error message is not helpful
- 4 kudos
- 4872 Views
- 1 replies
- 0 kudos
Resolved! Pros and cons - running SQL query in databricks notebook and serverless warehouse sql editor
Can you please help technical pros and cons for running SQL query in databricks notebook (data engineering workspace) and serverless warehouse SQL editor
- 4872 Views
- 1 replies
- 0 kudos
- 0 kudos
NotebookPROS: More traditional cluster, git integration, choose DBR versionCONS: Cluster startup time, photon not automatically part of the clusterServerlessPROS: Faster, almost immediate startup time, less expensive for a single query, photon enable...
- 0 kudos
- 6351 Views
- 6 replies
- 1 kudos
Resolved! Maven Package install failing on DBR 11.3 LTS
Hi Databricks Community,I ran into the following issue when setting up a new cluster with the latest LTS Databricks runtime (11.3). When trying to install the package with the coordinates com.microsoft.azure.kusto:kusto-spark_3.0_2.12:3.1.4 from Mave...
- 6351 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Andrei Bondarenko​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 1 kudos
- 8737 Views
- 8 replies
- 5 kudos
customer is trying to generate a Databricks token for a service principal (SP). They’ve created the SP in Azure AD and have used the Databricks rest ...
customer is trying to generate a Databricks token for a service principal (SP). They’ve created the SP in Azure AD and have used the Databricks rest api to add it as an admin. When using the Databricks rest api /api/2.0/token-management/on-behalf-of...
- 8737 Views
- 8 replies
- 5 kudos
- 3525 Views
- 2 replies
- 0 kudos
Delay in retrieving task variables from a notebook utilizing %run
I have a databricks workflow where the first task is to set up task parameters for other notebooks to use later in the process. Since these are variables that are used in all of my notebooks, I have opted to assign them in a shared notebook and call ...
- 3525 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Please refer: https://docs.databricks.com/notebooks/notebook-workflows.htmlCould you please provide the error received?
- 0 kudos
- 22897 Views
- 13 replies
- 13 kudos
How to deploy Azure Databricks Sql (Table, Views, etc) into higher environments?
We have build multiple tables and Views under Databricks Sql and unable to figure out how can we take this code and deploy at our higher environments? Need some guidance as we're unable to get any information from searched documentations.
- 22897 Views
- 13 replies
- 13 kudos
- 13 kudos
Hi, I agree with @Josef Prakljacic​ . If Databricks would like to compete SQL DWH/Synapse or Snowflake and target DWH users, It should prepare some guidelines how to manage "database" objects. Yea, @Werner Stinckens​ with Engineering workspace and py...
- 13 kudos
- 6266 Views
- 2 replies
- 1 kudos
How to write to Salesforce object using Spark Salesforce Library
Hi I'm facing an issue when writing to a salesforce object. I'm using the springml/spark-salesforce library. I have the above libraries installed as recommended based on my research.I try to write like this:(_sqldf .write .format("com.springml.spar...
- 6266 Views
- 2 replies
- 1 kudos
- 1 kudos
Im facing the same issue while trying to write to Salesforce, if you have found a resolution could you please share it ?
- 1 kudos
- 6350 Views
- 4 replies
- 1 kudos
Delta Live Tables: reading from output
I'm trying to implement an incremental ingestion logic in the following way:database tables have DbUpdatedDate columnDuring initial load I perform a full copy of the database tableDuring incremental load I:scan the data already in the DLT to see what...
- 6350 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Chris Nawara​, I had the same issue you had. I was trying to avoid the apply_changes but we in the end I implemented it and I'm happier that I expected heheand if you have any additional standardization columns that you need to implement, you can...
- 1 kudos
- 4206 Views
- 2 replies
- 3 kudos
Resolved! accidently delete paquet file in dbfs
I accidently delete manual paquet file in dbfs how can I recovery this recovery this file
- 4206 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @pansiri panaudom​ ,There is no option restore deleted files in databricks .
- 3 kudos
- 4252 Views
- 1 replies
- 1 kudos
Resolved! How to query Databricks audit logs?
Hi,I would like to ask where the Databricks Audit Log files are stored on the DBFS.And is there any way that I can query log files?Thanks.
- 4252 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Mohammad Saber​ ,I think first you need to configure audit log in databricks then you use it.Please refer below blog that will help you in this.Configure audit logging | Databricks on AWS
- 1 kudos
- 3573 Views
- 4 replies
- 1 kudos
Need help to understand Databricks workspace service principle token expire calculate
Hi Team, Need assistance to understand Databricks workspace service principle token expire calculation. Issue : when I am creating a token I have given lifetime =3600, but when I doing get token I am getting unexpected expiry number and even when I ...
- 3573 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Team, Please help on my issue,Is there any way to find expiry of token, i mean still how much time have token to expiry. creation_time - expiry_time is not giving me exact output.Kindly let me know if there is any way to find as soon as possibleT...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
