- 2145 Views
- 2 replies
- 0 kudos
Resolved! When does delta lake actually compute a table?
Maybe I'm completely wrong, but from my understanding delta lake only calculates a table at certain points, for instance when you display your data. Before that point, operations are only written to the log file and are not executed (meaning no chang...
- 2145 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Lukas Goldschmied​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 0 kudos
- 16069 Views
- 5 replies
- 0 kudos
Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"
This is the error which is coming while processing concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.. What are the o...
- 16069 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Abhishek Dutta​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 0 kudos
- 2991 Views
- 3 replies
- 0 kudos
Resolved! Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day.
Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day. For example , let’s assume we are doing multiple operation on table in a day every minute and want to keep time travel...
- 2991 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @vinay kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 0 kudos
- 5332 Views
- 4 replies
- 3 kudos
Resolved! how does databricks time travel work
Hi, Since it is not very well explained, I want to know if the table history is a snapshot of the whole table at that point of time containing all the data or it tracks only some metadata of the table changes.To be more precise : if I have a table in...
- 5332 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @data engineer​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so...
- 3 kudos
- 6531 Views
- 6 replies
- 7 kudos
Resolved! How can I delete a file in DBFS with Illegal character?
How can I delete a file in DBFS with Illegal character?Someone put the file named "planejamento_[4098.]___SHORT_SAIA_JEANS__.xlsx" inside the folder /FileStore and I can delete it, because of this error: java.net.URISyntaxException: Illegal character...
- 6531 Views
- 6 replies
- 7 kudos
- 7 kudos
try this %sh ls -li /dbfsif the file is located in a subdirectory you can change the path mentioned above.the %sh magic command gives you access to linux shell commands.
- 7 kudos
- 7642 Views
- 2 replies
- 0 kudos
Resolved! Pool Max Capacity vs Cluster Max Workers
Hi all, we have a databricks instance on Azure with a Compute Cluster version 7.3 LTS. Currently the cluster has 4 max workers (min workers: 1) of type: Standard_D13_v2 and 1 driver of the same type. There are several jobs that are running on this cl...
- 7642 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @EDDatabricks EDDatabricks​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear f...
- 0 kudos
- 5295 Views
- 5 replies
- 5 kudos
How to reduce time while loading data into the azure synapse table?
Hi All,I just wanted to know if is there any option to reduce time while loading Pyspark Dataframe into the Azure synapse table using Databricks.like..I have a pyspark dataframe that has around 40k records and I am trying to load data into the azure ...
- 5295 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @Tinendra Kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 5 kudos
- 9086 Views
- 3 replies
- 3 kudos
Resolved! Impacts of running multiple jobs in parallel that refers the same notebook
Can I run multiple jobs(for example: 100+) in parallel that refers the same notebook? I supply each job with a different parameter. If we can do this, what would be the impact? (for example: reliability, performance, troubleshooting etc. )Example: N...
- 9086 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Murthy Ramalingam​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 3 kudos
- 3675 Views
- 4 replies
- 3 kudos
Resolved! Databricks Design Help
Hello Community,I am currently working on populating gold layer tables. Source for these gold layer tables are silver layer tables. A query is going to run on silver layer tables, spark sql query contains joins between multiple tables.ex:select colum...
- 3675 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @bodempudi venkat​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 3 kudos
- 3713 Views
- 2 replies
- 0 kudos
Data in transit security when using jdbc connections.
When using spark.read.jdbc() for a direct connection, how is data in transit secured?
- 3713 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Melody Mazaiwana​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 0 kudos
- 3621 Views
- 3 replies
- 4 kudos
Internal error message: Failed to launch spark container on instance i-xxxxxxxxxxxxx. Exception: Unexpected internal error, please contact Databricks support
I have a docker image for debezium in my ECR repo and added IAM roles from Databricks to pull this image for my cluster and seeing this error when cluster is created
- 3621 Views
- 3 replies
- 4 kudos
- 4 kudos
I have been trying for more than a week now..... where is the logs for these pulls,....error message is not helpful
- 4 kudos
- 3865 Views
- 1 replies
- 0 kudos
Resolved! Pros and cons - running SQL query in databricks notebook and serverless warehouse sql editor
Can you please help technical pros and cons for running SQL query in databricks notebook (data engineering workspace) and serverless warehouse SQL editor
- 3865 Views
- 1 replies
- 0 kudos
- 0 kudos
NotebookPROS: More traditional cluster, git integration, choose DBR versionCONS: Cluster startup time, photon not automatically part of the clusterServerlessPROS: Faster, almost immediate startup time, less expensive for a single query, photon enable...
- 0 kudos
- 5098 Views
- 6 replies
- 1 kudos
Resolved! Maven Package install failing on DBR 11.3 LTS
Hi Databricks Community,I ran into the following issue when setting up a new cluster with the latest LTS Databricks runtime (11.3). When trying to install the package with the coordinates com.microsoft.azure.kusto:kusto-spark_3.0_2.12:3.1.4 from Mave...
- 5098 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Andrei Bondarenko​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 1 kudos
- 7138 Views
- 8 replies
- 5 kudos
customer is trying to generate a Databricks token for a service principal (SP). They’ve created the SP in Azure AD and have used the Databricks rest ...
customer is trying to generate a Databricks token for a service principal (SP). They’ve created the SP in Azure AD and have used the Databricks rest api to add it as an admin. When using the Databricks rest api /api/2.0/token-management/on-behalf-of...
- 7138 Views
- 8 replies
- 5 kudos
- 2895 Views
- 2 replies
- 0 kudos
Delay in retrieving task variables from a notebook utilizing %run
I have a databricks workflow where the first task is to set up task parameters for other notebooks to use later in the process. Since these are variables that are used in all of my notebooks, I have opted to assign them in a shared notebook and call ...
- 2895 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Please refer: https://docs.databricks.com/notebooks/notebook-workflows.htmlCould you please provide the error received?
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
