- 6840 Views
- 7 replies
- 0 kudos
Incremental Data copy from one SQL DB to another DB
Hi All,I have 20 tables in source sql DB and we need to create pipeline to incrementally load data into target database .Can some one please suggest me best approach to achieve this using Azure Databricks please?Should i use merge Into ? Copy Into? o...
- 6840 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @SUDHANSHU RAJ​,Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 0 kudos
- 12801 Views
- 1 replies
- 0 kudos
VScode extension - certificate signature failure
Hi everyone,I'm trying to use the new Databricks extension (v0.3.10) for VS code (v1.77.3).I face this problem when connecting to our workspace:This problem persists when I tried to login through az CLI with our SSO, or through local config using PAT...
- 12801 Views
- 1 replies
- 0 kudos
- 0 kudos
@Minh Duc Nguyen​ :It seems like the error you are facing is due to a failure in verifying the SSL certificate of your Databricks workspace. To resolve this, you need to add the custom CA certificate to your VS Code settings. Here's how you can do it...
- 0 kudos
- 6065 Views
- 3 replies
- 1 kudos
Resolved! Databricks CLI v0.17.6 issue
I do have multiple groups created in my databricks account and I have databricks cli installed on my mac. Some of the cli functions return errors that i cannot find solution for. databricks groups listReturns:Error: b'{"error_code":"INTERNAL_ERROR","...
- 6065 Views
- 3 replies
- 1 kudos
- 1 kudos
@kenan hasanov​ which version python you have installed on your machine please, you need to have 3-3.6 or 2-2.7.9 above , please try to go with latest one as you are only seeing issues with few functions. please raise issue in case if you are still f...
- 1 kudos
- 3736 Views
- 3 replies
- 0 kudos
- 3736 Views
- 3 replies
- 0 kudos
- 0 kudos
I am not able to. I get that error when I log in. and this one below if I click on any tab on the panel
- 0 kudos
- 1532 Views
- 2 replies
- 0 kudos
Need help recovering password
guys, i cannot recover my password at my databricks workspace. I tried to reset my password by using the reset my password buttons in both login and admin console, but i am not getting the reset email. I already checked my spam box and i'm getting em...
- 1532 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Victor Fernandes​ Just a friendly follow-up, Are you still looking for help regarding this issue? please let us know
- 0 kudos
- 3396 Views
- 2 replies
- 0 kudos
Lakehouse Architecture : How notebooks are organized and executed
We are implementing a lakehouse architecture and using Notebook to transform data from object storage. Most of the time our source is database for which there one folder per table in object storage.We have structure like below for various notebooksGO...
- 3396 Views
- 2 replies
- 0 kudos
- 0 kudos
Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help with your request
- 0 kudos
- 3275 Views
- 5 replies
- 1 kudos
I have completed Data Engineer Associate (Version 3) Certificate. when we will get the Certificate in default?
I have completed Data Engineer Associate (Version 3) Certificate.when we will get the Certificate in default?
- 3275 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @sherbin w​ Just a friendly follow-up, Are you still looking for help regarding this issue or you were able to received your voucher? please let us know
- 1 kudos
- 4069 Views
- 2 replies
- 1 kudos
Unable to login databricks
I am not able to login databricks academy, getting this error please help.
- 4069 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Renuka Bhutada​ Just a friendly follow-up, Are you still looking for help regarding this issue? please let us know
- 1 kudos
- 3239 Views
- 3 replies
- 0 kudos
Delta table cannot be previewed in the Data UI
I have saved a delta table which I can read using `spark.table(table_name)`. However, when I go to the "Data" panel in databricks I get an error when I choose that table.An error occurred while fetching table: <table_name>com.databricks.backend.commo...
- 3239 Views
- 3 replies
- 0 kudos
- 0 kudos
I don't know. The way I have saved the table was with autoloader as follows:​I am saving a structure stream into a table using:``` .writeStream .format("delta") # <----------- .option("checkpointLocation", checkpoint_path) .option("path", ou...
- 0 kudos
- 3656 Views
- 3 replies
- 2 kudos
Voucher not sent
I have concluded the Databricks Platform Administrator - ILT Training, that offered a $200 dollars voucher after completion. I answered the questions of the survey and sent my certificate. How can I get the code?
- 3656 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Thais Arantes​,Just a friendly follow-up, Are you still looking for help regarding this issue or you were able to received your voucher? please let us know
- 2 kudos
- 6217 Views
- 4 replies
- 2 kudos
How can I store timestamps using the local (not offset) timestamp instead of storing as UTC-Offset?
I have data that looks like this:2021-11-25T19:00:00.000-0500 2021-03-03T13:00:00.000-0500 2021-03-09T15:00:00.000-0500 2021-03-13T16:00:00.000-0500 2021-03-19T03:00:00.000-0400 2021-05-28T03:00:00.000-0400which is accurate, except I'm pulling the da...
- 6217 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Jonathan Dufault​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
- 2 kudos
- 13138 Views
- 2 replies
- 0 kudos
ignoreDeletes in DLT pipeline
Hi all,I have a DLT pipeline as so:raw -> cleansed (SCD2) -> curated. 'Raw' is utilizing autoloader, to continously read file from a datalake. These files can contain tons of duplicate, which causes our raw table to become quite large. Therefore, we ...
- 13138 Views
- 2 replies
- 0 kudos
- 0 kudos
Ok, i'll try an add additional details. Firstly: The diagram below shows our current dataflow: Our raw table is defined as such: TABLES = ['table1','table2'] def generate_tables(table_name): @dlt.table( name=f'raw_{table_name}', table_pro...
- 0 kudos
- 7912 Views
- 8 replies
- 1 kudos
Why is Delta Lake creating a 238.0TiB shuffle on merge?
I'm frankly at a loss here. I have a task that is consistently performing just awfully. I took some time this morning to try and debug it and the physical plan is showing a 238TiB shuffle:== Physical Plan == AdaptiveSparkPlan (40) +- == Current Plan...
- 7912 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @Jordan Yaker​,Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 1 kudos
- 30738 Views
- 10 replies
- 39 kudos
Why Python logs shows [REDACTED] literal in spaces when I use dbutils.secrets.get in my code?
When I use dbutils.secrets.get in my code, spaces in the log are replaced by "[REDACTED]" literal. This is very annoying and makes the log reading difficult. Any idea how to avoid this?See my screenshot...
- 30738 Views
- 10 replies
- 39 kudos
- 39 kudos
I ran into the same issue and found that the reason was that the notebook included some test keys with values of "A" and "B" for simple testing. I noticed that any string with a substring of "A" or "B" was "[REDACTED]".​So, in my case, it was an eas...
- 39 kudos
- 13227 Views
- 1 replies
- 0 kudos
Azure Databricks cluster driver config
Hi there,I am trying to set up databricks storage account access in Global init script. according to Azure Databricks document on creating cluster with driver config for all clusters (https://learn.microsoft.com/en-us/azure/databricks/archive/compute...
- 13227 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 562 | |
| 349 | |
| 287 |
