- 8694 Views
- 4 replies
- 1 kudos
Resolved! Managed table overwrites existing location for delta but not for oth
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 8694 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi,this is how the delta format work. With overwrite you are not deleting the files in the folder or replacing them. Delta is creating a new file with the overwritten schema and data. This way you are also able to return to former versions of the del...
- 1 kudos
- 11517 Views
- 1 replies
- 0 kudos
pyspark dropDuplicates performance issue
Hi,I am trying to delete duplicate records found by key but its very slow. Its continuous running pipeline so data is not that huge but still it takes time to execute this command.df = df.dropDuplicates(["fileName"])Is there any better approach to d...
- 11517 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,I am trying to delete duplicate records found by key but its very slow. Its continuous running pipeline so data is not that huge but still it takes time to execute this command.df = df.dropDuplicates(["fileName"])Is there any better approach to d...
- 0 kudos
- 994 Views
- 1 replies
- 0 kudos
Dashboard from Notebook - How to schedule
notebook is created with insight and have created dashboard (Not a SQL) from it.Need to schedule this. I have tried scheduling by workflow - it only takes you to the notebookeven the schedule from dashboard takes me to the notebook and not the dashbo...
- 994 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Accn , Thanks for bringing up your concerns, always happy to help We understand your concern but right now there is only way to refresh a notebook dashboard is via scheduled jobs. To schedule a dashboard to refresh at a specified interval, click...
- 0 kudos
- 7670 Views
- 1 replies
- 1 kudos
Resolved! After moving mounted s3 bucket under unity catalog control, python file paths no longer work
I have been using a mounted external s3 bucket with json files up until a few days ago, when my company changed to using all file mounts under control of the unity catalog. Suddenly I can no loner run a command like:with open("/mnt/my_files/my_json....
- 7670 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @chrisf_sts ,Thanks for bringing up your concerns, always happy to help May I know which cluster access mode you are using to run the notebook commands? Can you please try to run this below command on Single user cluster access mode. "with open(...
- 1 kudos
- 995 Views
- 0 replies
- 0 kudos
Unable to analyze external table | FileAlreadyExistsException
Hello experts, There's a csv file, "nyc_taxi.csv" saved under users/myfolder on DBFS, and I used this file created 2 tables:1. nyc_taxi : created using the UI, and it appeared as a managed table saved under dbfs:/user/hive/warehouse/mydatabase.db/nyc...
- 995 Views
- 0 replies
- 0 kudos
- 1375 Views
- 1 replies
- 0 kudos
Running unit tests and hyperopt causes a broadcast variable exception
Hello,We are using hyperopt to train a model with relatively large train dataset.We've experience some performance issues and following the suggestions in this notebook, we broadcasted the dataset.To verify that broadcasting the dataset resolved the ...
- 1375 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello,We are using hyperopt to train a model with relatively large train dataset.We've experience some performance issues and following the suggestions in this notebook, we broadcasted the dataset.To verify that broadcasting the dataset resolved the ...
- 0 kudos
- 7962 Views
- 9 replies
- 4 kudos
Resolved! How to refresh a single table in Delta Live Tables?
Suppose I have a Delta Live Tables framework with 2 tables: Table 1 ingests from a json source, Table 2 reads from Table 1 and runs some transformation.In other words, the data flow is json source -> Table 1 -> Table 2. Now if I find some bugs in the...
- 7962 Views
- 9 replies
- 4 kudos
- 4 kudos
Answering my own question: nowadays (February 2024) this can all be done via the UI.When viewing your DLT pipeline there is a "Select tables for refresh" button in the header. If you click this, you can select individual tables, and then in the botto...
- 4 kudos
- 11023 Views
- 3 replies
- 6 kudos
Resolved! How to add I custom logging in Databricks
I want to add custom logs that redirect in the Spark driver logs. Can I use the existing logger classes to have my application logs or progress message in the Spark driver logs.
- 11023 Views
- 3 replies
- 6 kudos
- 6 kudos
1) Is it possible to save all the custom logging to its own file? Currently it is being logging with all other cluster logs (see image) 2) Also Databricks it seems like a lot of blank files are also being created for this. Is this a bug? this include...
- 6 kudos
- 1135 Views
- 1 replies
- 0 kudos
Importing data from S3 to Azure DataBricks Cluster with Unity Catalog in Shared Mode
Environment details:DataBricks on Azure, 13.3 LTS, Unity Catalog, Shared Cluster mode.Currently in the environment I'm in, we run imports from S3 with code like:spark.read.option('inferSchema', 'true').json(s3_path). When running on a cluster in Sha...
- 1135 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Sha, We usually get such error while working with shared cluster mode assuming this your dev environment just to avoid errors, please use different clusters. However as a alternative solution in case if would like to keep the shared cluster the...
- 0 kudos
- 3508 Views
- 4 replies
- 0 kudos
CREATE TABLE does not overwrite location whereas CREATE OR REPLACE TABLE does
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 3508 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Dhruv-22 , Based on the information you shared above, the "CREATE OR REPLACE" and "CREATE" commands in Databricks do have different behaviours, particularly when it comes to handling tables with specific target locations. The "CREATE OR REPLACE"...
- 0 kudos
- 8343 Views
- 1 replies
- 2 kudos
REDACTED_POSSIBLE_SECRET_ACCESS_KEY as part of column value result form aes_encrypt
Hi, i've encountered an error using base64/aes_encrypt, as result the string saved contains 'REDACTED_POSSIBLE_SECRET_ACCESS_KEY' at the end destroying the original data, rendering it useless undecryptable, is there a way to avoid this replacement in...
- 8343 Views
- 1 replies
- 2 kudos
- 2 kudos
I had the same issue, and my usage was similar to OP:base64(aes_encrypt(<clear_text>, unbase64(secret(<scope>, <key>))))Databricks support suggested to not call secret within the insert/update operation that writes to the table. After updating the py...
- 2 kudos
- 3161 Views
- 3 replies
- 1 kudos
Resolved! REPLACE TABLE AS SELECT is not working with parquet whereas it works fine for delta
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 3161 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Dhruv-22 We understand that you are facing the following error when using REPLACE TABLE AS SELECT on the Parquet Table but at this moment the REPLACE TABLE AS SELECT operation you're trying to perform is not supported for Parquet tables. Accord...
- 1 kudos
- 1386 Views
- 2 replies
- 0 kudos
Near Real time Solutioning on data from Core System which gets updated
We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.Problem is that source tab...
- 1386 Views
- 2 replies
- 0 kudos
- 0 kudos
came across this matrix while reading about DLT what is read from complete and write to Incremental means ?
- 0 kudos
- 994 Views
- 2 replies
- 0 kudos
SQL warehourse l in AZURE databricks trial account
Can we create SQL warehourse 2xsmall cluster in AZURE databricks trial account
- 994 Views
- 2 replies
- 0 kudos
- 14989 Views
- 8 replies
- 4 kudos
how can I learn DataBricks
I am currently enrolled in data engineering boot camp. We go over various technologies azure , pyspark , airflow , Hadoop ,nosql,SQL, python. But not over something like databricks. I am in contact with lots of recent graduates who landed a job. Almo...
- 14989 Views
- 8 replies
- 4 kudos
- 4 kudos
I'd be glad to help you on your journey to learning Databricks! Whether you're a beginner or aiming to advance your skills, here's a comprehensive guide:Foundations:Solid understanding of core concepts: Begin with foundational knowledge in big data,...
- 4 kudos