- 3856 Views
- 2 replies
- 5 kudos
Resolved! Databricks SQL External Connections
Lakehouse architectures seem enticing, especially from the standpoint of querying the data lake directly as it sits (as opposed to first migrating the data to an external data warehouse). While documentation and support seems pretty clear regarding ...
- 3856 Views
- 2 replies
- 5 kudos
- 3319 Views
- 1 replies
- 1 kudos
Resolved! python 3.10 custom docker image
Goal: To use Python 3.10.4+Why: We have Python repos that are not backward compatible we want to use.What: I have created an image from Databricks example experimental containers already with Ubuntu 22.04 (2 major versions newer than the curre...
- 3319 Views
- 1 replies
- 1 kudos
- 1 kudos
After searching for an hour, I realized what I needed to look for. It's the importing the Iterable from collections, which is deprecated in 3.10. I guess Databricks hasn't migrated code, yet. In which case, I'm at a cross-roads. Databricks 3.9, local...
- 1 kudos
- 2487 Views
- 2 replies
- 1 kudos
Resolved! How do I reduce the size of a hive table's S3 bucket
I have a hive table in Delta format with over 1B rows, when I check the Data Explorer in the SQL section of Databricks it notes that the table size is 139.3GiB with 401 files but when I check the S3 bucket where the files are located (dbfs:/user/hive...
- 2487 Views
- 2 replies
- 1 kudos
- 1 kudos
When you run updates, deletes etc on a delta table, new files are created. However, the old files are not automatically deleted. This is to allow for features like time travel on the Delta tables. In order to delete older files for a delta table, you...
- 1 kudos
- 1419 Views
- 1 replies
- 5 kudos
Exciting news for Databricks users! #databricks launched a new feature that allows users to run job workflows continuously. Setting up a continuous jo...
Exciting news for Databricks users! #databricks launched a new feature that allows users to run job workflows continuously. Setting up a continuous job workflow is straightforward: create a job and select the continuous trigger option in the scheduli...
- 1419 Views
- 1 replies
- 5 kudos
- 5 kudos
Thank you for sharing!!!
- 5 kudos
- 1821 Views
- 1 replies
- 1 kudos
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notesFebruary 2...
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notesFebruary 21 - 28, 2023Ray on Databricks (Public Preview)With Databricks Runtime 12.0 and above, you can create ...
- 1821 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for sharing!!!
- 1 kudos
- 5154 Views
- 3 replies
- 3 kudos
How to by pass SSL cert verification, using Repos with Azure Devops
Hi all, after some time working with Devops and Repos and getting used to the convenience our SSL Cert situation got jacked up somehow. While not ideal, I'd like to be able to temporarily bypass cert verification. There are ways to do this in the she...
- 5154 Views
- 3 replies
- 3 kudos
- 1621 Views
- 0 replies
- 5 kudos
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and...
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and reliable platform for users to build and deploy their applications. As part of this release, the en...
- 1621 Views
- 0 replies
- 5 kudos
- 2479 Views
- 1 replies
- 2 kudos
Does DLT Support watermarking and Windowing ?
Yes it does.Here is the syntaxe for Watermarkinghttps://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-watermark.htmlHere it the syntaxe for Windowing https://docs.databricks.com/sql/language-manual/sql-ref-window-functions.html
- 2479 Views
- 1 replies
- 2 kudos
- 2 kudos
Thanks for sharing.
- 2 kudos
- 2604 Views
- 2 replies
- 0 kudos
- 2604 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Youssef Mrini​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...
- 0 kudos
- 3554 Views
- 2 replies
- 2 kudos
Resolved! Can I run Ray applications on Databricks ?
With Databricks Runtime 12.0 and above, you can create a Ray cluster and run Ray applications in Databricks with the Ray on Spark API.Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a ...
- 3554 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Youssef Mrini​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 2 kudos
- 7111 Views
- 8 replies
- 7 kudos
Databricks IDE - Notebooks
With the announcement of the official IDE support for VS Code, does any one know if there's a way to run notebooks in VSC Code on Databricks clusters?https://www.databricks.com/blog/2023/02/14/announcing-a-native-visual-studio-code-experience-for-dat...
- 7111 Views
- 8 replies
- 7 kudos
- 7 kudos
There has been no response regarding running notebook cells in VS Code, so no best response.
- 7 kudos
- 6200 Views
- 3 replies
- 0 kudos
Resolved! AmazonS3 with Autoloader consume "too many" requests or maybe not!
After successfully loading 3 small files (2 KB each) in from AWS S3 using Auto Loader for learning purposes, I got, few hours later, a "AWS Free tier limit alert", although I haven't used the AWS account for a while. Does this streaming service on ...
- 6200 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, ​​Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage. Auto Loader can load data files from AWS S3 (s3://), Azure Data Lake Storage Gen2 (ADLS Gen2, abfss://), Google Cloud Storage (GCS, gs://), Azur...
- 0 kudos
- 2990 Views
- 3 replies
- 7 kudos
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...
- 2990 Views
- 3 replies
- 7 kudos
- 10113 Views
- 4 replies
- 23 kudos
Encrypt and decrypt personal data with Spark Databricks.We create a table that will include personal information. However, we want to hide personal id...
Encrypt and decrypt personal data with Spark Databricks.We create a table that will include personal information. However, we want to hide personal identifiers so no one can see them.We set a key. A key need to have 16, 24, or 32 bytes. 1 byte = 1 ch...
- 10113 Views
- 4 replies
- 23 kudos
- 23 kudos
@Hubert Dudek​ how can we decrypt the data outside of databricks with python? which is encrypted with aes_encrypt
- 23 kudos
- 3487 Views
- 2 replies
- 0 kudos
how to dynamically perform aggregation on all columns in a data frame even when some columns have different types like int , double string datetime or float in pyspark (i have 140-200 columns and need to perform aggregation/avg on each column)
need to aggregate all the numerical columns but need to this dynamically
- 3487 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi ​@sandeep tummala​ , Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your fe...
- 0 kudos
| User | Count |
|---|---|
| 1623 | |
| 790 | |
| 501 | |
| 349 | |
| 287 |
