- 3769 Views
- 2 replies
- 2 kudos
Resolved! Can I run Ray applications on Databricks ?
With Databricks Runtime 12.0 and above, you can create a Ray cluster and run Ray applications in Databricks with the Ray on Spark API.Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a ...
- 3769 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Youssef Mrini​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 2 kudos
- 7674 Views
- 8 replies
- 7 kudos
Databricks IDE - Notebooks
With the announcement of the official IDE support for VS Code, does any one know if there's a way to run notebooks in VSC Code on Databricks clusters?https://www.databricks.com/blog/2023/02/14/announcing-a-native-visual-studio-code-experience-for-dat...
- 7674 Views
- 8 replies
- 7 kudos
- 7 kudos
There has been no response regarding running notebook cells in VS Code, so no best response.
- 7 kudos
- 6675 Views
- 3 replies
- 0 kudos
Resolved! AmazonS3 with Autoloader consume "too many" requests or maybe not!
After successfully loading 3 small files (2 KB each) in from AWS S3 using Auto Loader for learning purposes, I got, few hours later, a "AWS Free tier limit alert", although I haven't used the AWS account for a while. Does this streaming service on ...
- 6675 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, ​​Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage. Auto Loader can load data files from AWS S3 (s3://), Azure Data Lake Storage Gen2 (ADLS Gen2, abfss://), Google Cloud Storage (GCS, gs://), Azur...
- 0 kudos
- 3253 Views
- 3 replies
- 7 kudos
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...
- 3253 Views
- 3 replies
- 7 kudos
- 11200 Views
- 4 replies
- 23 kudos
Encrypt and decrypt personal data with Spark Databricks.We create a table that will include personal information. However, we want to hide personal id...
Encrypt and decrypt personal data with Spark Databricks.We create a table that will include personal information. However, we want to hide personal identifiers so no one can see them.We set a key. A key need to have 16, 24, or 32 bytes. 1 byte = 1 ch...
- 11200 Views
- 4 replies
- 23 kudos
- 23 kudos
@Hubert Dudek​ how can we decrypt the data outside of databricks with python? which is encrypted with aes_encrypt
- 23 kudos
- 3774 Views
- 2 replies
- 0 kudos
how to dynamically perform aggregation on all columns in a data frame even when some columns have different types like int , double string datetime or float in pyspark (i have 140-200 columns and need to perform aggregation/avg on each column)
need to aggregate all the numerical columns but need to this dynamically
- 3774 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi ​@sandeep tummala​ , Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your fe...
- 0 kudos
- 2267 Views
- 2 replies
- 2 kudos
Resolved! newly created account not working and when changing password it is getting hangged
My community user account is not working
- 2267 Views
- 2 replies
- 2 kudos
- 9817 Views
- 4 replies
- 3 kudos
Resolved! Writing PySpark DataFrame onto AWS Glue throwing error
I have followed the steps as mentioned in this blog : https://www.linkedin.com/pulse/aws-glue-data-catalog-metastore-databricks-deepak-rajak/ but when trying to saveAsTable(table_name), it is giving an error as IllegalArgumentException: Path must be ...
- 9817 Views
- 4 replies
- 3 kudos
- 3 kudos
Hey @Raghu Bharadwaj Tallapragada​ Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
- 3 kudos
- 2094 Views
- 1 replies
- 2 kudos
- 2094 Views
- 1 replies
- 2 kudos
- 2 kudos
Continuous Jobs trigger mode has been introduced recently. To prevent unnecessary resource usage and reduce cost, Databricks automatically pauses a continuous job if there are more than five consecutive failures within a 24 hour period.
- 2 kudos
- 3414 Views
- 1 replies
- 0 kudos
- 3414 Views
- 1 replies
- 0 kudos
- 0 kudos
The UNPIVOT clause is now supported by Databricks SQL. Use the UNPIVOT clause to rotate columns of a table-valued expression into column values https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-unpivot.html
- 0 kudos
- 16985 Views
- 8 replies
- 8 kudos
How to enable interactive Python matplotlib figures in DataBricks?
I want to make a matplolib interactive (I can zoom in/out, etc.) in databricks. This is achieved in Jupyter notebook by the following code: %matplotlib notebookHow to achieve this in databricks?Thank you
- 16985 Views
- 8 replies
- 8 kudos
- 8 kudos
Hi there, while facing a similar issue we switched to Altair python library and it works great with Databricks. (other options can be Bokeh or Plotly).
- 8 kudos
- 4851 Views
- 1 replies
- 2 kudos
Resolved! What to do when you face this error "Please delete your streaming query checkpoint and restart"
- 4851 Views
- 1 replies
- 2 kudos
- 2 kudos
The best way is to run a full refresh.
- 2 kudos
- 1599 Views
- 1 replies
- 2 kudos
- 1599 Views
- 1 replies
- 2 kudos
- 2 kudos
You will have to do a full refresh.
- 2 kudos
- 9698 Views
- 3 replies
- 0 kudos
How to trigger workflow job tasks from Autoloader
I have configured a File Notification Autoloader that monitors S3 bucket for binary files. I want to integrate autoloader with workflow job so that whenever a file is placed in S3 bucket, the pipeline job notebook tasks can pick-up new file and start...
- 9698 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Saravanan Ponnaiah​ Hope everything is going great.Does @odoll odoll​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 0 kudos
- 3646 Views
- 2 replies
- 1 kudos
Resolved! Font in Databricks Notebook is Greyed Out - Glitchy
The monospaced/code font in my databricks notebooks is greyed out, both in light and dark theme. I've tried playing with all the notebook settings, etc. and nothing will make the font 'normal'. I've tried Chrome and Edge, and the results are the same...
- 3646 Views
- 2 replies
- 1 kudos
- 1 kudos
I was having this exact same issue. I fixed it by uninstalling the Source Code Pro font from my Windows machine.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
