Encrypt and decrypt personal data with Spark Databricks.We create a table that will include personal information. However, we want to hide personal identifiers so no one can see them.We set a key. A key need to have 16, 24, or 32 bytes. 1 byte = 1 ch...
Hi ​@sandeep tummala​ , Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your fe...
I have followed the steps as mentioned in this blog : https://www.linkedin.com/pulse/aws-glue-data-catalog-metastore-databricks-deepak-rajak/ but when trying to saveAsTable(table_name), it is giving an error as IllegalArgumentException: Path must be ...
Hey @Raghu Bharadwaj Tallapragada​ Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
Continuous Jobs trigger mode has been introduced recently. To prevent unnecessary resource usage and reduce cost, Databricks automatically pauses a continuous job if there are more than five consecutive failures within a 24 hour period.
The UNPIVOT clause is now supported by Databricks SQL. Use the UNPIVOT clause to rotate columns of a table-valued expression into column values https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-unpivot.html
I want to make a matplolib interactive (I can zoom in/out, etc.) in databricks. This is achieved in Jupyter notebook by the following code: %matplotlib notebookHow to achieve this in databricks?Thank you
Hi there, while facing a similar issue we switched to Altair python library and it works great with Databricks. (other options can be Bokeh or Plotly).
I have configured a File Notification Autoloader that monitors S3 bucket for binary files. I want to integrate autoloader with workflow job so that whenever a file is placed in S3 bucket, the pipeline job notebook tasks can pick-up new file and start...
Hi @Saravanan Ponnaiah​ Hope everything is going great.Does @odoll odoll​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
The monospaced/code font in my databricks notebooks is greyed out, both in light and dark theme. I've tried playing with all the notebook settings, etc. and nothing will make the font 'normal'. I've tried Chrome and Edge, and the results are the same...
Hi,I am running autoloader which is running continuously and checks for new file every 1 minute. I need to store when file was received/processed but its giving me date when autoloader started. Here is my code.df = (spark .readStream .format("clo...
Hi @Sanjay Jain​ , You can use the File Metadata column functionality to collect that information.Ref doc:- https://docs.databricks.com/ingestion/file-metadata-column.html
I'm trying to install a python library but I'm not able, the status won't change from "pending". I get this message when I click on the library under the cluster's Libraries tab: "Library installation has been attempted on the driver node but has not...
Ok, looks like I was able to solve my problem.First, I needed to install all the required libraries one by one. These are the followings:pandassixrequestspyspnegocryptographykrb5requests-kerberosAfter that I was able to install the webAPI library.
We trigger different Worflows by ADF.These workflows will be shown triggered "manually".Is this behaviour intentional? At least for users, this is confusing.ADF-triggered Run: Databricks-Workflows:
Hi @J. G.​, Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does. Your feedback will...
