cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Hubert-Dudek
by • Esteemed Contributor III
  • 1900 Views
  • 3 replies
  • 7 kudos

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...

ezgif-3-d3fac0139c
Data Engineering
Reply
  • 1900 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 7 kudos

Informative Thanks for sharing.

  • 7 kudos
2 More Replies
Hubert-Dudek
by • Esteemed Contributor III
  • 5736 Views
  • 4 replies
  • 23 kudos

Encrypt and decrypt personal data with Spark Databricks.We create a table that will include personal information. However, we want to hide personal id...

Encrypt and decrypt personal data with Spark Databricks.We create a table that will include personal information. However, we want to hide personal identifiers so no one can see them.We set a key. A key need to have 16, 24, or 32 bytes. 1 byte = 1 ch...

image.png image.png image.png image.png
Data Engineering
Reply
  • 5736 Views
  • 4 replies
  • 23 kudos
View Replies
Latest Reply
MaheshDBR
New Contributor II
  • 23 kudos

@Hubert Dudek​ how can we decrypt the data outside of databricks with python? which is encrypted with aes_encrypt

  • 23 kudos
3 More Replies
STummala
by • New Contributor
  • 1906 Views
  • 2 replies
  • 0 kudos

how to dynamically perform aggregation on all columns in a data frame even when some columns have different types like int , double string datetime or float in pyspark (i have 140-200 columns and need to perform aggregation/avg on each column)

need to aggregate all the numerical columns but need to this dynamically

Data Engineering
Reply
  • 1906 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi ​@sandeep tummala​ , Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your fe...

  • 0 kudos
1 More Replies
raghub1
by • New Contributor II
  • 6743 Views
  • 4 replies
  • 3 kudos

Resolved! Writing PySpark DataFrame onto AWS Glue throwing error

I have followed the steps as mentioned in this blog : https://www.linkedin.com/pulse/aws-glue-data-catalog-metastore-databricks-deepak-rajak/ but when trying to saveAsTable(table_name), it is giving an error as IllegalArgumentException: Path must be ...

Data Engineering
Reply
  • 6743 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey @Raghu Bharadwaj Tallapragada​ Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!

  • 3 kudos
3 More Replies
youssefmrini
by • Databricks Employee
  • 1174 Views
  • 1 replies
  • 2 kudos

Resolved! Does Databricks Workflows support Continuous Jobs ?

Data Engineering
Reply
  • 1174 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
youssefmrini
Databricks Employee
  • 2 kudos

Continuous Jobs trigger mode has been introduced recently. To prevent unnecessary resource usage and reduce cost, Databricks automatically pauses a continuous job if there are more than five consecutive failures within a 24 hour period.

  • 2 kudos
youssefmrini
by • Databricks Employee
  • 1590 Views
  • 1 replies
  • 0 kudos

Can I use Unpivot clause in databricks SQL ?

Data Engineering
Reply
  • 1590 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

The UNPIVOT clause is now supported by Databricks SQL. Use the  UNPIVOT clause to rotate columns of a table-valued expression into column values https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-unpivot.html

  • 0 kudos
eabouzeid
by • New Contributor III
  • 12068 Views
  • 8 replies
  • 8 kudos

How to enable interactive Python matplotlib figures in DataBricks?

I want to make a matplolib interactive (I can zoom in/out, etc.) in databricks. This is achieved in Jupyter notebook by the following code: %matplotlib notebookHow to achieve this in databricks?Thank you

Data Engineering
Reply
  • 12068 Views
  • 8 replies
  • 8 kudos
View Replies
Latest Reply
amu
New Contributor II
  • 8 kudos

Hi there, while facing a similar issue we switched to Altair python library and it works great with Databricks. (other options can be Bokeh or Plotly).

  • 8 kudos
7 More Replies
brickster
by • New Contributor II
  • 3696 Views
  • 3 replies
  • 0 kudos

How to trigger workflow job tasks from Autoloader

I have configured a File Notification Autoloader that monitors S3 bucket for binary files. I want to integrate autoloader with workflow job so that whenever a file is placed in S3 bucket, the pipeline job notebook tasks can pick-up new file and start...

Data Engineering
Reply
  • 3696 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Saravanan Ponnaiah​ Hope everything is going great.Does @odoll odoll​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
2 More Replies
bradlindblad
by • New Contributor II
  • 2099 Views
  • 2 replies
  • 1 kudos

Resolved! Font in Databricks Notebook is Greyed Out - Glitchy

The monospaced/code font in my databricks notebooks is greyed out, both in light and dark theme. I've tried playing with all the notebook settings, etc. and nothing will make the font 'normal'. I've tried Chrome and Edge, and the results are the same...

db
Data Engineering
Reply
  • 2099 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
klaapbakken
New Contributor III
  • 1 kudos

I was having this exact same issue. I fixed it by uninstalling the Source Code Pro font from my Windows machine.

  • 1 kudos
1 More Replies
Gk
by • New Contributor III
  • 3724 Views
  • 10 replies
  • 1 kudos

DataBricks

How to find Mountpoints definitions

Data Engineering
Reply
  • 3724 Views
  • 10 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Govardhana Reddy​ Glad to hear!Please mark the answer as best, it will be highly appreciable.Have a great day!Regards

  • 1 kudos
9 More Replies
sanjay
by • Valued Contributor II
  • 2689 Views
  • 4 replies
  • 1 kudos

Resolved! How can I get date when autoloader processes the file

Hi,I am running autoloader which is running continuously and checks for new file every 1 minute. I need to store when file was received/processed but its giving me date when autoloader started. Here is my code.df = (spark   .readStream   .format("clo...

Data Engineering
Reply
  • 2689 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

Hi @Sanjay Jain​ , You can use the File Metadata column functionality to collect that information.Ref doc:- https://docs.databricks.com/ingestion/file-metadata-column.html

  • 1 kudos
3 More Replies
u2dragon
by • New Contributor III
  • 13088 Views
  • 5 replies
  • 0 kudos

Resolved! Can't install python library

I'm trying to install a python library but I'm not able, the status won't change from "pending". I get this message when I click on the library under the cluster's Libraries tab: "Library installation has been attempted on the driver node but has not...

Data Engineering
Reply
  • 13088 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
u2dragon
New Contributor III
  • 0 kudos

Ok, looks like I was able to solve my problem.First, I needed to install all the required libraries one by one. These are the followings:pandassixrequestspyspnegocryptographykrb5requests-kerberosAfter that I was able to install the webAPI library.

  • 0 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels