cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Dave_B_
by New Contributor III
  • 1264 Views
  • 2 replies
  • 0 kudos

DBX injected V-Net and Deployment

Due to the need for Azure storage private endpoints, we switched our databricks deployment to use an injected vnet. Now, when our deployment pipeline tries to re-create the workspace (e.g. az databricks workspace delete), it seems to leave the MS cre...

  • 1264 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @David Benedict​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 0 kudos
1 More Replies
FabriceDeseyn
by Contributor
  • 3992 Views
  • 5 replies
  • 1 kudos

Resolved! Autoloader directory listing not listing all files

Hi communityI have an Autoloader pipeline running with following configuration. Unfortunately, it does not detect all files. (see below query definition). The folder that needs to be read has 38.246 files that all have the same schema and structure.:...

image.png image.png image.png image.png
  • 3992 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Fabrice Deseyn​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 1 kudos
4 More Replies
Nilofar
by New Contributor II
  • 2003 Views
  • 7 replies
  • 0 kudos

i am not able to reset the password for data bricks cloud community

Hi,i am not log in to https://community.cloud.databricks.com/login.html. Please assist .

  • 2003 Views
  • 7 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Nilofar Sharma​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 0 kudos
6 More Replies
matthewe97
by New Contributor
  • 2207 Views
  • 3 replies
  • 2 kudos

Resolved! Are window functions more performant than self joins?

I have a table with data for each month end and want to know the LEAD and LAG data points either side of each month. For example:SELECT month_date, LEAD(month_date) OVER (PARTITION BY id ORDER BY month_date) next_month_date,  LAG(month_date) OVER (PA...

  • 2207 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Matthew Elsham​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 2 kudos
2 More Replies
lightningStrike
by New Contributor III
  • 2791 Views
  • 5 replies
  • 3 kudos

Give customized notebook run id/name in console using dbutils.notebook.run()

I am running below code in Azure databricks:Now the running status shows as highlighted in yellow. I want to give a custom name to this running status instead of Notebook job #<job id>. This will help me identify which status is for which notebook so...

image
  • 2791 Views
  • 5 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Gaurav Tanwar​  I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest...

  • 3 kudos
4 More Replies
shama1
by New Contributor II
  • 1614 Views
  • 2 replies
  • 2 kudos

Resolved! Not able to reset password with password re-set link

I am not able to reset password https://community.cloud.databricks.com/When I use link provided in the email to re-set the password, and I enter new password, request never gets completed.Password reset link keeps on loading even after entering passw...

image
  • 1614 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Shama Khadpekar​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedb...

  • 2 kudos
1 More Replies
Chhaya
by New Contributor III
  • 1389 Views
  • 3 replies
  • 2 kudos

Bamboolib with Databricks

Hi Everyone,I am wondering if anyone has experience using the bamboolib library within Databricks. I am currently using it for a client to display table data on the UI and potentially allow users to edit existing rows and insert new ones. While I hav...

bamboolib
  • 1389 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Chhaya Vishwakarma​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sug...

  • 2 kudos
2 More Replies
RichardSCarchit
by New Contributor III
  • 12209 Views
  • 4 replies
  • 3 kudos

How to extract DDL from tables/views using a SQL query?

I want to write a SQL query that queries the information_schema to generate a list of objects, their columns, relationships etc. - basically a data dictionary. For each object I want to show the DDL code, and I know I can get it by executing show c...

  • 12209 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Richard Architect​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sugg...

  • 3 kudos
3 More Replies
prasadvaze
by Valued Contributor II
  • 2124 Views
  • 2 replies
  • 2 kudos

Resolved! Delta sharing (databricks-to-databricks) between azure regions issue

we have 2 unity meta stores in 2 regions ( useast2 contains data and westeurope contains clusters) and enabled delta sharing between them. We use azure storage firewall / vent whitelisting to allow secure connection to storage from compute cluste...

  • 2124 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@prasad vaze​ :Delta sharing between Unity metastores in different regions can be achieved in several ways, depending on your specific requirements and constraints. One common approach is to use Azure Private Link to establish a private connection be...

  • 2 kudos
1 More Replies
Herkimer
by New Contributor II
  • 1969 Views
  • 3 replies
  • 1 kudos

dbsqlcli.main ERROR - getaddrinfo failed

I had to rebuild my laptop and am trying to re-install Databricks and Databricks-sql-cli so I can use the CLI to pull info from the databricks repository.When I try to run the command: dbsqlcli -e "select distinct tableName from databases_info.all_db...

  • 1969 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @John Zajic​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 1 kudos
2 More Replies
Erik_L
by Contributor II
  • 1694 Views
  • 2 replies
  • 1 kudos

Resolved! Pyspark read multiple Parquet type expansion failure

ProblemReading nearly equivalent parquet tables in a directory with some having column X with type float and some with type double fails.Attempts at resolvingUsing streaming filesRemoving delta caching, vectorizationUsing ,cache() explicitlyNotesThis...

  • 1694 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Erik Louie​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Regards

  • 1 kudos
1 More Replies
abi-tosh
by New Contributor III
  • 2636 Views
  • 6 replies
  • 4 kudos

Databricks Attribute Error: 'IPythonShell' object has no attribute 'kernel'

I have been getting this error repeatedly when trying to run a notebook. I have tried attaching multiple different clusters and installing some of the libraries that it wanted me to update. I have also tried to clear the state of the notebook and res...

  • 2636 Views
  • 6 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Toshali Mohapatra​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 4 kudos
5 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 798 Views
  • 2 replies
  • 8 kudos

Databricks has announced that users can now create notebooks in Jupyter format in Repos, offering a familiar experience for creating and editing noteb...

Databricks has announced that users can now create notebooks in Jupyter format in Repos, offering a familiar experience for creating and editing notebooks. This update allows users to integrate with the broader data science ecosystem, import and expo...

Untitled
  • 798 Views
  • 2 replies
  • 8 kudos
Latest Reply
Anonymous
Not applicable
  • 8 kudos

Hi @Hubert Dudek​ Thanking you for helping us to build a vibrant and resourceful community by recognizing and highlighting insightful contributions.Regards

  • 8 kudos
1 More Replies
xhh
by New Contributor
  • 661 Views
  • 2 replies
  • 0 kudos
  • 661 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @令辉 孔​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback will he...

  • 0 kudos
1 More Replies
969091
by New Contributor
  • 8637 Views
  • 2 replies
  • 2 kudos

Send custom emails from databricks notebook without using third party SMTP server. Would like to utilize databricks existing smtp or databricks api.

We want to use existing databricks smtp server or if databricks api can used to send custom emails. Databricks Workflows sends email notifications on success, failure, etc. of jobs but cannot send custom emails. So we want to send custom emails to di...

  • 8637 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Krishna Prasad​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedba...

  • 2 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels