cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
lightningStrike
by • New Contributor III
  • 4144 Views
  • 5 replies
  • 3 kudos

Give customized notebook run id/name in console using dbutils.notebook.run()

I am running below code in Azure databricks:Now the running status shows as highlighted in yellow. I want to give a custom name to this running status instead of Notebook job #<job id>. This will help me identify which status is for which notebook so...

image
Data Engineering
Reply
  • 4144 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Gaurav Tanwar​  I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest...

  • 3 kudos
4 More Replies
shama1
by • New Contributor II
  • 2598 Views
  • 2 replies
  • 3 kudos

Resolved! Not able to reset password with password re-set link

I am not able to reset password https://community.cloud.databricks.com/When I use link provided in the email to re-set the password, and I enter new password, request never gets completed.Password reset link keeps on loading even after entering passw...

image
Data Engineering
Reply
  • 2598 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Shama Khadpekar​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedb...

  • 3 kudos
1 More Replies
Chhaya
by • New Contributor III
  • 2319 Views
  • 3 replies
  • 2 kudos

Bamboolib with Databricks

Hi Everyone,I am wondering if anyone has experience using the bamboolib library within Databricks. I am currently using it for a client to display table data on the UI and potentially allow users to edit existing rows and insert new ones. While I hav...

bamboolib
Data Engineering
Reply
  • 2319 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Chhaya Vishwakarma​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sug...

  • 2 kudos
2 More Replies
RichardSCarchit
by • New Contributor III
  • 31022 Views
  • 4 replies
  • 4 kudos

How to extract DDL from tables/views using a SQL query?

I want to write a SQL query that queries the information_schema to generate a list of objects, their columns, relationships etc. - basically a data dictionary. For each object I want to show the DDL code, and I know I can get it by executing show c...

Data Engineering
Reply
  • 31022 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Richard Architect​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sugg...

  • 4 kudos
3 More Replies
prasadvaze
by • Valued Contributor II
  • 3220 Views
  • 2 replies
  • 2 kudos

Resolved! Delta sharing (databricks-to-databricks) between azure regions issue

we have 2 unity meta stores in 2 regions ( useast2 contains data and westeurope contains clusters) and enabled delta sharing between them. We use azure storage firewall / vent whitelisting to allow secure connection to storage from compute cluste...

Data Engineering
Reply
  • 3220 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@prasad vaze​ :Delta sharing between Unity metastores in different regions can be achieved in several ways, depending on your specific requirements and constraints. One common approach is to use Azure Private Link to establish a private connection be...

  • 2 kudos
1 More Replies
Herkimer
by • New Contributor II
  • 5302 Views
  • 3 replies
  • 1 kudos

dbsqlcli.main ERROR - getaddrinfo failed

I had to rebuild my laptop and am trying to re-install Databricks and Databricks-sql-cli so I can use the CLI to pull info from the databricks repository.When I try to run the command: dbsqlcli -e "select distinct tableName from databases_info.all_db...

Data Engineering
Reply
  • 5302 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @John Zajic​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 1 kudos
2 More Replies
Erik_L
by • Contributor II
  • 3437 Views
  • 2 replies
  • 1 kudos

Resolved! Pyspark read multiple Parquet type expansion failure

ProblemReading nearly equivalent parquet tables in a directory with some having column X with type float and some with type double fails.Attempts at resolvingUsing streaming filesRemoving delta caching, vectorizationUsing ,cache() explicitlyNotesThis...

Data Engineering
Reply
  • 3437 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Erik Louie​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Regards

  • 1 kudos
1 More Replies
abi-tosh
by • New Contributor III
  • 4071 Views
  • 6 replies
  • 4 kudos

Databricks Attribute Error: 'IPythonShell' object has no attribute 'kernel'

I have been getting this error repeatedly when trying to run a notebook. I have tried attaching multiple different clusters and installing some of the libraries that it wanted me to update. I have also tried to clear the state of the notebook and res...

Data Engineering
Reply
  • 4071 Views
  • 6 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Toshali Mohapatra​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 4 kudos
5 More Replies
Hubert-Dudek
by • Esteemed Contributor III
  • 3491 Views
  • 2 replies
  • 8 kudos

Databricks has announced that users can now create notebooks in Jupyter format in Repos, offering a familiar experience for creating and editing noteb...

Databricks has announced that users can now create notebooks in Jupyter format in Repos, offering a familiar experience for creating and editing notebooks. This update allows users to integrate with the broader data science ecosystem, import and expo...

Untitled
Data Engineering
Reply
  • 3491 Views
  • 2 replies
  • 8 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 8 kudos

Hi @Hubert Dudek​ Thanking you for helping us to build a vibrant and resourceful community by recognizing and highlighting insightful contributions.Regards

  • 8 kudos
1 More Replies
xhh
by • New Contributor
  • 1077 Views
  • 2 replies
  • 0 kudos

How Spark clusters are integrated with TensorFlow

Data Engineering
Reply
  • 1077 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @令辉 孔​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback will he...

  • 0 kudos
1 More Replies
William_Scardua
by • Valued Contributor
  • 1862 Views
  • 2 replies
  • 1 kudos

Cosmos DB Connector for 12.1 Cluster and above

Hi guys,​You know what`s the version suporte the databricks cluster version above 12.1 ?my cluster:error: Thank you

databricks-cluster cosmosdb-connector-erro
Data Engineering
Reply
  • 1862 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 1 kudos
1 More Replies
883702
by • New Contributor III
  • 1986 Views
  • 1 replies
  • 0 kudos

Resolved! TypeError on DataFrame via spark readStream transform invocation of UDF

Our use case is to "clean up" column names (remove spaces, etc) on ingestion of CSV data using the Delta Live Table capability. We desire to use the schema inference capability during ingestion so schema specification (up front) will not be happenin...

Data Engineering
Reply
  • 1986 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
883702
New Contributor III
  • 0 kudos

The issue was erroneously believing the transform function needed UDF decorator. With the decorator removed the transform invokes (and works) as expected.

  • 0 kudos
Phani1
by • Valued Contributor II
  • 1764 Views
  • 3 replies
  • 0 kudos

Performance issue while loading bulk data into Post Gress DB from data bricks.

We are facing a performance issue while loading bulk data into Postgress DB from data bricks. We are using spark JDBC connections to move the data. However, the rate of transfer is very low which is causing performance bottleneck. is there any better...

Data Engineering
Reply
  • 1764 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
User16502773013
Databricks Employee
  • 0 kudos

Hello @Janga Reddy​ @Daniel Sahal​ and @Vidula Khanna​ ,To enhance performance in general we need to design for more parallelism, in Spark JDBC context this controlled by the number of partitions for the data to be writtenThe example here shows how t...

  • 0 kudos
2 More Replies
Avvar2022
by • Contributor
  • 3027 Views
  • 2 replies
  • 2 kudos

Resolved! I am new to data bricks. setting up Workspace for NON-prod environment Separate workspaces for DEV, QA or Just one work space for NON-prod ?

What i learned based on learning materials, documents, etc.. For data bricks it is a good practice to set up 1 non-prod workspace but separate clusters for Dev, QA, SIT, etc.Is it best practice to set up only 1 NON-PROD Workspace instead of separate ...

Databricks non-prod workspace set up options
Data Engineering
Reply
  • 3027 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Avvar2022
Contributor
  • 2 kudos

Thank you. This helps.

  • 2 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels