cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Vidyasankar
by New Contributor
  • 1172 Views
  • 3 replies
  • 1 kudos
  • 1172 Views
  • 3 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Vidya sankar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 1 kudos
2 More Replies
Vignesh2806
by New Contributor II
  • 7154 Views
  • 2 replies
  • 3 kudos

I would like to get the below error solved Cluster scoped init script dbfs:/FileStore/tables/***.sh failed: Script exit status is non-zero

I am trying to run the databricks cluster, but at times the cluster takes long time to get set up & After some time it throws the below error. Cluster scoped init script dbfs:/FileStore/tables/***.sh failed: Script exit status is non-zeroThe init scr...

  • 7154 Views
  • 2 replies
  • 3 kudos
Latest Reply
Vidula
Honored Contributor
  • 3 kudos

Hi @Vignesh Ravichandran​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 3 kudos
1 More Replies
chandan_a_v
by Valued Contributor
  • 8095 Views
  • 2 replies
  • 4 kudos

Best way to run the Databricks notebook in a parallel way

Hi All,I need to run a Databricks notebook in a parallel way for different arguments. I tried with the threading approach but only the first 2 threads successfully execute the notebook and the rest fail. Please let me know if there is any best way to...

  • 8095 Views
  • 2 replies
  • 4 kudos
Latest Reply
Vidula
Honored Contributor
  • 4 kudos

Hey there @Chandan Angadi​ Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 4 kudos
1 More Replies
lizou
by Contributor II
  • 1762 Views
  • 4 replies
  • 2 kudos

call saved query in sql warehouse

in python cursor.executecan you call a saved query with a parameter? like call a stored procedure in relational db?https://docs.microsoft.com/en-us/azure/databricks/dev-tools/python-sql-connector#cursor-method

  • 1762 Views
  • 4 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @lizou​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 2 kudos
3 More Replies
BeginnerBob
by New Contributor III
  • 10240 Views
  • 4 replies
  • 2 kudos

Flatten a complex JSON file and load into a delta table

Hi,I am loading a JSON file into Databricks by simply doing the following:from pyspark.sql.functions import *from pyspark.sql.types import *bronze_path="wasbs://....../140477.json"df_incremental = spark.read.option("multiline","true").json(bronze_pat...

image
  • 10240 Views
  • 4 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @Lloyd Vickery​ Does @Werner Stinckens​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 2 kudos
3 More Replies
Dan-K
by New Contributor III
  • 12085 Views
  • 6 replies
  • 6 kudos

Resolved! How to display markdown output in databricks notebook from a python cell

With IPython/Jupyter it's possible to output markdown using the IPython display module and its `MarkDown`class. QuestionHow can I accomplish this with Azure Databricks?What I triedDatabricks `display`Tried using Databrick's display with the IPython M...

Example markdown output Example failed ouput Failed display_markdown Databricks display
  • 12085 Views
  • 6 replies
  • 6 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 6 kudos

Hi, Thanks for reaching out to community.databricks.com.In a notebook cell, type "%md" and type some markdown and it will render. Please refer: https://community.databricks.com/s/question/0D53f00001HKHhNCAX/markup-in-databricks-notebook

  • 6 kudos
5 More Replies
ArjunS310
by New Contributor III
  • 715 Views
  • 2 replies
  • 2 kudos

Resolved! Did not receive a badge upon completing databricks fundamentals assessment

Team,I completed the training and assessment on databricks assesment and passed with 80% and received a certificate of completion but did not receive a badge as mentioned in the description of the course. Could you please help.​

  • 715 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Arjun Shaji​ , Thank you for reaching out!Let us look into this for you, and we'll follow up with an update.

  • 2 kudos
1 More Replies
parthibsg
by New Contributor II
  • 875 Views
  • 2 replies
  • 2 kudos

When to use Dataframes API over Spark SQL

Hello Experts,I am new to Databricks. Building data pipelines, I have both batch and streaming data.Should I use Dataframes API to read csv files then convert to parquet format then do the transformation? orwrite to table using CSV then use Spark SQL...

  • 875 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Parthib Rathnam​, Thank you for reaching out!Let us look into this for you, and we'll follow up with an update.

  • 2 kudos
1 More Replies
BananaHotSauce
by New Contributor III
  • 517 Views
  • 2 replies
  • 3 kudos

Can I use PrivateLink and Customer Managed Policy for Cross Account Role

Hello, Im trying to enable Privatelink on my AWS Databricks quickstart, ​I use the customer managed VPC policy for the ​cross account role and supply it on the template. Im having an error that it cannot create a VPC Endpoint.​Do i need to change the...

  • 517 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Chris Joshua Manuel​  , We haven’t heard from you on the last response from @Debayan Mukherjee​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be...

  • 3 kudos
1 More Replies
c038644
by New Contributor II
  • 1271 Views
  • 4 replies
  • 3 kudos

Use of venv pack

Hi, I very new so this probably sounds stupid...I'm following the blog on How to Manage Python Dependencies in PySpark:https://www.databricks.com/blog/2020/12/22/how-to-manage-python-dependencies-in-pyspark.html...but when I try the packing works fin...

  • 1271 Views
  • 4 replies
  • 3 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 3 kudos

Can you try using an absolute path instead of a relative path for the same? For example: https://stackoverflow.com/questions/38661464/filenotfounderror-winerror-3

  • 3 kudos
3 More Replies
Giorgi
by New Contributor III
  • 2357 Views
  • 2 replies
  • 3 kudos

GitLab integration

I've followed instructions and did gitlab integration:Generated Personal Access Token from GitLabAdd token (from step 1) to User settings (GitLab, email, token)In Admin console -> Repos Git URL Allow List permissions: Disabled (no restrictions)In Adm...

  • 2357 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Giorgi ARABIDZE​ , We haven't heard from you on the last response from @Hubert Dudek​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to oth...

  • 3 kudos
1 More Replies
him
by New Contributor III
  • 810 Views
  • 2 replies
  • 3 kudos
  • 810 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Himanshu yadav​ , We haven't heard from you on the last response from @Debayan Mukherjee​ , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful t...

  • 3 kudos
1 More Replies
al_joe
by Contributor
  • 978 Views
  • 2 replies
  • 0 kudos

Can we have a better UI for navigating Workspace and Repos?

Navigating through multiple vertical panes of information as we navigate deeper into a folder structure is not very convenient -- we lose the context of parent folder and sibling folders very soon.Can we not have a simple tree view (similar to VS Cod...

image.png
  • 978 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @Al Jo​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 0 kudos
1 More Replies
dbrick
by New Contributor II
  • 840 Views
  • 2 replies
  • 1 kudos

Multiple Jobs with different resource requirements on the same cluster

I have a big cluster with the auto-scaling(min:1, max: 25) feature enabled. I want to run multiple jobs on that cluster with different values of spark properties( `--executor-cores` and `–executor-memory) but I don't see any option to specify the sam...

  • 840 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Neelesh databricks​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell ...

  • 1 kudos
1 More Replies
nhgil
by New Contributor II
  • 1726 Views
  • 5 replies
  • 1 kudos

Accessing Workspace?

Trying to follow along with the instructions for the 'AWS and Databricks' training yet the documentation gives no indication of how to access the workspace that is ostensibly generated in AWS. I subscribed through the marketplace and cloud formation...

  • 1726 Views
  • 5 replies
  • 1 kudos
Latest Reply
nhgil
New Contributor II
  • 1 kudos

Unfortunately by the time I received a response my trial had long lapsed so I had already unsubscribed in aws.

  • 1 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels