- 27395 Views
- 4 replies
- 2 kudos
I would like to access S3 data in databricks
Hi all,I am new to the databricks. I am trying to get the data from S3. The video tutoirals from the streaming platforms are accessing via access ID and secret access key. However, databricks is throwing a different options. I dont know what to fill...
- 27395 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Karthikeyan Palanisamy Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...
- 2 kudos
- 21871 Views
- 15 replies
- 0 kudos
Connecting Databricks Spark Cluster to Postgresql RDS Instance
I am trying to connect my Spark cluster to a Postgresql RDS instance. The Python notebook code that was used is seen below:df = ( spark.read \ .format("jdbc") \ .option("url", "jdbc:postgresql://<connection-string>:5432/database”)\ .option("dbt...
- 21871 Views
- 15 replies
- 0 kudos
- 0 kudos
"Caused by: java.net.SocketTimeoutException: connect timed out" indicate the network connection between Databricks cluster and the postgress database on 5432 port was not established and eventually timed out.As a first step, please ensure the connect...
- 0 kudos
- 6092 Views
- 1 replies
- 0 kudos
How to trigger Azure Logic App from Azure Databricks?
I have an Azure Logic app which triggers whenever a HTTP Post request is received. I want to send this request from my notebook present in Azure Databricks workspace using scala and spark. Is it possible? If yes, then please guide on how to do it. T...
- 6092 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey there @Ayushri Jain Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...
- 0 kudos
- 5237 Views
- 4 replies
- 1 kudos
Partition pruning with generated columns
I have a large table which contains a date_time column.The table contains 2 generated columns year, and month which are extracted from the date_time values and are used for partitioning.I have the following question.If I run the querySELECT *FROM tab...
- 5237 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Andrej Znidarsic Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 1 kudos
- 3835 Views
- 2 replies
- 1 kudos
- 3835 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Vidya sankar Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 1 kudos
- 11609 Views
- 2 replies
- 3 kudos
I would like to get the below error solved Cluster scoped init script dbfs:/FileStore/tables/***.sh failed: Script exit status is non-zero
I am trying to run the databricks cluster, but at times the cluster takes long time to get set up & After some time it throws the below error. Cluster scoped init script dbfs:/FileStore/tables/***.sh failed: Script exit status is non-zeroThe init scr...
- 11609 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Vignesh Ravichandran Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
- 3 kudos
- 10648 Views
- 2 replies
- 4 kudos
Best way to run the Databricks notebook in a parallel way
Hi All,I need to run a Databricks notebook in a parallel way for different arguments. I tried with the threading approach but only the first 2 threads successfully execute the notebook and the rest fail. Please let me know if there is any best way to...
- 10648 Views
- 2 replies
- 4 kudos
- 4 kudos
Hey there @Chandan Angadi Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 4 kudos
- 4442 Views
- 4 replies
- 2 kudos
call saved query in sql warehouse
in python cursor.executecan you call a saved query with a parameter? like call a stored procedure in relational db?https://docs.microsoft.com/en-us/azure/databricks/dev-tools/python-sql-connector#cursor-method
- 4442 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @lizou Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 2 kudos
- 27403 Views
- 4 replies
- 2 kudos
Flatten a complex JSON file and load into a delta table
Hi,I am loading a JSON file into Databricks by simply doing the following:from pyspark.sql.functions import *from pyspark.sql.types import *bronze_path="wasbs://....../140477.json"df_incremental = spark.read.option("multiline","true").json(bronze_pat...
- 27403 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Lloyd Vickery Does @Werner Stinckens response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 2 kudos
- 38526 Views
- 6 replies
- 6 kudos
Resolved! How to display markdown output in databricks notebook from a python cell
With IPython/Jupyter it's possible to output markdown using the IPython display module and its `MarkDown`class. QuestionHow can I accomplish this with Azure Databricks?What I triedDatabricks `display`Tried using Databrick's display with the IPython M...
- 38526 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi, Thanks for reaching out to community.databricks.com.In a notebook cell, type "%md" and type some markdown and it will render. Please refer: https://community.databricks.com/s/question/0D53f00001HKHhNCAX/markup-in-databricks-notebook
- 6 kudos
- 2895 Views
- 1 replies
- 0 kudos
Can we have a better UI for navigating Workspace and Repos?
Navigating through multiple vertical panes of information as we navigate deeper into a folder structure is not very convenient -- we lose the context of parent folder and sibling folders very soon.Can we not have a simple tree view (similar to VS Cod...
- 2895 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey there @Al Jo Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
- 0 kudos
- 2299 Views
- 1 replies
- 1 kudos
Multiple Jobs with different resource requirements on the same cluster
I have a big cluster with the auto-scaling(min:1, max: 25) feature enabled. I want to run multiple jobs on that cluster with different values of spark properties( `--executor-cores` and `–executor-memory) but I don't see any option to specify the sam...
- 2299 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Neelesh databricks Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell ...
- 1 kudos
- 1670 Views
- 1 replies
- 3 kudos
Can I use PrivateLink and Customer Managed Policy for Cross Account Role
Hello, Im trying to enable Privatelink on my AWS Databricks quickstart, I use the customer managed VPC policy for the cross account role and supply it on the template. Im having an error that it cannot create a VPC Endpoint.Do i need to change the...
- 1670 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Chris Joshua Manuel , Thanks for reaching out to Community.databricks.com. Cross account VPC access works. Please refer below: https://tomgregory.com/cross-account-vpc-access-in-aws/Also, please let us know in case of any further clarification n...
- 3 kudos
- 3982 Views
- 3 replies
- 0 kudos
- 3982 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, I believe you are using a single tenant shard where SQL option is not there.
- 0 kudos
- 5061 Views
- 2 replies
- 2 kudos
Resolved! Errors on Redash when queries are in cache
We got the following error when running queries on Redash connected toDatabricks early today (2022-08-24):```Error running query: [HY000] [Simba][Hardy] (35) Error from server:error code: '0' error message:'org.apache.spark.sql.catalyst.expressions.U...
- 5061 Views
- 2 replies
- 2 kudos
- 2 kudos
This can be related to user permission, particularly necessary permission to access the table in the database instance. I understand in SQL editor it is working fine, still can we check the permissions?
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
