- 2799 Views
- 4 replies
- 2 kudos
call saved query in sql warehouse
in python cursor.executecan you call a saved query with a parameter? like call a stored procedure in relational db?https://docs.microsoft.com/en-us/azure/databricks/dev-tools/python-sql-connector#cursor-method
- 2799 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @lizou​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 2 kudos
- 19695 Views
- 4 replies
- 2 kudos
Flatten a complex JSON file and load into a delta table
Hi,I am loading a JSON file into Databricks by simply doing the following:from pyspark.sql.functions import *from pyspark.sql.types import *bronze_path="wasbs://....../140477.json"df_incremental = spark.read.option("multiline","true").json(bronze_pat...
- 19695 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Lloyd Vickery​ Does @Werner Stinckens​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 2 kudos
- 22628 Views
- 6 replies
- 6 kudos
Resolved! How to display markdown output in databricks notebook from a python cell
With IPython/Jupyter it's possible to output markdown using the IPython display module and its `MarkDown`class. QuestionHow can I accomplish this with Azure Databricks?What I triedDatabricks `display`Tried using Databrick's display with the IPython M...
- 22628 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi, Thanks for reaching out to community.databricks.com.In a notebook cell, type "%md" and type some markdown and it will render. Please refer: https://community.databricks.com/s/question/0D53f00001HKHhNCAX/markup-in-databricks-notebook
- 6 kudos
- 1567 Views
- 1 replies
- 0 kudos
Can we have a better UI for navigating Workspace and Repos?
Navigating through multiple vertical panes of information as we navigate deeper into a folder structure is not very convenient -- we lose the context of parent folder and sibling folders very soon.Can we not have a simple tree view (similar to VS Cod...
- 1567 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey there @Al Jo​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
- 0 kudos
- 1327 Views
- 1 replies
- 1 kudos
Multiple Jobs with different resource requirements on the same cluster
I have a big cluster with the auto-scaling(min:1, max: 25) feature enabled. I want to run multiple jobs on that cluster with different values of spark properties( `--executor-cores` and `–executor-memory) but I don't see any option to specify the sam...
- 1327 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Neelesh databricks​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell ...
- 1 kudos
- 946 Views
- 1 replies
- 3 kudos
Can I use PrivateLink and Customer Managed Policy for Cross Account Role
Hello, Im trying to enable Privatelink on my AWS Databricks quickstart, ​I use the customer managed VPC policy for the ​cross account role and supply it on the template. Im having an error that it cannot create a VPC Endpoint.​Do i need to change the...
- 946 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Chris Joshua Manuel​ , Thanks for reaching out to Community.databricks.com. Cross account VPC access works. Please refer below: https://tomgregory.com/cross-account-vpc-access-in-aws/Also, please let us know in case of any further clarification n...
- 3 kudos
- 2252 Views
- 3 replies
- 0 kudos
- 2252 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, I believe you are using a single tenant shard where SQL option is not there.
- 0 kudos
- 3078 Views
- 2 replies
- 2 kudos
Resolved! Errors on Redash when queries are in cache
We got the following error when running queries on Redash connected toDatabricks early today (2022-08-24):```Error running query: [HY000] [Simba][Hardy] (35) Error from server:error code: '0' error message:'org.apache.spark.sql.catalyst.expressions.U...
- 3078 Views
- 2 replies
- 2 kudos
- 2 kudos
This can be related to user permission, particularly necessary permission to access the table in the database instance. I understand in SQL editor it is working fine, still can we check the permissions?
- 2 kudos
- 2043 Views
- 2 replies
- 1 kudos
Resolved! why this change in UI
The change in the UI is really confusing on what to use where. Earlier i had HC clusters and now I cant find it in new UI. It says HC clusters are not available. I want to use the HC cluster functions. where can I get that?
- 2043 Views
- 2 replies
- 1 kudos
- 1784 Views
- 2 replies
- 3 kudos
Resolved! Azure Databricks Phiton
Does the photon engine work against Azure SQL? Is there any documentation for this?​
- 1784 Views
- 2 replies
- 3 kudos
- 3 kudos
Photon is a type of engine available within databricks workspace..In data engineering, you can opt to have photon enabled on your cluster, whereas in Databricks SQL, photon is enabled by default.
- 3 kudos
- 2605 Views
- 2 replies
- 1 kudos
Using built-in SQL functions in Delta Live tables
Do Delta Live Tables have different built-in SQL functions than the corresponding Databricks runtime? I created a cluster with Databricks runtime 10.3 (the current DLT runtime) so I could test my Delta Live Tables code before running it as a pipeline...
- 2605 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Colt Kesselring​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 1 kudos
- 1635 Views
- 2 replies
- 1 kudos
How to get the list of parameters passed from widget
Hi ,Could someone help me understand how I would be able to get all the parameters in the task (from the widget). ie I want to get input as parameter 'Start_Date' , but the case is that this will not always be passed . It could be 'Run_Date' as well ...
- 1635 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Bency Mathew​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 1 kudos
- 3431 Views
- 3 replies
- 1 kudos
SQL query with leads and lags
I'm trying to create a new column that fills in the nulls below. I tried using leads and lags but isn't turning out right. Basically trying to figure out who is in "possession" of the record, given the TransferFrom and TransferTo columns and sequence...
- 3431 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi there @Eric Lohbeck​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 4735 Views
- 1 replies
- 0 kudos
Setting up custom tags (JobName, JobID, UserId) on an all-purpose cluster
Hi i want to set up custom tags on an all-purpose cluster for purposes of cost break down and chargebacks. What: specifically, i want to capture JobName, JobID, UserId who ran jobI can set other custom tags such as Business Unit, Owner... However,...
- 4735 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey there @DIEUDONNE MAYI​ Does @Kaniz Fatma​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 0 kudos
- 1886 Views
- 2 replies
- 1 kudos
About privileges that clone a Git repository on Databricks
Hi,​allDo I need admin privileges to clone a Git repository on Databricks?​Cloning was not possible with an account that did not have administrator privileges.​Regards.
- 1886 Views
- 2 replies
- 1 kudos
- 1 kudos
Navigate to Settings > admin console > Workspace settings > Repos and check the value for "Repos Git URL Allow List permissions".When set to 'Disabled (no restrictions)', users can clone or commit and push to any Git repository.When set to 'Restrict ...
- 1 kudos