Can we please update Databricks so that running multiple SQL queries in the same cell displays the results of all the queries? Right now, if you run multiple SQL queries in a single Databricks cell, you only see the results of the last query.Anirban ...
Can anyone please look into https://community.databricks.com/t5/community-discussions/dab-not-updating-zone-id-when-redeployed/td-p/71655 zone_id will update fromauto --> ap-south-1aauto --> ap-south-1bap-south-1a <--> ap-south-1b But not fromap-sou...
Scala, I am trying to write dataframe to sqlmi table, it works for all other table when dataframe columns matches table columns, and I created a new table with the identity and auto-initialized column, from what I understand, by default, I don't need...
@JamesY Try adding the missing columns as a dummy columns. I've seen this issue happening some time ago - it looks like it's still not fixed.
Is there a way to get the directory size in ADLS(gen2) using dbutils in databricks? If I run this dbutils.fs.ls("/mnt/abc/xyz") I get the file sizes inside the xyz folder( there are about 5000 files), I want to get the size of the XYZ folder how ca...
File size is only specified for files. So, if you specify a directory as your source, you have to iterate through the directory. The below snippet should work (and should be faster than the other solutions).import glob def get_directory_size_in_byt...
Hi ExpertI have created an external table in databricks community edition. Table is external table. But when i cluster is terminated, i am not able to query the table any more. What is the reason? What i need to do so that table is not dropped. Table...
Separation of Storage and Compute:Databricks separates storage from compute. Data stored in DBFS (Databricks File System) or external storage systems is persistent and not tied to the lifecycle of a cluster.When you create tables and databases in Dat...
Hi, I've recently been prototyping on Databricks, I was hoping to develop using DLT pipelines in medallion architecture but with isolation of bronze/silver & gold layers in different catalogs in UC for security purposes.At the moment is there a lim...
Hi @Frustrated_DE ,This seems to be a long-requested feature based on this previous post: Solved: Re: DLT pipeline - Databricks Community - 45740An alternative solution maybe to delegate permissions on the table level for this pipeline specifically. ...
Hello, I need to read secrets_value from AWS Secrets Manager, reading via python using boto3 I can retrieve the secret.I need to run it on a notebook, is it possible to read without informing the credentials, only through the IAM role? I tried to add...
If you're trying to run the Ataccama profiler on tables with multiple joins or are incredibly large, please note that there are some processes on the Ataccama profiler that will lead to bad performance issues. If you are having jobs that are crashing...
Hi, I have datbricks on AWS, I created some secrets in AWS Secrets Manger, I would need to create the scopes based on AWS secrets manager.When I use Azure's Key Vault, when creating the scope, it uses the option -scope-backend-type AZURE_KEYVAULT, bu...
Hi @thiagoawstest Step 1: Create Secret ScopeYou can create a secret scope using the Databricks REST API as shown below: python import requests import json # Define the endpoint and headers url = "https://<databricks-instance>/api/2.0/secrets/scope...
I am trying to work on a Databricks notebook that can dynamically create alerts using the Databricks API (Documentation linked here: https://docs.databricks.com/api/azure/workspace/alerts/list). I can successfully create an alert, however what is mis...
Hi Saugat,Is there a way that i can get the owner of a schema/table dynamically and send out a notification to him using Rest API.
When trying to use `dbutils.fs.cp` in the #databricks-connect #databricks-connect context to upload files to Azure Datalake Gen2 I get a malformed URI errorI have used the code provided here:https://learn.microsoft.com/en-gb/azure/databricks/dev-tool...
I noticed that enabling photon acceleration is increasing the number of DBU utilized per hour which in turn increases our cost.In light of this, I am interested in gaining clarity on the costing of Photon acceleration as I was led to believe that Pho...
well that depends on what kinds of tests you do. In data warehousing there are different kinds of loads.What have you tested? Data transformations or analytical queries. Because for the latter databricks sql is a better choice than a common spark ...
Frequently I see this syntax to access a series in DBX. df['column_name'] However, I get this as my output from that.Column<'derived_value'>What's the correct way to access a series
I realized I was looking at the wrong dataframe type.I needed a Pandas dataframe, not a databricks dataframe.
When i tried to create a table name with variable like this-- Set a string variableSET table_suffix = 'suffix';-- Use dynamic SQL to create a table with the variable as a suffix in the table nameCREATE TABLE IF NOT EXISTS <dbname>.my_table_${table_su...
Hi,It's possible that the `identifier` clause is what you're looking for (https://docs.databricks.com/en/sql/language-manual/sql-ref-names-identifier-clause.html#identifier-clause). If so, this basic example should work: DECLARE mytab = '`mycatalog`....
How to add a column from Table1 as a filter to Table2 (Also called as on-click action filter) in databricks Dashboards?Both the tables are getting data through sql query
To add a column from Table1 as a filter to Table2 in Databricks Dashboards, you can use the dashboard parameters feature. Here are the steps: Create a visualization for each of your SQL queries. You can do this by clicking the '+' next to the Resul...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group