Hi I'm experiencing a problem when installing UCX Assessment through an automation script in Databricks. The script fails with this error:13:38:06 WARNING [databricks.labs.ucx.hive_metastore.tables] {listing_tables_0} failed-table-crawl: listing tabl...
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
With more students turning to online learning, platforms like Colegia.me are proving to be incredibly valuable. It’s efficient, reliable, and built for modern learners.
I'm looking at this page (Databricks Asset Bundles development work tasks) in the Databricks documentation.When repo assets are deployed to a databricks workspace, it is not clear if the "databricks bundle deploy" will remove files from the target wo...
@JamesGraham this issue is related to "databricks bundle deploy" command itself, when run inside ci/cd pipeline, i am still seeing old configs in bundle.tf.json. Ideally it should be updated to changes done from previous run. But i am still seeing er...
I want to test create a custom connector in a Power App that connects to table in Databricks. The issue is if I have any columns like int or bigint. No matter what I define in the response in my swagger definition See below), it is not correct type...
Hi @LRALVA This might warrant another post to keep the conversation focussed, but I found a couple of things with the custom connector that make it a bit cumbersome to use.1) I don't seem to be able to have two post operations under /statments so I c...
Hi everyone,I'm currently working on an ETL process using Azure Databricks (Standard Tier) where I load data from Azure SQL Database into Databricks. I run a notebook daily to extract, transform, and load the data for Power BI reports.Right now, the ...
In case you do not want to use dlt (and there are reasons not to), you can also check the docs for autoloader and merge notebooksThese 2 do basically the same as dlt but without the extra cost and more control. You have to write more code though.For...
Is there any way to export the ​output data in the Excel format into the dbfs?, I'm only able to do it in the CSV format
As shared above I tested it and worked fine for loading and updating and saving import openpyxlwb = openpyxl.load_workbook('Test.xlsx')ws = wb.activefor row in ws.iter_rows():print([col.value for col in row]) #show all dataws['A1']='Data'wb.save('Tes...
Hello,Let say we have an empty table S that represents the schema we want to keepABCDEWe have another table T partionned by column A with a schema that depends on the file we have load into. Say :ABCF1b1c1f12b2c2f2Now to make T having the same schema...
Hi @oneill, please check this if it helps for your use case: https://docs.databricks.com/aws/en/delta/selective-overwrite#dynamic-partition-overwrites
I'm observing different behavior between Databricks Runtime versions when working with DataFrames and temporary views, and would appreciate any clarification.In both environments, I performed the following steps in a notebook (each connected to its o...
Hi @Einsatz, this is expected in DBR version 14.3 and above since we don't have a Spark context. This is happening due to the cache invalidation. To resolve the issue, please go with the dynamic name for the view every time.
Hello everyone,I'm working on analyzing cost projections for Serverless SQL Warehouses using system tables, and I’d like to share a visualization approach I’m using to highlight some key differences between classic and serverless SQL warehouses. (Loo...
Hi @mstfkmlbsbdk, great analysis. However, as you mentioned, the classic warehouse is not terminating even though there are no active queries on it. The reason for that is while creating a classic warehouse, by default it takes 45 minutes to termina...
I'd like to have a couple of widgets, one for the start and another for end date. I want them to appear in that order but when I run the code below, end date shows up before the start date. How can order in the way I I desired?dbutils.widgets.text("s...
Doing it manually is not optimal. It should be possible to do this automatically - by creation date or extra function to sort widgets.
Hi everyone,I'm currently working on an ETL process using Azure Databricks (Standard Tier) where I load data from Azure SQL Database into Databricks. I run a notebook daily to extract, transform, and load the data for Power BI reports.Right now, the ...
Hi @chexa_Wee, you can leverage DLT feature to do so. Please check: https://docs.databricks.com/aws/en/dlt/transform https://docs.databricks.com/aws/en/dlt/stateful-processing Here is the step-by-step tutorial: https://docs.databricks.com/aws/en/dlt...
Why can't we just copy all the DLT tables and materialized views from one UC catalog to another to get the historical data in place and then run the DLT pipelines on those UC tables?We are migrating many very large tables from our TEST catalog to our...
Hi @BF7, we do not support moving the streaming table yet. If we clone the DLT streaming table, it will get converted into a normal delta table instead of a streaming table. In that condition, we need to go with the "full refresh all" option and inde...
We have a bunch of cloud AD groups in Databricks and I can see what users are in each group by using the user interface Manage Account -> Users and groups -> GroupsI would like to be able to produce this full list in SQL. I have found the below code ...
Ok. Well, my last suggestion is to have a look at the SCIM Users API, and SCIM Groups API. You should be able to make the API calls right in the notebook. Cheers, Lou.
Does DBFS support GCS?
Yes you need just to create service account for databricks and than assign storage admin role to bucket. After that you can mount GCS standard way:bucket_name = "<bucket-name>" mount_name = "<mount-name>" dbutils.fs.mount("gs://%s" % bucket_name, "/m...
In some workspaces, I can get system.compute.warehouses table But in some other workspaces, it is not available how can i enable it?Both are in same account but assigned to different metastore.
| User | Count |
|---|---|
| 1612 | |
| 769 | |
| 348 | |
| 286 | |
| 252 |
