Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
856 Posts
Activity in Databricks Platform Discussions
- 22 Views
- 2 replies
- 0 kudos
Overwriting delta table takes lot of time
I'm trying simply to overwrite data into delta table. The Table size is not really huge it has 50 Mil of rows and 1.9Gb in size.For running this code I use various cluster configurations starting from 1 node cluster 64Gb 16 Vcpu and also I tried to s...
- 22 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Also, have you tried without the reparttion transformation? Have you also tried without autoOptimize options?
- 0 kudos
- 11 Views
- 1 replies
- 0 kudos
Any documentation mentioning connectivity from Azure SQL database connectivity to Azure Databricks
Any documentation available to connect from the Azure SQL database to Azure Databricks SQL workspace. We created a SQL warehouse personal access token for a user in a different team who can connect from his on-prem SQL DB to Databricks using the conn...
- 11 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Can I ask what types of operations do you plan on taking against Databricks from withing Azure SQL Database?
- 0 kudos
- 50 Views
- 3 replies
- 0 kudos
Resolved! Re-Ingest Autoloader files foreachbatch
Hi all,I'm using autoloader to ingest files, each files contains changed data from a table and I merge it into a delta table. It works fine.But if i want re-ingest all the files (deleting the checkpoint location, at example) i need to re-ingest the f...
- 50 Views
- 3 replies
- 0 kudos
- 200 Views
- 2 replies
- 0 kudos
Can't "run all below" - "command is part of a batch that is still running"
Weirdness in Databricks on AWS. In a notebook that is doing absolutely nothing, I click the "Run All Above" or "Run All Below" button on a cell, and it won't do anything at all except pop up a little message near the general "Run All" button, saying...
- 200 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @dkxxx-rc! Can you check if any background processes are still running in your notebook that might be interfering with new executions? If you are using Databricks Runtime 14.0 or above, cells run in batches, so any error halts execution, and in...
- 0 kudos
- 27 Views
- 0 replies
- 0 kudos
DAB - Common cluster configs possible?
I've been trying various solutions and perhaps maybe just thinking about this the wrong way.We're migrating over from Synapse where we're used to have a defined set of DBX Cluster profiles to run our jobs against, these are all job clusters created v...
- 27 Views
- 0 replies
- 0 kudos
- 41 Views
- 1 replies
- 0 kudos
Serverless Compute
Hi Team,We are using Azure Databricks Serverless Compute to execute workflows and notebooks. My question is :Does serverless compute support Maven library installations?I appreciate any insights or suggestions you might have. Thanks in advance for yo...
- 41 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Azure Databricks Serverless Compute does support Maven library installations; however, there are some important details and limitations to consider: Support for Maven Libraries: You can install libraries from a public Maven repository when working w...
- 0 kudos
- 2282 Views
- 1 replies
- 2 kudos
Accessing the regions that are disabled by default in AWS from Databricks. In AWS we have 4 regions that are disabled by default. You must first enabl...
Accessing the regions that are disabled by default in AWS from Databricks.In AWS we have 4 regions that are disabled by default. You must first enable it before you can create and manage resources. The following Regions are disabled by default:Africa...
- 2282 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Hello - We are looking to deploy and run Databricks in AWS in Bahrain, or UAE. Is this possible? This post is older so wondering if this is a viable option.
- 2 kudos
- 179 Views
- 3 replies
- 3 kudos
How to make .py files available for import?
Hello,I've looked around, but cannot find an answer. In my Azure Databricks workspace, users have Python notebooks which all make use of the same helper functions and classes. Instead of housing the helper code in notebooks and having %run magics in ...
- 179 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi Brahmareddy,Thanks for your reply. Your second approach is quite close to what I already tried earlier. Your post got me to do some more testing, and although I don't know how to set the sys.path via the init script (it says here and here that it'...
- 3 kudos
- 99 Views
- 2 replies
- 1 kudos
Schema updating with CI/CD development in SQL
Hi all,I am working to resolve how to build tables in a development workspace catalog and then easily migrate the code to a production catalog without manually altering the schema name. For those unaware, you can't have the same catalog names in deve...
- 99 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Thanks for this. Now to work out how much I want to work out alembic or dbt. I don't see any reason to go with Liquibase.Still an area for some improvements, actually a lot of improvement. Being able to build manageable, governed data warehouse schem...
- 1 kudos
- 923 Views
- 9 replies
- 4 kudos
(Big) Problem with SQL Warehouse Auto stop
Long story short, I'm not sure if this is an already known problem, but the Auto Stop feature on SQL Warehouses after minutes of inactivity is not working properly.We started using SQL Warehouses more aggressively this December when we scaled up one ...
- 923 Views
- 9 replies
- 4 kudos
Latest Reply
- 4 kudos
One thing we noticed is that the "Monitoring" tab of the SQL Warehouse page can be misleading because it doesn't always account for time spent fetching & serving the data to the client, only the actual query execution time. This was confirmed by Supp...
- 4 kudos
- 67 Views
- 2 replies
- 0 kudos
Problem with df.first() or collect() when collation different from UTF8_BINARY
I'm getting a error when I want to select the first() or collect() from a dataframe when using a collation different than UTF8_BINARYExample that reproduces the issue :This works :df_result = spark.sql(f""" SELECT 'en-us' AS ET...
- 67 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @MDV I guess the issue likely comes from how non-default collations like UTF8_LCASE behave during serialization when using first() or collect(). As a workaround wrap the value in a subquery and re-cast the collation back to UTF8_BINARY before acce...
- 0 kudos
- 49 Views
- 1 replies
- 0 kudos
Delta live tables - ignore updates on some columns
Hello Team,I have scenario where in apply_changes, i want to ignore updates on 1 column. Is there any way we can achieve this in Delta live tables?
- 49 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi there @Anish_2 , Yes you can do that Here is the doc link : https://docs.databricks.com/aws/en/dlt/cdc?language=PythonFor python you can simply add an attribute except_columns_list like thisdlt.apply_changes( target = "target", source = "users...
- 0 kudos
- 79 Views
- 3 replies
- 1 kudos
Resolved! Dynamic inference tasks in workflows using dabs
I have some workflows where we use dynamic inference to set task values or capture job executions counts or output rows. Is there any way I can set these dynamic values using the ui but can i do the same at the time of dabs workflow creation. Can you...
- 79 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Thanks @ashraf1395 , I got the idea of what I was looking for.
- 1 kudos
- 27 Views
- 0 replies
- 0 kudos
Create Catalog via UI
Hi ! I started using Databricks one month ago. I have a question.When I try to create a catalog, I can do it using SQL, but I can't find the "Create Catalog" option in the UI.Do you know what might be causing this?I read the following document, but i...
- 27 Views
- 0 replies
- 0 kudos
- 32 Views
- 0 replies
- 0 kudos
Read file from AWS S3 using Azure Databricks
Hi TeamI am currently working on a project to read CSV files from an AWS S3 bucket using an Azure Databricks notebook. My ultimate goal is to set up an autoloader in Azure Databricks that reads new files from S3 and loads the data incrementally. Howe...
- 32 Views
- 0 replies
- 0 kudos
