Is there a way to create a global ini file that will reference databricks-backed secrets? Not from Azure, we use databricks on AWS.
I have written a CTE in Spark SQL WITH temp_data AS ( ...... ) CREATE VIEW AS temp_view FROM SELECT * FROM temp_view; I get a cryptic error. Is there a way to create a temp view from CTE using Spark SQL in databricks?
In the CTE you can't do a CREATE. It expects an expression in the form of expression_name [ ( column_name [ , ... ] ) ] [ AS ] ( query )where expression_name specifies a name for the common table expression.If you want to create a view from a CTE, y...
if i update the value in xml then autoloader not detecting the changes.same for delete/remove column or property in xml. So request to you please help me to fix this issue
It seems that the issue you're experiencing with Autoloader not detecting changes in XML files might be related to how Autoloader handles schema inference and evolution. Autoloader can automatically detect the schema of loaded XML data, allowing you...
HiI'm trying to deploy the databricks jobs from dev to prod environment. I have jobs in dev environment and using azure devops, I deployed the jobs in the code format to prod environment. Now when I use the post method to create the job programmatica...
@SyedGhouri You need to setup self-hosted Azure DevOps Agent inside your VNET.
Hi,Is there a quick and easy way to copy files between different environments? I have copied a large number of files on my dev environment (unity catalog) and want to copy them over to production environment. Instead of doing it from scratch, can I j...
If you want to copy files in Azure, ADF is usually the fastest option (for example TB of csvs, parquets). If you want to copy tables, just use CLONE. If it is files with code just use Repos and branches.
Do asset bundles support DLT pipelines unity catalog as a destination? How to specify catalog and target schema?
Looked through some previous posts and documentation and couldn't find anything related to use of Git stash in Databricks Repos. Perhaps I missed it. I also don't see an option in the UI.Does anyone know if there's a way to stash changes either in th...
This is actually a big hurdle when trying to switch between working in two different branches, it would be a welcome addition to the Databricks IDE.
I have used .option("cloudFiles.schemaEvolutionMode", "addNewColumns")\ for newly added property in xml file but autoloader not detected the changes. As per .option("cloudFiles.schemaEvolutionMode", "addNewColumns")\ behavior it has failed at first t...
I'm building my own Docker images to use for a cluster. The problem is that the only image I seem to be able to run is the official base image "databricksruntime/python:13.3-LTS". If I install a pip package, I get the following on standard error:/dat...
I found the culprit: --ignore-installed upgraded matplotlib too much, and broke it.
I have developed a azure databricks notebook where data will be copied from landing zone to STG delta table, used Try and except blocks in the code to catch the errors, if their is an error the except block will catch the error message. In the except...
below is my code
R2 (egress-free) can now be quickly registered as an external location. You can use it not only for Delta Sharing! #databricks
Thank you for sharing this @Hubert-Dudek!!!
I need to delete 50TB of data out of dfbs storage. It is overpartitioned and dbutils does not work. Also, limiting partition size and iterating over data to delete doesn't work. Azure locks access from storage from the resource group permissions and ...
For anyone else with this issue, there is no solution other than deleting the whole databricks workspace which then deletes all the resources locked up in the managed resource group. The data could not be deleted in any other way, not even by Microso...
I'm using the Databricks Connect VS Code plugin. It's cool how it figures out what things need to be run on the cluster vs. run locally. However, is it possible to force it to run specific Python statements remotely instead of locally?For context, th...
Aim-Installation of external libraries(wheel file) in Data bricks through synapse using new job clusterSolution- I have followed the below steps:I have created a pipeline in synapse that consists of a notebook activity that is using a new job cluster...
I am unable to display the below stream after reading it.df= spark.readStream.format("cloudFiles")\.option("cloudFiles.format", "csv")\.option("header", "true")\.option("delimiter", "\t")\.option("inferSchema", "true")\.option("cloudFiles.connectionS...
| User | Count |
|---|---|
| 1612 | |
| 769 | |
| 348 | |
| 286 | |
| 252 |
