- 12954 Views
- 1 replies
- 0 kudos
Resolved! Problems with DLT, Unity catalog and external connection
I have following code:org = spark.read.table("catalog.dbo.organisation") @dlt.create_table() def organization(): return orgThe catalog is an external azure sql database (using external connector)When i validate this in Delta live table workflow I...
- 12954 Views
- 1 replies
- 0 kudos
- 0 kudos
use preview channel while create pipeline instead of current
- 0 kudos
- 3600 Views
- 2 replies
- 0 kudos
Unable to load data in DLT tables from Federated data sources
I tried to run this query and failing to load the data .What do I need to do load from federated data sources using DLT if this is not correct CREATE OR REPLACE LIVE TABLE bulkuploadhistory COMMENT 'Table generated for bulkuploadhistory.' TBLPROPERTI...
- 3600 Views
- 2 replies
- 0 kudos
- 0 kudos
Considering the databricks release cycles , is there any tenative dates that I can be aware of so that I can plan for my production releases
- 0 kudos
- 2583 Views
- 2 replies
- 0 kudos
Development Feedback Loop
I've noticed that the current development cycle for DLT jobs is quite time-consuming. The process of coding, saving, running in a workflow, and debugging seems arduous, and the feedback loop is slow. Is there a way to run DLT jobs without relying on ...
- 2583 Views
- 2 replies
- 0 kudos
- 12207 Views
- 2 replies
- 3 kudos
Resolved! How to write a Spark DataFrame to CSV file with our .CRC in Azure Databricks?
val spark:SparkSession = SparkSession.builder() .master("local[3]") .appName("SparkByExamples.com") .getOrCreate()//Spark Read CSV Fileval df = spark.read.option("header",true).csv("address.csv")//Write DataFrame to address directorydf.write...
- 12207 Views
- 2 replies
- 3 kudos
- 3 kudos
Will your csv have the name prefix 'part-' or can you name it whatever you like?
- 3 kudos
- 7317 Views
- 6 replies
- 0 kudos
Unsupported datatype 'TimestampNTZType' with liquid clustering
I'm experimenting with liquid clustering and have some questions about compatible types (somewhat similar to Liquid clustering with boolean columns ).Table created as CREATE TABLE IF NOT EXISTS <TABLE> ( _time DOUBLE , timestamp TIMESTAMP_NT...
- 7317 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi,just educated guess:There is limitation in liquid clustering docs: You can only specify columns with statistics collected for clustering keysPerhaps it is related to data types for which you can collect statistics?But i could not find related docs...
- 0 kudos
- 1893 Views
- 1 replies
- 0 kudos
Creating an ADLS storage credential for an AWS Workspace
I'd like to create a storage credential for an Azure Storage Account in an AWS workspace. I then plan to use this storage credential to create an external volume.Is this possible, and if so what are the steps? Thanks for any help!
- 1893 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your help.I'm struggling to create the Storage Credential. I have created a managed identity via an Azure Databricks Access Connector and am making an API call based on what I'm reading in the API docs: Create a storage credential | Storag...
- 0 kudos
- 1748 Views
- 0 replies
- 0 kudos
Downloading multiple excel files at once from repo
I have a notebook that produces lots of excel files which I want downloading on my local machine.I can only currently download one by one which takes a long time when there are a lot of them.Is there a way without using Azure CLI to download all of t...
- 1748 Views
- 0 replies
- 0 kudos
- 3234 Views
- 0 replies
- 0 kudos
how max_retry_interval_millis works with retry_on_timeout in Data bricks.
my project I want if job take longer time then it will terminate and again it will try even if there is timeout error and in databricks launched status should show retry by scheduler and it should follow min_retry_interval_millis before start retry...
- 3234 Views
- 0 replies
- 0 kudos
- 6635 Views
- 2 replies
- 1 kudos
Want to set execution termination time/timeout limit for job in job config
Hi , I Want to set execution termination time/timeout limit for job in job config file. please help me how I can do this by pass some parameter in job config file.
- 6635 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Pratibha You can configure optional duration thresholds for a job, including an expected completion time for the job and a maximum completion time for the job. To configure duration thresholds, click Set duration thresholds. If you are creating j...
- 1 kudos
- 3401 Views
- 0 replies
- 1 kudos
Stop a interactive cluster running a continuous job to allow for a restart
Hello,I am wondering if there is a way in Databricks to run a job continuously except for 1 or 2 hours every night in which the cluster could restart. We are using interactive clusters for our jobs and development in Dev and UAT. In Prod we are still...
- 3401 Views
- 0 replies
- 1 kudos
- 4344 Views
- 2 replies
- 1 kudos
Can we use "Access Connector for Azure Databricks" to access Azure SQL Server?
Is it possible to avoid using Service Principal (and managing their secrets) via the Python MSAL library and, instead, use the "Access Connector for Azure Databricks" to access Azure SQL Server (just like we do for connecting to Azure Data Lake Stora...
- 4344 Views
- 2 replies
- 1 kudos
- 1 kudos
Unfortunately, I guess the answer is no (as for today, see @Wojciech_BUK reply).
- 1 kudos
- 2450 Views
- 2 replies
- 1 kudos
Query endpoint on Azure sql or databricks?
Hi Currently all data reauired resides in Az sql database. We have a project in which we need to query on demand this data in Salesforce data cloud to be further used for reporting in CRMA dashboard.do we need to move this data from az sql to delta l...
- 2450 Views
- 2 replies
- 1 kudos
- 1 kudos
It depends. If Salesforce Data Cloud has a connector for AZ SQL (being a native one or odbc/jdbc), you can query directly. MS also has something like OData. AFAIK AZ SQL does not have a query API, only for DB-management purposes.If all above is no...
- 1 kudos
- 1458 Views
- 0 replies
- 1 kudos
Reports on list of users and roles that have access to Databricks workspace
Are there any in-built reports available in Databricks UI that list the users and roles with access to the workspace?
- 1458 Views
- 0 replies
- 1 kudos
- 6075 Views
- 0 replies
- 0 kudos
java.lang.OutOfMemoryError on Data Ingestion and Storage Pipeline
I have around 25GBs of data in my Azure storage. I am performing data ingestion using Autoloader in databricks. Below are the steps I am performing:Setting the enableChangeDataFeed as true.Reading the complete raw data using readStream.Writing as del...
- 6075 Views
- 0 replies
- 0 kudos
- 5834 Views
- 0 replies
- 0 kudos
How to run Powershell file script.ps1 using the databricks notebook
Hello All,Following command on running through databricks notebook is not working Command%sh# Bash code to print 'Hello, PowerShell!'echo 'Hello, PowerShell!'# powershell.exe -ExecutionPolicy Restricted -File /dbfs:/FileStore/Read_Vault_Inventory.ps1...
- 5834 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 570 | |
| 349 | |
| 287 |
