- 7521 Views
- 3 replies
- 0 kudos
Resolved! ignoreCorruptFiles behavior with CSV and COPY INTO
HiI'm using the COPY INTO command to insert new data (in form of CSVs) into an already existing table.The SQL query takes care of the conversion of the fields to the target table schema (well, there isn't other way to do that), and schema update is n...
- 7521 Views
- 3 replies
- 0 kudos
- 0 kudos
I actually found an option that could solve the newline issue I mentioned in my previous post:setting spark.sql.csv.parser.columnPruning.enabled to false withspark.conf.set("spark.sql.csv.parser.columnPruning.enabled", False)will consider malformed r...
- 0 kudos
- 4012 Views
- 1 replies
- 0 kudos
Best practice for Azure Key vault secrets in spark config
HelloI created a compute in which I refer the secret inside the spark config like this: spark.hadoop.fs.azure.account.key.xxxxxxxxxx.dfs.core.windows.net {{secrets/kv-xxxxxxx-xxxx/secret-name}} This, however, gives me the following warning. I've l...
- 4012 Views
- 1 replies
- 0 kudos
- 0 kudos
Extra info: I have used the format following the instructions on this page for spark configurationhttps://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage#:~:text=Use%20the%20following%20format%20to%20set%20the%20cluster%20Spa...
- 0 kudos
- 12958 Views
- 1 replies
- 0 kudos
Resolved! Problems with DLT, Unity catalog and external connection
I have following code:org = spark.read.table("catalog.dbo.organisation") @dlt.create_table() def organization(): return orgThe catalog is an external azure sql database (using external connector)When i validate this in Delta live table workflow I...
- 12958 Views
- 1 replies
- 0 kudos
- 0 kudos
use preview channel while create pipeline instead of current
- 0 kudos
- 3602 Views
- 2 replies
- 0 kudos
Unable to load data in DLT tables from Federated data sources
I tried to run this query and failing to load the data .What do I need to do load from federated data sources using DLT if this is not correct CREATE OR REPLACE LIVE TABLE bulkuploadhistory COMMENT 'Table generated for bulkuploadhistory.' TBLPROPERTI...
- 3602 Views
- 2 replies
- 0 kudos
- 0 kudos
Considering the databricks release cycles , is there any tenative dates that I can be aware of so that I can plan for my production releases
- 0 kudos
- 2583 Views
- 2 replies
- 0 kudos
Development Feedback Loop
I've noticed that the current development cycle for DLT jobs is quite time-consuming. The process of coding, saving, running in a workflow, and debugging seems arduous, and the feedback loop is slow. Is there a way to run DLT jobs without relying on ...
- 2583 Views
- 2 replies
- 0 kudos
- 12213 Views
- 2 replies
- 3 kudos
Resolved! How to write a Spark DataFrame to CSV file with our .CRC in Azure Databricks?
val spark:SparkSession = SparkSession.builder() .master("local[3]") .appName("SparkByExamples.com") .getOrCreate()//Spark Read CSV Fileval df = spark.read.option("header",true).csv("address.csv")//Write DataFrame to address directorydf.write...
- 12213 Views
- 2 replies
- 3 kudos
- 3 kudos
Will your csv have the name prefix 'part-' or can you name it whatever you like?
- 3 kudos
- 7320 Views
- 6 replies
- 0 kudos
Unsupported datatype 'TimestampNTZType' with liquid clustering
I'm experimenting with liquid clustering and have some questions about compatible types (somewhat similar to Liquid clustering with boolean columns ).Table created as CREATE TABLE IF NOT EXISTS <TABLE> ( _time DOUBLE , timestamp TIMESTAMP_NT...
- 7320 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi,just educated guess:There is limitation in liquid clustering docs: You can only specify columns with statistics collected for clustering keysPerhaps it is related to data types for which you can collect statistics?But i could not find related docs...
- 0 kudos
- 1893 Views
- 1 replies
- 0 kudos
Creating an ADLS storage credential for an AWS Workspace
I'd like to create a storage credential for an Azure Storage Account in an AWS workspace. I then plan to use this storage credential to create an external volume.Is this possible, and if so what are the steps? Thanks for any help!
- 1893 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your help.I'm struggling to create the Storage Credential. I have created a managed identity via an Azure Databricks Access Connector and am making an API call based on what I'm reading in the API docs: Create a storage credential | Storag...
- 0 kudos
- 1748 Views
- 0 replies
- 0 kudos
Downloading multiple excel files at once from repo
I have a notebook that produces lots of excel files which I want downloading on my local machine.I can only currently download one by one which takes a long time when there are a lot of them.Is there a way without using Azure CLI to download all of t...
- 1748 Views
- 0 replies
- 0 kudos
- 3236 Views
- 0 replies
- 0 kudos
how max_retry_interval_millis works with retry_on_timeout in Data bricks.
my project I want if job take longer time then it will terminate and again it will try even if there is timeout error and in databricks launched status should show retry by scheduler and it should follow min_retry_interval_millis before start retry...
- 3236 Views
- 0 replies
- 0 kudos
- 6638 Views
- 2 replies
- 1 kudos
Want to set execution termination time/timeout limit for job in job config
Hi , I Want to set execution termination time/timeout limit for job in job config file. please help me how I can do this by pass some parameter in job config file.
- 6638 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Pratibha You can configure optional duration thresholds for a job, including an expected completion time for the job and a maximum completion time for the job. To configure duration thresholds, click Set duration thresholds. If you are creating j...
- 1 kudos
- 3403 Views
- 0 replies
- 1 kudos
Stop a interactive cluster running a continuous job to allow for a restart
Hello,I am wondering if there is a way in Databricks to run a job continuously except for 1 or 2 hours every night in which the cluster could restart. We are using interactive clusters for our jobs and development in Dev and UAT. In Prod we are still...
- 3403 Views
- 0 replies
- 1 kudos
- 4345 Views
- 2 replies
- 1 kudos
Can we use "Access Connector for Azure Databricks" to access Azure SQL Server?
Is it possible to avoid using Service Principal (and managing their secrets) via the Python MSAL library and, instead, use the "Access Connector for Azure Databricks" to access Azure SQL Server (just like we do for connecting to Azure Data Lake Stora...
- 4345 Views
- 2 replies
- 1 kudos
- 1 kudos
Unfortunately, I guess the answer is no (as for today, see @Wojciech_BUK reply).
- 1 kudos
- 2450 Views
- 2 replies
- 1 kudos
Query endpoint on Azure sql or databricks?
Hi Currently all data reauired resides in Az sql database. We have a project in which we need to query on demand this data in Salesforce data cloud to be further used for reporting in CRMA dashboard.do we need to move this data from az sql to delta l...
- 2450 Views
- 2 replies
- 1 kudos
- 1 kudos
It depends. If Salesforce Data Cloud has a connector for AZ SQL (being a native one or odbc/jdbc), you can query directly. MS also has something like OData. AFAIK AZ SQL does not have a query API, only for DB-management purposes.If all above is no...
- 1 kudos
- 1459 Views
- 0 replies
- 1 kudos
Reports on list of users and roles that have access to Databricks workspace
Are there any in-built reports available in Databricks UI that list the users and roles with access to the workspace?
- 1459 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
