- 12087 Views
- 3 replies
- 2 kudos
how do we use delta sharing between databricks to snowflake
Hi all,Is there any way to implement delta sharing in databricks to snowflake direct connect ?
- 12087 Views
- 3 replies
- 2 kudos
- 2 kudos
I don't think that Snowflake has implemented the ability to read from a table via Delta Sharing as of December 2023. Please reach out to your Snowflake representatives and urge them to consider this feature from their side. Alternatively, you can qu...
- 2 kudos
- 5905 Views
- 1 replies
- 0 kudos
AWS S3 External Location Size in Unity Catalog
Hi,I am trying to get the raw size (total size) of delta table. I could get delta table size from DeltaTable api but that gives only latest version size. I need to find the actual S3 size the tables takes on S3.Is there any way, to find the S3 size ...
- 5905 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Kaniz,Thank you for your suggestions. As per my understanding, the "snapshot.sizeInBytes" gives only current snapshot size. But I am looking for total size (all versions) of the table on S3.
- 0 kudos
- 4588 Views
- 3 replies
- 0 kudos
The operation CHANGE DATA FEED is not allowed on Streaming Tables.
Hello everyone,I have a workflow that starts by reading the CDF data for a change data feed.The syntax is exactly the following : (spark.readStream .format("delta") .option("readChangeFeed", "true") .option("startingVersion", 10) .table("my.str...
- 4588 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, this seems to be related to the issue I've been getting around the same time here: Change data feed from target tables of APPLY CHANG... - Databricks - 54436Would be great to get an explanation for the sudden change in behaviour.
- 0 kudos
- 1402 Views
- 0 replies
- 0 kudos
Access from DBT job to Azure DevOps repository using Service Principal
Hi,We are using Databricks bundles to deploy our DBT project. Everything is set up to deploy and run as a Service Principal.The DBT job is connected to an Azure DevOps repository. The problem is that we cannot find a way to properly authenticate the ...
- 1402 Views
- 0 replies
- 0 kudos
- 1672 Views
- 1 replies
- 0 kudos
Save default language of notebook into variable dynamically
For one of the requirements of project, I want to save default language of notebook into variable based on notebook path mentioned dynamically.For eg: if first notebook given by user in widget is having default language as Python then variable value...
- 1672 Views
- 1 replies
- 0 kudos
- 1769 Views
- 0 replies
- 0 kudos
Wrong FS: abfss://....., expected: dbfs:/ Error in DLT pipeline
Dear Databricks community members:SymptomReceived the error for a delta load, after a successful initial load with a Unity Catalog Volume as a data source.org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = xxx, runId...
- 1769 Views
- 0 replies
- 0 kudos
- 4055 Views
- 1 replies
- 0 kudos
Thousands of ETL pipelines with long execution times and small dataset sizes
Hi,I work for a small company, we're mostly focussing on small retail and e-commerce customers. We provides data analysis and automated data connections between their platforms. Most of our datasets are things like order data, google ads click data, ...
- 4055 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Thanks for the information, there is a lot to unpack and some assumptions that need to be made without fully understanding the details, so here are a few thoughts: If the cluster start times longer because of the libraries you're installing, can ...
- 0 kudos
- 3340 Views
- 1 replies
- 0 kudos
Query Delta table from .net
Hi Team,How can expose data stored in delta table through API like exposing sql data through .net api?
- 3340 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use the SQL Statement Execution API.At energinet.dk we have created this open-source .NET client, which we use internally in the company.
- 0 kudos
- 8281 Views
- 2 replies
- 3 kudos
Resolved! best way to store config files in a Unity workspace (Scala/typesafe)
We use typesafe (scala) to read configuration values from hocon files.When not using Unity, we read the configuration files from /dbfs/... works fine.However, with Unity, usage of dbfs is frowned upon.So I started looking into alternatives.And unfor...
- 8281 Views
- 2 replies
- 3 kudos
- 3 kudos
In the end we will continue to use dbfs. Maybe in the future when volumes are supported by scala io we can re-evaluate, but for now dbfs seems the way to go.
- 3 kudos
- 3877 Views
- 1 replies
- 6 kudos
I am getting an SSLError: HTTPSConnectionPool while making a call to https restapis from azure databricks I have tried to set a verify=false parameter in the call too.
response = requests.request("POST", url, verify=False, headers=headers, data=payload) SSLError: HTTPSConnectionPool(host='dcs.adobedc.net', port=443): Max retries exceeded with url: /collection/d99e6dfcffb0b5aeaec2cf76cd3bc2b9e9c414b0c74a528d13dd39...
- 3877 Views
- 1 replies
- 6 kudos
- 6 kudos
Any luck with this?
- 6 kudos
- 2521 Views
- 2 replies
- 0 kudos
Resolved! Regarding Exam got suspended
Hello Team,I had a disappointing experience while attempting my first DataBricks certification. Abruptly, the proctor asked me to show my desk, and after complying. Eventually, they suspended my exam, citing excessive eye movement and other practices...
- 2521 Views
- 2 replies
- 0 kudos
- 0 kudos
@mkrish28 I'm sorry to hear you had this experience. Thank you for logging at ticket with the support team. They have informed me they have rescheduled your exam. Good luck!
- 0 kudos
- 2299 Views
- 1 replies
- 1 kudos
DBR 14.1 - foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.
I am getting this error on DBR 14.1AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.This is the code: wstream = df.writeStream.foreac...
- 2299 Views
- 1 replies
- 1 kudos
- 2232 Views
- 1 replies
- 0 kudos
CANNOT_RENAME_ACROSS_SCHEMA message error
Hello...We enabled Unity Catalog and we are migrating schemas. When I ran the command sync schema catalog01.schema01 FROM hive_metastore.schema01 dry run I got the error CANNOT_RENAME_ACROSS_CATALOG, reviewing your documentation it only said CANNO...
- 2232 Views
- 1 replies
- 0 kudos
- 9342 Views
- 1 replies
- 0 kudos
List parameter in Python SQL connector 3.0.1
Hi,up to recently in version of the Python SQL connector 2.9.3, I was using a list as a parameter in the cursor.execute(operation, parameters) method without any trouble. It seems that it is not possible anymore in version 3.0.1 as the parsing of par...
- 9342 Views
- 1 replies
- 0 kudos
- 0 kudos
I should better read the documentation : https://github.com/databricks/databricks-sql-python/blob/v3.0.0/docs/parameters.md.
- 0 kudos
- 979 Views
- 0 replies
- 0 kudos
Retrieving Logs with Job API Get-outputs service
Hello,I would like to retrieve the logs of some job that where launched using the Job Rest Api 2.0. I see in the doc that can be done with the service get-ouputs, however each time I check the service I just get the metadata part of the response but ...
- 979 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
