- 5151 Views
- 3 replies
- 0 kudos
The operation CHANGE DATA FEED is not allowed on Streaming Tables.
Hello everyone,I have a workflow that starts by reading the CDF data for a change data feed.The syntax is exactly the following : (spark.readStream .format("delta") .option("readChangeFeed", "true") .option("startingVersion", 10) .table("my.str...
- 5151 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, this seems to be related to the issue I've been getting around the same time here: Change data feed from target tables of APPLY CHANG... - Databricks - 54436Would be great to get an explanation for the sudden change in behaviour.
- 0 kudos
- 1649 Views
- 0 replies
- 0 kudos
Access from DBT job to Azure DevOps repository using Service Principal
Hi,We are using Databricks bundles to deploy our DBT project. Everything is set up to deploy and run as a Service Principal.The DBT job is connected to an Azure DevOps repository. The problem is that we cannot find a way to properly authenticate the ...
- 1649 Views
- 0 replies
- 0 kudos
- 2000 Views
- 1 replies
- 0 kudos
Save default language of notebook into variable dynamically
For one of the requirements of project, I want to save default language of notebook into variable based on notebook path mentioned dynamically.For eg: if first notebook given by user in widget is having default language as Python then variable value...
- 2000 Views
- 1 replies
- 0 kudos
- 1944 Views
- 0 replies
- 0 kudos
Wrong FS: abfss://....., expected: dbfs:/ Error in DLT pipeline
Dear Databricks community members:SymptomReceived the error for a delta load, after a successful initial load with a Unity Catalog Volume as a data source.org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = xxx, runId...
- 1944 Views
- 0 replies
- 0 kudos
- 4951 Views
- 1 replies
- 0 kudos
Thousands of ETL pipelines with long execution times and small dataset sizes
Hi,I work for a small company, we're mostly focussing on small retail and e-commerce customers. We provides data analysis and automated data connections between their platforms. Most of our datasets are things like order data, google ads click data, ...
- 4951 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Thanks for the information, there is a lot to unpack and some assumptions that need to be made without fully understanding the details, so here are a few thoughts: If the cluster start times longer because of the libraries you're installing, can ...
- 0 kudos
- 3727 Views
- 1 replies
- 0 kudos
Query Delta table from .net
Hi Team,How can expose data stored in delta table through API like exposing sql data through .net api?
- 3727 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use the SQL Statement Execution API.At energinet.dk we have created this open-source .NET client, which we use internally in the company.
- 0 kudos
- 9011 Views
- 2 replies
- 3 kudos
Resolved! best way to store config files in a Unity workspace (Scala/typesafe)
We use typesafe (scala) to read configuration values from hocon files.When not using Unity, we read the configuration files from /dbfs/... works fine.However, with Unity, usage of dbfs is frowned upon.So I started looking into alternatives.And unfor...
- 9011 Views
- 2 replies
- 3 kudos
- 3 kudos
In the end we will continue to use dbfs. Maybe in the future when volumes are supported by scala io we can re-evaluate, but for now dbfs seems the way to go.
- 3 kudos
- 4172 Views
- 1 replies
- 6 kudos
I am getting an SSLError: HTTPSConnectionPool while making a call to https restapis from azure databricks I have tried to set a verify=false parameter in the call too.
response = requests.request("POST", url, verify=False, headers=headers, data=payload) SSLError: HTTPSConnectionPool(host='dcs.adobedc.net', port=443): Max retries exceeded with url: /collection/d99e6dfcffb0b5aeaec2cf76cd3bc2b9e9c414b0c74a528d13dd39...
- 4172 Views
- 1 replies
- 6 kudos
- 6 kudos
Any luck with this?
- 6 kudos
- 2958 Views
- 2 replies
- 0 kudos
Resolved! Regarding Exam got suspended
Hello Team,I had a disappointing experience while attempting my first DataBricks certification. Abruptly, the proctor asked me to show my desk, and after complying. Eventually, they suspended my exam, citing excessive eye movement and other practices...
- 2958 Views
- 2 replies
- 0 kudos
- 0 kudos
@mkrish28 I'm sorry to hear you had this experience. Thank you for logging at ticket with the support team. They have informed me they have rescheduled your exam. Good luck!
- 0 kudos
- 2538 Views
- 1 replies
- 1 kudos
DBR 14.1 - foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.
I am getting this error on DBR 14.1AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.This is the code: wstream = df.writeStream.foreac...
- 2538 Views
- 1 replies
- 1 kudos
- 2501 Views
- 1 replies
- 0 kudos
CANNOT_RENAME_ACROSS_SCHEMA message error
Hello...We enabled Unity Catalog and we are migrating schemas. When I ran the command sync schema catalog01.schema01 FROM hive_metastore.schema01 dry run I got the error CANNOT_RENAME_ACROSS_CATALOG, reviewing your documentation it only said CANNO...
- 2501 Views
- 1 replies
- 0 kudos
- 11376 Views
- 1 replies
- 0 kudos
List parameter in Python SQL connector 3.0.1
Hi,up to recently in version of the Python SQL connector 2.9.3, I was using a list as a parameter in the cursor.execute(operation, parameters) method without any trouble. It seems that it is not possible anymore in version 3.0.1 as the parsing of par...
- 11376 Views
- 1 replies
- 0 kudos
- 0 kudos
I should better read the documentation : https://github.com/databricks/databricks-sql-python/blob/v3.0.0/docs/parameters.md.
- 0 kudos
- 1192 Views
- 0 replies
- 0 kudos
Retrieving Logs with Job API Get-outputs service
Hello,I would like to retrieve the logs of some job that where launched using the Job Rest Api 2.0. I see in the doc that can be done with the service get-ouputs, however each time I check the service I just get the metadata part of the response but ...
- 1192 Views
- 0 replies
- 0 kudos
- 3866 Views
- 3 replies
- 0 kudos
How to convert entity_id to notebook name or job
Hi, Databricks developers!I use system.access.table_lineage refering to this page.It's difficult for us to recognize which notebook was indicated by the entity_id.How do I get the table to convert entity_ids to Job names or Notebook names?
- 3866 Views
- 3 replies
- 0 kudos
- 1338 Views
- 0 replies
- 0 kudos
Define VIEW in Databricks Asset Bundles?
Is it possible to define a Unity Catalog VIEW in a Databricks Asset Bundle, or specify in the bundle that a specific notebook gets run once per deployment?
- 1338 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
