- 256 Views
- 3 replies
- 0 kudos
Resolved! AUTO CDC API and sequence column
The docs for AUTO CDC API stateYou must specify a column in the source data on which to sequence records, which Lakeflow Declarative Pipelines interprets as a monotonically increasing representation of the proper ordering of the source data.Can this ...
- 256 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks Szymon, I'm familiar with the Postgre SQL implementation and was hoping Databricks would behave the same.
- 0 kudos
- 1836 Views
- 1 replies
- 2 kudos
DLT schema evolution/changes in the logs
Hi all,I want to figure out how to find when the schema evolution/changes are happening in the objects in DLT pipelines through the DLT logs.Could you please share some sample DLT logs which explains about the schema changes?Thank you for your help.
- 1836 Views
- 1 replies
- 2 kudos
- 2 kudos
To find when schema evolution or changes are happening in objects within DLT (Delta Live Table) pipelines, you need to monitor certain entries within the DLT logs or Delta transaction logs that signal modifications to the underlying schema of a table...
- 2 kudos
- 2484 Views
- 3 replies
- 0 kudos
DLT Flow Failed Due to Missing Flow Checkpoints Directory When Using Unity Catalog
I’m encountering an issue while running a Delta Live Tables (DLT) pipeline that is managed using Unity Catalog on Databricks. The pipeline has failed and is not restarting, showing the following error:java.lang.IllegalArgumentException: flow checkpoi...
- 2484 Views
- 3 replies
- 0 kudos
- 0 kudos
The best practices for setting up checkpointing in Delta Live Tables (DLT) pipelines when using Unity Catalog are largely centered on leveraging Databricks' managed services, adhering to Unity Catalog's table management conventions, and minimizing th...
- 0 kudos
- 299 Views
- 4 replies
- 1 kudos
Not able to refresh powerbi dashboar form databricks jobs
I am trying to refresh Power BI Dashboard using Databricks jobs and constantly getting this error, but I am providing optional parameters which includes catalog and database. Also, things to note that I am able to do refresh on Power BI UI using both...
- 299 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @sumitkumar_284 ,Can you provide us more details? Are you using Unity Catalog? Which authentication mechanism you have? In which version of Power BI Desktop you've developed your semantic model/dashboard? Do you meet all below requirements?Publish...
- 1 kudos
- 29 Views
- 1 replies
- 0 kudos
Unable to create workspace
Hi Team,we have challenge creating workspace in data bricks account created on top of aws.below are the details:Databricks account name : saichaitanya.vaddadhi@healthfirsttech.com's LakehouseAWS Account id : 720016114009databricks id: 1ee8765f-b472-4...
- 29 Views
- 1 replies
- 0 kudos
- 0 kudos
@hf-databricks there's a quickstart guide for creating a workspace with AWS: https://docs.databricks.com/aws/en/admin/workspace/quick-start There's a list of requirements:There's more options for creating workspaces. Above, I just listed the recommen...
- 0 kudos
- 49 Views
- 8 replies
- 0 kudos
Rasterio on shared/standard cluster has no access to proj.db
We try to use rasterio on a Databricks shared/standard cluster with DBR 17.1. Rasterio is directly installed on the cluster as library. Code:import rasterio rasterio.show_versions()Output: rasterio info:rasterio: 1.4.3GDAL: 3.9.3PROJ: 9.4.1GEOS: 3.11...
- 49 Views
- 8 replies
- 0 kudos
- 0 kudos
Hi @der Can you try adding this in your test script.import osos.environ["PROJ_LIB"]="/databricks/native/proj-data"Hope users have access to this path /databricks/native/proj-data
- 0 kudos
- 227 Views
- 3 replies
- 0 kudos
DLT Pipeline Stopped working
Hello, Suddenly our DLT pipelines we're getting failures saying thatLookupError: Traceback (most recent call last): result_df = result_df.withColumn("input_file_path", col("_metadata.file_path")).withColumn( ...
- 227 Views
- 3 replies
- 0 kudos
- 0 kudos
Greetings @maninegi05 , I did some digging internally and I believe some recent changes to the DLT image may be to blame. We are aware of regression issue and are actively working to address them. TL/DR Why you might see “LookupError: ContextVar 'par...
- 0 kudos
- 99 Views
- 7 replies
- 0 kudos
Encountering an error while setting up a single-node cluster on top of aws
Hi Team,I'm trying to create a single-node cluster in Databricks on AWS, but I'm encountering an error. Could you please assist me with this?{ "reason": { "code": "INVALID_ARGUMENT", "type": "CLIENT_ERROR", "parameters": { "databr...
- 99 Views
- 7 replies
- 0 kudos
- 0 kudos
@devpavan Are you using API or terraform to create. Can you please share the json config that you are passing?
- 0 kudos
- 19379 Views
- 22 replies
- 4 kudos
vocareum lab access
Hi I am doing a data engineering course in databricks(Partner labs) and would like to have access to vocareum workspace to practice using the demo sessions.can you please help me to get the access to this workspace?regards,Aravind
- 19379 Views
- 22 replies
- 4 kudos
- 4 kudos
You can log into databricks, search for "Canada Sales" in the Marketplace and find "Simulated Canada Sales and Opportunities Data". Get free instant access, wait a few seconds for the warehouse to be built for you et voila: the tables for building th...
- 4 kudos
- 2320 Views
- 1 replies
- 1 kudos
Cant view DAB deployed pipelines in Databricks UI
I am using the databricks asset pipeline to version control the jobs and pipelines in my workspace. I recently pulled these pipelines from the workspace using the `databricks bundle generate pipeline` command and deployed them back using `databricks ...
- 2320 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @lezwon Thanks for the details and screenshots—this looks like a permissions/ownership issue with your newly deployed Delta Live Tables pipelines. What’s going on Pipelines run under the pipeline owner’s identity (Databricks recommends a ser...
- 1 kudos
- 29 Views
- 3 replies
- 0 kudos
databricks serverless cluster and poetry private repository
Currently we are evaluating the databricks serverless. It support public repository in poetry as dependency path but it is not supporting private repository as we are not sure whether put the credentials details regarding privare repository.
- 29 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @mahfooz_iiitian ,Databricks supports private repositories only for Notebook-scoped libraries.In serverless you can use do it using pip install (of course store you token in a safe palce):Notebook-scoped Python libraries - Azure Databricks | Micro...
- 0 kudos
- 13 Views
- 0 replies
- 0 kudos
transformWithStateInPandas. Invalid pickle opcode when updating ValueState with large (float) array
I am getting an error when the entity I need to store in a ValueState is a large array (over 15k-20k items). No error (and works correctly) if I trim the array to under 10k samples. The same error is raised when using it as a value for MapState or as...
- 13 Views
- 0 replies
- 0 kudos
- 1765 Views
- 2 replies
- 1 kudos
ModuleNotFound error when using transformWithStateInPandas via a class defined outside the notebook
As per Databricks documentation when I define the class that extends `StatefulProcessor` in a Notebook everything works ok however, execution fails with ModuleNotFound error as soon as the class definition is moved to a file (module) of it's own in a...
- 1765 Views
- 2 replies
- 1 kudos
- 1 kudos
This is no longer an issue; it must be some patch version of DBX Runtime 16.4 fixed it and it works now without doing any changes to original code.Thanks.
- 1 kudos
- 50 Views
- 5 replies
- 2 kudos
Resolved! when automatic liquid clustering is enabled, how to know which columns are used for clustering?
Let's say a table is configured to have automatic liquid clustering:ALTER TABLE table1 CLUSTER BY AUTO; How to know which columns were chosen by Databricks?
- 50 Views
- 5 replies
- 2 kudos
- 2 kudos
From the documentation, it seems that in Python, there is such an option, only when creating or replacing a table.# To set clustering columns and auto, which serves as a way to give a hint # for the initial selection. df.writeTo(...).using("delta") ...
- 2 kudos
- 62 Views
- 1 replies
- 0 kudos
Migrating Talend ETL Jobs to Databricks – Best Practices & Challenges
Hi All,I’m currently working on a Proof of Concept (POC) to migrate existing Talend ETL jobs to Databricks. The goal is to leverage Databricks for data processing and orchestration while moving away from Talend.I’d appreciate insights on the followin...
- 62 Views
- 1 replies
- 0 kudos
- 0 kudos
Migrating from Talend’s jobs to Databricks requires rebuilding ETL logic using Spark. There is no native one-click converter provided by Databricks, so the typical approach is to audit and refactor each Talend job into code (PySpark or Spark SQL) in ...
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 467 | |
| 349 | |
| 287 |
