- 7185 Views
- 3 replies
- 1 kudos
Resolved! can I use volume for external table location?
Hello,I have a parquet file test.parquet in the volume volume_ext_test. Tried to create an external table as below, it failed and says it "is not a valid URI".create table catalog_managed.schema_test.tbl_vol asselect * from parquet.`/Volumes/catalog_...
- 7185 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @PaulineX As per the documentation, you cannot use volume for storing table data. It's for loading, storing and accessing files. You cannot use volumes as a location for tables. Volumes are intended for path-based data access only. Use tables for ...
- 1 kudos
- 2651 Views
- 0 replies
- 0 kudos
Setting Column Only on Insert with DLT's apply_changes CDC Merge
Hi Databricks Community,We've encountered an issue with setting a column only on insert when using DLT's apply_changes CDC merge functionality. It's important to note that this capability is available when using the regular Delta merge operation, spe...
- 2651 Views
- 0 replies
- 0 kudos
- 2185 Views
- 1 replies
- 0 kudos
Migrating Job Orchestration to Shared Compute and avoiding(?) refactoring
In an effort to migrate our data objects to the Unity Catalog, we must migrate our Job Orchestration to leverage Shared Compute to interact with the 3-namespace hierarchy.We have some functions and references to code that are outside of the features ...
- 2185 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod Okay, that makes sense, thank you.What about the approach to identifying these unsupported methods? Is there any documentation of what is unsupported between Unrestricted and Shared?
- 0 kudos
- 2198 Views
- 0 replies
- 0 kudos
duplicate files in bronze delta table
Hello All,I am facing this issue from long time but so far there is no solution. I have delta table. My bronze layer is picking up the old files (mostly 8 days old file) randomly. My source of files is azure blob storage.Those files and not being upd...
- 2198 Views
- 0 replies
- 0 kudos
- 2951 Views
- 2 replies
- 1 kudos
Resolved! Cost of running a job on databricks
Hi All, I need assistance with the cost of running the job on Databricks where I have 20-30 TB (one-time job) and daily data would be around 2GB. The level of transformation would be medium. Source and destination is AWS s3.Looking for your quick re...
- 2951 Views
- 2 replies
- 1 kudos
- 1 kudos
Is there a way to predict the cost before building the solution? I mean we wanted to see our option on different platforms.
- 1 kudos
- 1505 Views
- 1 replies
- 0 kudos
Is it possible to migrate SQL Objects from one workspace to another?
We have SQL Queries and dashboards in workspace dev_01. New workspace dev_02 is created and unity catalog is enabled.I was able to migrate jobs, clusters, DLTs, SQL warehouses, users using APIs. But, while migrating queries using APIs, I can't get th...
- 1505 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm doing something similar, but I haven't run into this parent directory issue. [Actually to be clear I ran into an issue around missing user directories, but I believe that was different than what you describe]. Before migrating the queries, I'm re...
- 0 kudos
- 9404 Views
- 0 replies
- 0 kudos
AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWithinWatermark'
Hello,I have some trouble deduplicating rows on the "id" column, with the method "dropDuplicatesWithinWatermark" in a pipeline. When I run this pipeline, I get the error message:"AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWith...
- 9404 Views
- 0 replies
- 0 kudos
- 3659 Views
- 3 replies
- 1 kudos
CDC Delta table select using startingVersion on Shared cluster running DBR 14.3 does not work
Hello everyone,We have switched from DBR 13.3 to 14.3 on our Shared development cluster and I am no longer able to run following read from a delta table with CDC enabled:data = ( spark.read.format("delta") .option("readChangeFeed", "true") .op...
- 3659 Views
- 3 replies
- 1 kudos
- 6965 Views
- 1 replies
- 1 kudos
Connection closed by foreign host
Hi, I am setting up Unity Catalog in my environment, I have a workspace with 2 clusters, the older one (10.4 runtime and non-isolated shared mode) without Unity Catalog configuration and the new one with Unity Catalog enabled (13.3 runtime and shared...
- 6965 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @m-lopez , hope you are doing well! Were you able to connect to Kafka before? Also, do you get the same issue if you run the telnet command against port 9092? As per the error message "Trying 10.40.12.243...Connected to 10.40.12.243. Escape char...
- 1 kudos
- 7222 Views
- 5 replies
- 1 kudos
dbfs rest api - Access denied on dbfs rest api call but access works in dataricks notebook
Hello, We are trying to interact with dbfs to upload files and list files in dbfs directory ( we are tring to upload files in volumes ). Although we have necessary permissions on databricks, we are still getting permission denied when we are making r...
- 7222 Views
- 5 replies
- 1 kudos
- 1 kudos
I am experiencing the same issue as well. Any ideas about how we can upload to volumes using the api? Tried both the /api/2.0/dbfs/put and the streaming apis (/api/2.0/dbfs/create,/api/2.0/dbfs/add-block,/api/2.0/dbfs/close) but no luck so getting it...
- 1 kudos
- 11820 Views
- 1 replies
- 0 kudos
Empty xml tag
<ItemMaintenance> <Batch> <BathInfo>info</BathInfo> <Item attr1="tekst" attr2="Tekst2"> <ItemId type="Type" id="id"/> <Dates> <Start>2023-11-09</Start> <End>2024-01-02</End> </Dates> <MoreData> More data </MoreData> <...
- 11820 Views
- 1 replies
- 0 kudos
- 10042 Views
- 1 replies
- 0 kudos
Databricks Vs ADLS for reporting.
Hi everyone, I'm a business analyst currently facing a decision on how best to develop a report. I need to choose between using Databricks or Azure Data Lake Storage (ADLS) as the data source on the transformed data in csv or excel file format from d...
- 10042 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @Learnit I'd be glad to help, thanks for posting your concern. To offer the most effective advice, I might need some additional context about your specific situation as looks like your use case is to create reports (dahboards) using local CSV/exc...
- 0 kudos
- 949 Views
- 1 replies
- 0 kudos
Resonable long running time
Hi there, I have a block of code which can be executed around a month ago within 20 mins. But I came back recently and try to execute it again. It takes over 50 minutes still cannot complete (finally got kick out). Any advise and hints is appreciated...
- 949 Views
- 1 replies
- 0 kudos
- 0 kudos
Haha... Should be "unreasonable long"
- 0 kudos
- 3792 Views
- 3 replies
- 0 kudos
Where to move cluster-init scripts after latest message (Storing initialization scripts on DBFS is being deprecated. We recommend using a different storage location)?
Recently Databricks has started showing "Storing initialization scripts on DBFS is being deprecated. We recommend using a different storage location" . Is there an alternative of still keep using DBFS or do we need to move to ABSFS ? I could not find...
- 3792 Views
- 3 replies
- 0 kudos
- 0 kudos
It is suggested that we can use Workspace but databricks CLI not supporting importing shell scripts, but we can import shell script using web page?
- 0 kudos
- 3431 Views
- 2 replies
- 1 kudos
Workflows 7 second delay between tasks
When you have a job in Workflows with multiple tasks running after one another, there seems to be a consistent 7 seconds delay between execution of the tasks. Or, more precisely, every task has an approximate 7 second overhead before the code actuall...
- 3431 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @bergmaal , I am experiencing the same issue.My Databricks consultant suggested opening a support ticket as this should not be normal behavior.Did you solve this issue yet?We observed these delays do not seem to occur in workflows that use noteboo...
- 1 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 523 | |
| 349 | |
| 287 |
