- 10845 Views
- 3 replies
- 0 kudos
Disable Machine Learning and Job Creation
We are working on creating a new databricks workspace for external entities. We have disabled Cluster and Warehouse creation permission but the external users are still able to create Jobs and job clusters. Is there a way to revoke Job creation permi...
- 10845 Views
- 3 replies
- 0 kudos
- 0 kudos
It permits cluster creation during Workflow/Job/DLT pipeline creation. However, when attempting to start any of these, it fails with a 'Not authorized to create compute' error. Please try it and inform me of the outcome
- 0 kudos
- 1599 Views
- 1 replies
- 0 kudos
duplicate files in delta table
I am facing this issue from long time but so far there is no solution. I have delta table. My bronze layer is picking up the old files (mostly 8 days old file) randomly. My source of files is azure blob storage.
- 1599 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @jaimeperry12345 I will need more information to direct you in the right direction: Confirm the behavior: Double-check that your Delta table is indeed reading 8-day-old files randomly. Provide any logs or error messages you have regarding this.Ex...
- 0 kudos
- 8613 Views
- 3 replies
- 1 kudos
Resolved! can I use volume for external table location?
Hello,I have a parquet file test.parquet in the volume volume_ext_test. Tried to create an external table as below, it failed and says it "is not a valid URI".create table catalog_managed.schema_test.tbl_vol asselect * from parquet.`/Volumes/catalog_...
- 8613 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @PaulineX As per the documentation, you cannot use volume for storing table data. It's for loading, storing and accessing files. You cannot use volumes as a location for tables. Volumes are intended for path-based data access only. Use tables for ...
- 1 kudos
- 3121 Views
- 0 replies
- 0 kudos
Setting Column Only on Insert with DLT's apply_changes CDC Merge
Hi Databricks Community,We've encountered an issue with setting a column only on insert when using DLT's apply_changes CDC merge functionality. It's important to note that this capability is available when using the regular Delta merge operation, spe...
- 3121 Views
- 0 replies
- 0 kudos
- 2619 Views
- 1 replies
- 0 kudos
Migrating Job Orchestration to Shared Compute and avoiding(?) refactoring
In an effort to migrate our data objects to the Unity Catalog, we must migrate our Job Orchestration to leverage Shared Compute to interact with the 3-namespace hierarchy.We have some functions and references to code that are outside of the features ...
- 2619 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod Okay, that makes sense, thank you.What about the approach to identifying these unsupported methods? Is there any documentation of what is unsupported between Unrestricted and Shared?
- 0 kudos
- 2507 Views
- 0 replies
- 0 kudos
duplicate files in bronze delta table
Hello All,I am facing this issue from long time but so far there is no solution. I have delta table. My bronze layer is picking up the old files (mostly 8 days old file) randomly. My source of files is azure blob storage.Those files and not being upd...
- 2507 Views
- 0 replies
- 0 kudos
- 3464 Views
- 2 replies
- 1 kudos
Resolved! Cost of running a job on databricks
Hi All, I need assistance with the cost of running the job on Databricks where I have 20-30 TB (one-time job) and daily data would be around 2GB. The level of transformation would be medium. Source and destination is AWS s3.Looking for your quick re...
- 3464 Views
- 2 replies
- 1 kudos
- 1 kudos
Is there a way to predict the cost before building the solution? I mean we wanted to see our option on different platforms.
- 1 kudos
- 1930 Views
- 1 replies
- 0 kudos
Is it possible to migrate SQL Objects from one workspace to another?
We have SQL Queries and dashboards in workspace dev_01. New workspace dev_02 is created and unity catalog is enabled.I was able to migrate jobs, clusters, DLTs, SQL warehouses, users using APIs. But, while migrating queries using APIs, I can't get th...
- 1930 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm doing something similar, but I haven't run into this parent directory issue. [Actually to be clear I ran into an issue around missing user directories, but I believe that was different than what you describe]. Before migrating the queries, I'm re...
- 0 kudos
- 9784 Views
- 0 replies
- 0 kudos
AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWithinWatermark'
Hello,I have some trouble deduplicating rows on the "id" column, with the method "dropDuplicatesWithinWatermark" in a pipeline. When I run this pipeline, I get the error message:"AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWith...
- 9784 Views
- 0 replies
- 0 kudos
- 4316 Views
- 3 replies
- 1 kudos
CDC Delta table select using startingVersion on Shared cluster running DBR 14.3 does not work
Hello everyone,We have switched from DBR 13.3 to 14.3 on our Shared development cluster and I am no longer able to run following read from a delta table with CDC enabled:data = ( spark.read.format("delta") .option("readChangeFeed", "true") .op...
- 4316 Views
- 3 replies
- 1 kudos
- 8365 Views
- 1 replies
- 1 kudos
Connection closed by foreign host
Hi, I am setting up Unity Catalog in my environment, I have a workspace with 2 clusters, the older one (10.4 runtime and non-isolated shared mode) without Unity Catalog configuration and the new one with Unity Catalog enabled (13.3 runtime and shared...
- 8365 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @m-lopez , hope you are doing well! Were you able to connect to Kafka before? Also, do you get the same issue if you run the telnet command against port 9092? As per the error message "Trying 10.40.12.243...Connected to 10.40.12.243. Escape char...
- 1 kudos
- 8406 Views
- 5 replies
- 1 kudos
dbfs rest api - Access denied on dbfs rest api call but access works in dataricks notebook
Hello, We are trying to interact with dbfs to upload files and list files in dbfs directory ( we are tring to upload files in volumes ). Although we have necessary permissions on databricks, we are still getting permission denied when we are making r...
- 8406 Views
- 5 replies
- 1 kudos
- 1 kudos
I am experiencing the same issue as well. Any ideas about how we can upload to volumes using the api? Tried both the /api/2.0/dbfs/put and the streaming apis (/api/2.0/dbfs/create,/api/2.0/dbfs/add-block,/api/2.0/dbfs/close) but no luck so getting it...
- 1 kudos
- 12243 Views
- 1 replies
- 0 kudos
Empty xml tag
<ItemMaintenance> <Batch> <BathInfo>info</BathInfo> <Item attr1="tekst" attr2="Tekst2"> <ItemId type="Type" id="id"/> <Dates> <Start>2023-11-09</Start> <End>2024-01-02</End> </Dates> <MoreData> More data </MoreData> <...
- 12243 Views
- 1 replies
- 0 kudos
- 13350 Views
- 1 replies
- 0 kudos
Databricks Vs ADLS for reporting.
Hi everyone, I'm a business analyst currently facing a decision on how best to develop a report. I need to choose between using Databricks or Azure Data Lake Storage (ADLS) as the data source on the transformed data in csv or excel file format from d...
- 13350 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @Learnit I'd be glad to help, thanks for posting your concern. To offer the most effective advice, I might need some additional context about your specific situation as looks like your use case is to create reports (dahboards) using local CSV/exc...
- 0 kudos
- 1237 Views
- 1 replies
- 0 kudos
Resonable long running time
Hi there, I have a block of code which can be executed around a month ago within 20 mins. But I came back recently and try to execute it again. It takes over 50 minutes still cannot complete (finally got kick out). Any advise and hints is appreciated...
- 1237 Views
- 1 replies
- 0 kudos
- 0 kudos
Haha... Should be "unreasonable long"
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
