Hello All,I am facing this issue from long time but so far there is no solution. I have delta table. My bronze layer is picking up the old files (mostly 8 days old file) randomly. My source of files is azure blob storage.Those files and not being upd...
Hi All, I need assistance with the cost of running the job on Databricks where I have 20-30 TB (one-time job) and daily data would be around 2GB. The level of transformation would be medium. Source and destination is AWS s3.Looking for your quick re...
Is there a way to predict the cost before building the solution? I mean we wanted to see our option on different platforms.
We have SQL Queries and dashboards in workspace dev_01. New workspace dev_02 is created and unity catalog is enabled.I was able to migrate jobs, clusters, DLTs, SQL warehouses, users using APIs. But, while migrating queries using APIs, I can't get th...
I'm doing something similar, but I haven't run into this parent directory issue. [Actually to be clear I ran into an issue around missing user directories, but I believe that was different than what you describe]. Before migrating the queries, I'm re...
Hello,I have some trouble deduplicating rows on the "id" column, with the method "dropDuplicatesWithinWatermark" in a pipeline. When I run this pipeline, I get the error message:"AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWith...
Hello everyone,We have switched from DBR 13.3 to 14.3 on our Shared development cluster and I am no longer able to run following read from a delta table with CDC enabled:data = ( spark.read.format("delta") .option("readChangeFeed", "true") .op...
Hi, I am setting up Unity Catalog in my environment, I have a workspace with 2 clusters, the older one (10.4 runtime and non-isolated shared mode) without Unity Catalog configuration and the new one with Unity Catalog enabled (13.3 runtime and shared...
Hi @m-lopez , hope you are doing well! Were you able to connect to Kafka before? Also, do you get the same issue if you run the telnet command against port 9092? As per the error message "Trying 10.40.12.243...Connected to 10.40.12.243. Escape char...
Hello, We are trying to interact with dbfs to upload files and list files in dbfs directory ( we are tring to upload files in volumes ). Although we have necessary permissions on databricks, we are still getting permission denied when we are making r...
I am experiencing the same issue as well. Any ideas about how we can upload to volumes using the api? Tried both the /api/2.0/dbfs/put and the streaming apis (/api/2.0/dbfs/create,/api/2.0/dbfs/add-block,/api/2.0/dbfs/close) but no luck so getting it...
<ItemMaintenance> <Batch> <BathInfo>info</BathInfo> <Item attr1="tekst" attr2="Tekst2"> <ItemId type="Type" id="id"/> <Dates> <Start>2023-11-09</Start> <End>2024-01-02</End> </Dates> <MoreData> More data </MoreData> <...
Hi everyone, I'm a business analyst currently facing a decision on how best to develop a report. I need to choose between using Databricks or Azure Data Lake Storage (ADLS) as the data source on the transformed data in csv or excel file format from d...
Hey @Learnit I'd be glad to help, thanks for posting your concern. To offer the most effective advice, I might need some additional context about your specific situation as looks like your use case is to create reports (dahboards) using local CSV/exc...
Hi there, I have a block of code which can be executed around a month ago within 20 mins. But I came back recently and try to execute it again. It takes over 50 minutes still cannot complete (finally got kick out). Any advise and hints is appreciated...
Haha... Should be "unreasonable long"
Recently Databricks has started showing "Storing initialization scripts on DBFS is being deprecated. We recommend using a different storage location" . Is there an alternative of still keep using DBFS or do we need to move to ABSFS ? I could not find...
It is suggested that we can use Workspace but databricks CLI not supporting importing shell scripts, but we can import shell script using web page?
When you have a job in Workflows with multiple tasks running after one another, there seems to be a consistent 7 seconds delay between execution of the tasks. Or, more precisely, every task has an approximate 7 second overhead before the code actuall...
Hi @bergmaal , I am experiencing the same issue.My Databricks consultant suggested opening a support ticket as this should not be normal behavior.Did you solve this issue yet?We observed these delays do not seem to occur in workflows that use noteboo...
Trying to do a url_decode on a column, which works great in development, but running via DLT fails when trying multiple ways.1. pyspark.sql.functions.url_decode - This is new as of 3.5.0, but isn't supported using whatever version running a DLT pipel...
Thanks @Retired_mod, for reference if anybody finds this, the DLT release docs are here: https://docs.databricks.com/en/release-notes/delta-live-tables/index.htmlThis shows which versions are running for CURRENT and PREVIEW channels. In this case, wa...
Hi Databricks Community,If only possible I would like to use Shared Jobs Cluster on external orchestrator like Azure Data Factory (ADF) or Synapse Workspace.The main reasons for using Shared Job cluster are:reduction of start-up time (<1min vs 5 min ...
Hi Sai Kumar,Many thanks for your response.Unfortunately using analytical clusters is not really an option for for me due to cost differences between job clusters and analytical clusters.Job cluster also offer assurance that the latest deployed versi...
Can anyone give me Steps to achieve this.
Hi, @prasad95 Thank you for sharing your concern here. In addition to the @Retired_mod comments you can follow below To capture Change Data (CDC) from DynamoDB Streams and write it into a Delta table in Databricks: 1. Connect to DynamoDB Streams and...
| User | Count |
|---|---|
| 1644 | |
| 791 | |
| 554 | |
| 349 | |
| 287 |
