- 6209 Views
- 3 replies
- 0 kudos
Control query caching using SQL statement execution API
I want to execute this statement using databricks SQL Statement Execution API. curl -X POST -H 'Authorization: Bearer <access-token>' -H 'Content-Type: application/json' -d '{"warehouse_id": "<warehouse_id>", "statement": "set us...
- 6209 Views
- 3 replies
- 0 kudos
- 785 Views
- 3 replies
- 3 kudos
Resolved! Delta live table: Retrieve CDF columns
I have want to use the apply_changes feature from a bronze table to a silver table.The bronze table have no "natural" sequence_by column. Therefore, I want to use the CDF column "_commit_timestamp" as the sequence_by.How do I retrieve the columns in ...
- 785 Views
- 3 replies
- 3 kudos
- 3 kudos
Thank you @raphaelblg!I chose to write an article on the subject after this discussion: https://www.linkedin.com/pulse/databricks-delta-live-tables-merging-lau-johansson-cdtce/?trackingId=L872gj0yQouXgJudM75gdw%3D%3D
- 3 kudos
- 1231 Views
- 2 replies
- 0 kudos
workflow permissions errors
I have a notebook that outputs an Excel file. Through trial and error, and after consulting with various forums I discovered the .xlsx file needed to be written to a temp file and then copied to the volume in Unity Catalog.When I run the notebook by...
- 1231 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, yes of course you need to write the excel file in the tmp folder, but then you can move it to whatever you want without problem. In my current project we implemented this method to create the file in the tmp folder, and then move it to one spe...
- 0 kudos
- 771 Views
- 5 replies
- 0 kudos
Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all data
Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all data. I want to load all the data which are not processed . I don't want to relaod all the data.
- 771 Views
- 5 replies
- 0 kudos
- 0 kudos
Do checkpoint has some benchmark capacity after that it stops writing data?
- 0 kudos
- 1798 Views
- 8 replies
- 5 kudos
Can use graphframes DBR 14.3
I get the following error when trying to run GraphFrame on DBR 14.3. Anyone has an idea of how I can solve this? """import pyspark.sql.functions as Ffrom graphframes import GraphFrame vertices = spark.createDataFrame([ ("a", "Alice", 34), ("b"...
- 1798 Views
- 8 replies
- 5 kudos
- 5 kudos
Serverless compute has limitations, like installing libraries. So at the moment that won't be possible.Oldschool clusters have way more configuration possibilities, so hopefully you can fix the issue you experience on deploying clusters.
- 5 kudos
- 690 Views
- 3 replies
- 2 kudos
Run pyspark queries from outside databricks
I have written a Notebook which would execute pyspark query. I then execute it remotely from outside databricks environment using /api/2.1/jobs/run-now, which would then run the notebook. I also want to retrieve the results from this job execution. H...
- 690 Views
- 3 replies
- 2 kudos
- 2 kudos
Thanks for responding. I did go through this link. It talks about executing on SQL warehouse though. Is there a way we can execute queries on Databricks clusters instead?Databricks has this connector for SQL https://docs.databricks.com/en/dev-tools/p...
- 2 kudos
- 7197 Views
- 4 replies
- 0 kudos
[TASK_WRITE_FAILED] Task failed while writing rows to abfss
I am trying to insert into an already created delta table in Unity Catalog. I am getting the error:[TASK_WRITE_FAILED] Task failed while writing rows to abfss://xxxx@xxxxxxxxxxxxxxxx.dfs.core.windows.net/__unitystorage/catalogs/xxxxxxxx-c6c8-45d8-ac3...
- 7197 Views
- 4 replies
- 0 kudos
- 0 kudos
Any chance this issue got resolved?I am also seeing the same error when I am trying to incrementally read the system tables using the read stream method and writing it using the writestream method. This generally comes for the audit table but other t...
- 0 kudos
- 3116 Views
- 1 replies
- 0 kudos
DLT: Waiting for resources took a long time
Hi Team,I have a DLT pipeline running in Production for quite some time now. When I check the pipeline, a couple of jobs took longer than expected. Usually, 1 job only took 10-15 minutes to complete with 2 to 3 mins to provision a resource. Then I ha...
- 3116 Views
- 1 replies
- 0 kudos
- 0 kudos
I am currently trying projects from dbdemos [Full Delta Live Tables Pipeline - Loan].I keep running into this error. how do I resolve this?
- 0 kudos
- 593 Views
- 2 replies
- 1 kudos
Resolved! Testing PySpark - Document links broken
The top paragraph of this Testing PySpark page from Apache Spark team states the following - where it points to some links with title 'see here'. But no link is provided to click on. Can someone please provide those links the document is referring to...
- 593 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Saf4Databricks ,Sure, here they are:- To view the docs for PySpark test utils, see here. spark.apache.org- To see the code for PySpark built-in test utils, check out the Spark repositorypyspark.testing.utils — PySpark 3.5.2 documentation (apache....
- 1 kudos
- 3307 Views
- 5 replies
- 2 kudos
Job cluster support in jobs/runs/submit API
We are using jobs/runs/submit API of databricks to create and trigger a one-time run with new_cluster and existing_cluster configuration. We would like to check if there is provision to pass "job_clusters" in this API to reuse the same cluster across...
- 3307 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi, Any update on the above mentioned issue? Unable to submit a one time new job run (api/2.0 or 21/jobs/runs/submit) with shared job cluster or one new cluster has to be used for all TASKs in the job
- 2 kudos
- 4419 Views
- 7 replies
- 1 kudos
Resolved! Regarding Exam got Suspended at middle without any reason.
Hi Team,My Databricks Certified Data Engineer Associate (Version 3) exam got suspended on 25th August and it is in progress state.I was there continuously in front of the camera and suddenly the alert appeared, and support person asked me to show the...
- 4419 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @UserAnalyst, I'm sorry to hear your exam was suspended. Thank you for filing a ticket with our support team. Please allow the support team 24-48 hours for a resolution. In the meantime, you can review the following documentation:Room requirements...
- 1 kudos
- 434 Views
- 1 replies
- 1 kudos
Resolved! schema is not enforced when using autoloader
Hi everyone,I am currently trying to enforce the following schema: StructType([ StructField("site", StringType(), True), StructField("meter", StringType(), True), StructField("device_time", StringType(), True), StructField("data", St...
- 434 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @sakuraDev ,I'm afraid your assumption is wrong. Here you define data field as struct type and the result is as expected. So once you have this column as struct type, you can refer to nested object using dot notation. So if you would like to get e...
- 1 kudos
- 677 Views
- 2 replies
- 0 kudos
Info on Databricks AWS High Availability during zone selection
Hi Team,During the zone selection in the Databricks environment, there is an option for High Availability (HA), which selects instances from other zones to ensure prolonged uptimes.My question is: Does the HA option only select instances from other a...
- 677 Views
- 2 replies
- 0 kudos
- 596 Views
- 4 replies
- 0 kudos
Data Ingestion from GCP
Hi,I'm ingesting data from GCP to Databricks, and I think I've noticed a bug in that any datatables that have a numerical starting character are not ingested at all.Has anyone else experienced this?Please let me know if there is a way around this apa...
- 596 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Slash, thanks for getting back to meSo for instance - I have data tables such as "20240901_demographics_data_v1" which I'm trying to move from BQOther data tables that don't include a date (or other numerical characters) at the front are being ing...
- 0 kudos
- 16576 Views
- 13 replies
- 3 kudos
Resolved! Databricks SQL - Unable to Escape Dollar Sign ($) in Column Name
It seems that due to how Databricks processes SQL cells, it's impossible to escape the $ when it comes to a column name.I would expect the following to work:%sql SELECT 'hi' `$id`The backticks ought to escape everything. And indeed that's exactly wha...
- 16576 Views
- 13 replies
- 3 kudos
