- 12468 Views
- 5 replies
- 0 kudos
Questions about the design of bronze, silver, and gold for live streaming pipelines
I'm envisioning a live streaming pipeline.The bronze, or data ingestion, is being fetched using the directory listing mode of the autoloader.I'm not using File Notification Mode because I detect about 2-300 data changes per hour.I'm thinking about im...
- 12468 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey @rt-slowth Thank you for sharing the code snippets. The code structure appears to be on the right track, and its dynamic nature is promising. With a few minor adjustments, it should achieve the desired outcome. Also, find the attached code syntax...
- 0 kudos
- 9395 Views
- 2 replies
- 0 kudos
Resolved! How does Delta solve the large number of small file problems?
Delta creates more small files during merge and updates operations.
- 9395 Views
- 2 replies
- 0 kudos
- 0 kudos
Delta solves the large number of small file problems using the below operations available for a Delta table. Optimize writes helps to optimizes the write operation by adding an additional shuffle step and reducing the number of output files. By defau...
- 0 kudos
- 26618 Views
- 22 replies
- 28 kudos
How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?
Hi Team,We are wondering if there is a recommended way to install the chromium browser and chrome driver on Databricks Runtime 10.4 and above ?I have been through the site and have come across several links to this effect, but they all seem to be ins...
- 26618 Views
- 22 replies
- 28 kudos
- 28 kudos
Look into Playwrite instead of Selenium. I went through the same process y'all went through here (ended up writing a init script to install the drivers etc)This is all done for you in playwright. Refer to this post - I hope it helps!!https://communit...
- 28 kudos
- 1534 Views
- 2 replies
- 0 kudos
cluster metrics collection
Hello @Debayan please how can i collect metrics provided by clusters metrics for databricks runtime 13.1 or latest using shell bash script. Cordially, Aubert EMAKO
- 1534 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Cluster metrics is an UI tool and available in the UI only. For reference: https://docs.databricks.com/en/compute/cluster-metrics.html
- 0 kudos
- 8745 Views
- 2 replies
- 0 kudos
writing spark dataframe as CSV to a repo
Hi,I wrote a spark dataframe as csv to a repo (synced with github). But when I checked the folder, the file wasn't there. Here is my code:spark_df.write.format('csv').option('header','true').mode('overwrite').save('/Repos/abcd/mno/data') No error mes...
- 8745 Views
- 2 replies
- 0 kudos
- 0 kudos
the folder 'Repos' is not your repo, it's `dbfs:/Repos`, please checkdbutils.fs.ls('/Repos/abcd/mno/data')
- 0 kudos
- 1445 Views
- 0 replies
- 0 kudos
Missing 'DBAcademy DLT' as a Cluster Policy when creating Delta Live Tables pipeline
Hi There,I'm currently going through Module 4 of the Data Engineering Associate pathway, specifically lesson 4.1 - DLT UI Walkthrough. We are instructed to specify the Cluster Policy as 'DBAcademy DLT' when configuring the pipeline. However, this opt...
- 1445 Views
- 0 replies
- 0 kudos
- 1237 Views
- 0 replies
- 0 kudos
Cannot find UDF on subsequent job runs on same cluster.
Hello, I am trying to run jobs with a JAR task type using databricks on AWS on an all-purpose cluster. The issue I'm facing is that the job will complete the first run successfully but on any subsequent runs, it will fail. I have to restart my cluste...
- 1237 Views
- 0 replies
- 0 kudos
- 4193 Views
- 2 replies
- 0 kudos
Fatal error when writing a big pandas dF
Hello DB community,I was trying to write a pandas dataframe containing 100000 rows as excel. Moments in the execution I received a fatal error : "Python kernel is unresponsive."However, I am constrained from increasing the number of clusters or other...
- 4193 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @chari ,Thanks for bringing up your concerns, always happy to help We understand that you are facing the following error while you are writing a pandas dataframe containing 100000rows in excel. As per the Error >>> Fatal error: The Python kernel ...
- 0 kudos
- 9973 Views
- 2 replies
- 1 kudos
[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs
In my project I defined a UDF: @udf(returnType=IntegerType()) def ends_with_one(value, bit_position): if bit_position + len(value) < 0: return 0 else: return int(value[bit_position] == '1') spark.udf.register("ends_with_one"...
- 9973 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Yaacoub, Just a friendly follow-up. Have you had a chance to review my colleague's reply? Please inform us if it contributes to resolving your query.
- 1 kudos
- 8404 Views
- 4 replies
- 0 kudos
Intermittent error databricks job kept running
Hi i have the following error, but the job kept running, is that normal?{ "message": "The service at /api/2.0/jobs/runs/get?run_id=899157004942769 is temporarily unavailable. Please try again later. [TraceId: -]", "error_code": "TEMPORARILY_U...
- 8404 Views
- 4 replies
- 0 kudos
- 0 kudos
@Ayushi_Suthar also when ever it happens the job status does not change to "failed". But it keeps running. Is that normal?
- 0 kudos
- 9704 Views
- 1 replies
- 2 kudos
Change Data Capture with Databricks
How to leverage Change Data Capture (CDC) from your databases to DatabricksChange Data Capture allows you to ingest and process only changed records from database systems to dramatically reduce data processing costs and enable real-time use cases suc...
- 9704 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, @isaac_gritz can you provide any reference resource to achieve the AWS DynamoDB CDC to Delta Tables.Thank You,
- 2 kudos
- 8908 Views
- 2 replies
- 3 kudos
Resolved! What happens to table created with CTAS statement when data in source table has changed
Hey all - I am sure this has been documented / answered before but what happens to a table created with a CTAS statement when data in the source table has changed? Does the sink table reflect the changes? Or is the data stored when the table is defin...
- 8908 Views
- 2 replies
- 3 kudos
- 3 kudos
CREATE TABLE AS (CTAS) is a "one and done" kind of statement.The new table retains no memory on how it came to be.Therefore it will be oblivious to changes in the source.Views, as you say, stored queries, no data is persisted. And therefore the query...
- 3 kudos
- 11816 Views
- 3 replies
- 3 kudos
List of Nodes Types
Hi Guys,Where can I find the list f nodes types on the Databricks website ? (standard_ds3_v2, etc. etc. etc)Thank you
- 11816 Views
- 3 replies
- 3 kudos
- 12562 Views
- 4 replies
- 1 kudos
Resolved! Managed table overwrites existing location for delta but not for oth
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 12562 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi,this is how the delta format work. With overwrite you are not deleting the files in the folder or replacing them. Delta is creating a new file with the overwritten schema and data. This way you are also able to return to former versions of the del...
- 1 kudos
- 1587 Views
- 1 replies
- 0 kudos
Dashboard from Notebook - How to schedule
notebook is created with insight and have created dashboard (Not a SQL) from it.Need to schedule this. I have tried scheduling by workflow - it only takes you to the notebookeven the schedule from dashboard takes me to the notebook and not the dashbo...
- 1587 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Accn , Thanks for bringing up your concerns, always happy to help We understand your concern but right now there is only way to refresh a notebook dashboard is via scheduled jobs. To schedule a dashboard to refresh at a specified interval, click...
- 0 kudos
| User | Count |
|---|---|
| 1615 | |
| 779 | |
| 349 | |
| 287 | |
| 266 |
