- 1307 Views
- 2 replies
- 0 kudos
SAS token issue for long running micro-batches
Hi everyone,I'm having an issue with some of our Databricks workloads. We're processing these workloads using the forEachBatch stream processing method. Whenever we are performing a full reload on some of our datasources, we get the following error. ...
- 1307 Views
- 2 replies
- 0 kudos
- 0 kudos
@deecee Can you please confirm there are no external locations or volumes which can lead to this overlap of locations? what you actually have in "some_catalog.some_schema.some_table" and the "abfss://some-container@somestorageaccount.dfs.core.window...
- 0 kudos
- 1109 Views
- 2 replies
- 0 kudos
FileReadException error when creating materialized view reading two schemas
Hi all. I'm getting an error `com.databricks.sql.io.FileReadException` when attempting to create a materialized view which reads tables from two different schemas in the same catalog. Is this just a limitation in databricks or do I potentially have s...
- 1109 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @kalebkemp ,The error you're encountering (com.databricks.sql.io.FileReadException) when creating a materialized view that reads from two different schemas in the same catalog might not necessarily be a Databricks limitation. It is more likely rel...
- 0 kudos
- 5192 Views
- 6 replies
- 6 kudos
Azure Synapse vs Databricks
Hi there,I would like to know the difference between Azure Databricks and Azure Synapse, which use case is Databricks appropriate and which use case is Synapse appropriate? What are the differences in their functions? What are the differences in thei...
- 5192 Views
- 6 replies
- 6 kudos
- 6 kudos
share you use case i will suggest you about technology difference and which could be benefical for you. I love Data brick due to many awesome feature that help sql developer to programmer(python/Scala) to solve the use case on DataBricks. but if you ...
- 6 kudos
- 761 Views
- 1 replies
- 0 kudos
Getting Different results when I am running the Global Templary table (Transform Query)
We are running a logic in an Azure Databricks notebook using Python with Spark. Initially, we read data from ADLS and load it into a global temporary table to perform data quality checks. We then recreate the same temporary table. Afterward, we use t...
- 761 Views
- 1 replies
- 0 kudos
- 0 kudos
The script is too large to paste in here, so please get in touch with me to to obtain it.
- 0 kudos
- 33321 Views
- 21 replies
- 18 kudos
Resolved! How to limit number of files in each batch in streaming batch processing
Hi,I am running batch job which processes incoming files. I am trying to limit number of files in each batch process so added maxFilesPerTrigger option. But its not working. It processes all incoming files at once.(spark.readStream.format("delta").lo...
- 33321 Views
- 21 replies
- 18 kudos
- 18 kudos
Hi @Sandeep ,Can we usespark.readStream.format("delta").option(""maxBytesPerTrigger", "50G").load(silver_path).writeStream.option("checkpointLocation", gold_checkpoint_path).trigger(availableNow=True).foreachBatch(foreachBatchFunction).start()
- 18 kudos
- 2977 Views
- 5 replies
- 2 kudos
Resolved! Cloud_files function
Hi I'm fairly new to to Databricks and in some examples, blogs,... I see the cloud_files() function being used. But I'm always unable to find any documentation on it? Is there any reason for this? And what is the exact use case for the function? Most...
- 2977 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Jefke ,The cloud_files() function in Databricks is part of the Databricks Auto Loader, a tool used for incremental data ingestion from cloud storage like Azure Blob Storage, Amazon S3, or Google Cloud Storage. This function is specifically optimi...
- 2 kudos
- 954 Views
- 1 replies
- 0 kudos
Workflow Fail safe query
I have a large SQL query that includes multiple Common Table Expressions (CTEs) and joins across various tables, totaling approximately 2,500 lines. I want to ensure that if any part of the query or a specific CTE fails—due to a missing table or colu...
- 954 Views
- 1 replies
- 0 kudos
- 0 kudos
There are few options you can try. Based of the information shared, I am assuming a skeleton for you complicated query as follows: WITH cte_one AS (SELECT *FROM view_one),-- Other CTEs...-- Your main query logicSELECTFROM cte_one-- Joins and other cl...
- 0 kudos
- 7767 Views
- 7 replies
- 5 kudos
Resolved! Migrating data from synapse to databricks
Hello team,I have a requirement of moving all the table from Azure Synapse (dedicated sql pool) to databricks.we have a data coming up from source to azure data lake frequently.we have Azure data factory to load data (data flow does the basic transfo...
- 7767 Views
- 7 replies
- 5 kudos
- 5 kudos
Hi @Krizofe , Just gone through you deatils and thought our similar experience with Azure Synapse to databrick migration. We faced a similar situation and were initially hesitant, One of the my colleague recommanded to use Travinto Technologies acc...
- 5 kudos
- 1466 Views
- 3 replies
- 0 kudos
Databricks Setting Dynamic Local Configuration Properties
It seems that Databricks is somehow setting the properties of local spark configurations for each notebook. Can someone point me to exactly how and where this is being done? I would like to set the scheduler to utilize a certain pool by default, but ...
- 1466 Views
- 3 replies
- 0 kudos
- 0 kudos
You will need to leverage cluster-level Spark configurations or global init scripts. This will allow you to set "spark.scheduler.poo" property automatically for all workloads on the cluster. You can try navigationg to "Compute", select the cluster y...
- 0 kudos
- 2537 Views
- 1 replies
- 0 kudos
cannot import name 'Buffer' from 'typing_extensions' (/databricks/python/lib/python3.10/site-package
I am trying to add messages to an azure service bus from a notebook. But I get error from title. Any suggestions how to solve this?import asynciofrom azure.servicebus.aio import ServiceBusClientfrom azure.servicebus import ServiceBusMessagefrom azure...
- 2537 Views
- 1 replies
- 0 kudos
- 0 kudos
@Sega2 it sounds like the error occurs because the typing_extensions library version in your Databricks environment is outdated and does not include the Buffer class, which is being imported by one of the Azure libraries. Can you first try: %pip inst...
- 0 kudos
- 1570 Views
- 1 replies
- 0 kudos
Bookmark in pdf
I am creating a pdf using pyspark and trying to make bookmarks for each table in the pages. All the bookmarks end up pointing to the first table in the first page. Please help me out here.
- 1570 Views
- 1 replies
- 0 kudos
- 0 kudos
@Sudic29 can you please share more about what you have implemented so far? This requires dynamically tracking the page number during the PDF creation process. Example in python: from PyPDF2 import PdfReader, PdfWriter def add_bookmarks_to_pdf(inpu...
- 0 kudos
- 1123 Views
- 2 replies
- 0 kudos
AI_QUERY does not accept modelParameters argument
I am trying to pass a column of data from python/pandas to Spark, then run AI_QUERY. However, when I attempt to pass modelParameters (such as temperature), the function fails. Below is a minimal example: import pandas as pdqueries = pd.DataFrame([ ...
- 1123 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Walter_C , yes, I am receiving this error when only attempting to set temperature, which should be supported on most if not all models, including the specific models I'm working with. The error message seems to indicate this is a problem with AI_...
- 0 kudos
- 1289 Views
- 1 replies
- 1 kudos
Resolved! Databricks Workflows - Generate Tasks Programmatically
Hi there,I've used databricks workflows to explicitly create tasks with known input parameters (either user input or default parameters). But I'm wondering, what if I want the output of one task to be a list of specific ID's (e.g. id = [7,8,10,13,27]...
- 1289 Views
- 1 replies
- 1 kudos
- 1 kudos
This sounds like a great fit for the For Each task type! Here is the blog, and the documentation
- 1 kudos
- 2361 Views
- 1 replies
- 0 kudos
How to pass task name as parameter in scheduled job that will be used as a schema name in query
I want to run a parametrized sql query in a task. Query: select * from {{client}}.catalog.table with client value being {{task.name}}.if client is a string parameter, it is replaced with quotes which throws an error.if table is a dropdown list parame...
- 2361 Views
- 1 replies
- 0 kudos
- 0 kudos
Try this:select * from identifier(:catalog||'.schema.table') The :catalog is a parameter within DBSQLReplace schema and table with actual names
- 0 kudos
- 2138 Views
- 3 replies
- 2 kudos
How to use custom whl file + pypi repo with a job cluster in asset bundles?
I tried looking through the documentation but it is confusing at best and misses important parts at worst. Is there any place where the entire syntax and ALL options for asset bundle YAMLs are described? I found this https://docs.databricks.com/en/d...
- 2138 Views
- 3 replies
- 2 kudos
- 2 kudos
It took me a while to realize the distinction of the keys inside the task - so for anyone else looking into this: only one of the following keys can exist in a task definition: tasks: - task_key: ingestion_delta # existing_c...
- 2 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
