Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I have a Python script workflow with 2 tasks: Task A and Task B.When task A has data, this is shared to Task B via createOrReplaceGlobalTempView with no issues.The goal is: When A has no data, skip the Task B and also set the workflow status to "Skip...
To achieve the goal of setting the workflow status to "Skipped" when Task A has no data, you can use the "Run if" conditional task type in Databricks Jobs. This allows you to specify conditionals for later tasks based on the outcome of other tasks.ht...
I have some legacy software that only runs on Windows, but that can be driven via Python. Is it possible to set up compute resources that run Databricks Container Service on a windows base image, so that I can then add this legacy software and work w...
Unfortunately this is not possible, as part of the requirements you need to use an Ubuntu image: https://docs.databricks.com/en/compute/custom-containers.html#option-2-build-your-own-docker-base
Hello EveryoneRecently, my team integrated an AWS Redshift database as a foreign catalog in Azure Databricks. We can successfully run SELECT queries and create regular views on top of the foreign catalog table. However, when attempting to create a ma...
Hi @Dnirmania, Materialized views in SQL often use serverless Delta Live Tables pipelines, which might be causing the connection timeout due to IP whitelisting restrictions. Serverless compute might not be able to connect to the federated source if t...
Hello all,I'm quite new to Databricks world, and currently in the process of analyzing a migration from Oracle on-premise (with a lot of SQL, PL/SQL, custom things, etc.) to Databricks.Let's try to illustrate my situation in Oracle (summary):. Let's ...
Hello, sure if any additional information is needed just let us know and we will be happy to assist.
I have to migrate the data from Azure Synapse Analytics to Databricks. Could anyone share the different approaches to migrate data, and from those, which is the best approach to use?
Can we mount the Syanpse storage account ADLS Gen2 directly to Databricks and then we can only migrate pipeline/code to databricks
Hello all,I want to install Oracle Instant Client to be able to use python-oracledb in Thick-Mode because one of our databases is old and cannot be reached in Thin-Mode. I have tried the solution from this post, but it doesn't help me. It seems that ...
What about shared compute, is it possible? Neither of those scripts is working for me really...
Hi All,I'd like to do some benchmarking and I need to turn off caching on my SQL Warehouse Server. However, whatever I try I receive quite high level of caching after running my queries (>60%). I tried to turn off my server, but it automatically wake...
To disable query result caching, you can run SET use_cached_result = false in the SQL editor.
We set up the SQL warehouse IAM role in the settings option. This is applied to all warehouses. How do I create sql warehouses with multiple IAM roles to maintain access control.
Unfortunately there is no way to restrict the access the compute has, the restrictions are being performed via the users permissions. Only option here will be to submit a feature request through https://docs.databricks.com/en/resources/ideas.html#ide...
Hello Databricks Community,I have a hard time understanding how is Databricks SQL different from microsoft SQL ? Also, why does databricks provide spark SQL ? If you direct me to a well-written webpage or document its of immense help!Thanks,
Databricks SQL and Spark SQL are built for distributed big data analytic. Databricks SQL is great for business intelligence tools and uses Delta Lake for efficient data storage. Spark SQL works with Spark's programming features for data processing. U...
Hello there,Wasn't sure if this was just an error on my part, but I'm using a Databricks Pro SQL warehouse and unity catalogue to pull some data from my tables. I'm having this issue where whenever I try and use a wildcard operator with my LIKE claus...
Hi @nomnomnom543 ,Try this SELECT * FROM table_name WHERE LEFT(column_name, LENGTH('string')) = 'string';
Hello,I have an Azure sql warehouse serverless instance that I can connect to using databricks-sql-connector. But, when I try to use pyspark and jdbc driver url, I can't read or write.See my code belowdef get_jdbc_url(): # Define your Databricks p...
The error does not look specific to the warehouse that you are connecting to. The error message "Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern" indicates that there is an issue with the logging configuration in...
With Databricks BI, Is there anyway to create a KPI or Measures that can be used in a visual? If so, how to achieved that one?
I have seen in the release notes of 15.3 that this was introduced and couldn't wrap my head around it.Does someone has an example of a plan before and after?Quote:Performance improvement for some window functionsThis release includes a change that im...
Before Optimization: Consider a query that calculates the sum of a column value partitioned by category without an ORDER BY clause or a window_frame parameter: SELECT category, SUM(value) OVER (PARTITION BY category) AS total_value FROM sales; ...
I have two tasks. The first task (Sample_Notebook) returns a JSON array (Input_List). Sample data in Input_List['key1':value1, 'key2':value2, 'key3':value3]The second task is a "For Each" task that executes a notebook for each entry in the Input_List...
To access each item of the iteration within the notebook of the second task in your Databricks workflow, you need to utilize the parameterization feature of the For Each task. Instead of trying to retrieve the entire list using dbutils.jobs.taskValue...
https://docs.databricks.com/en/query-federation/index.htmlAre there plans to provide Oracle support for Databricks on AWS lakehouse federation? Not sure if that's possible or not. Thanks!
Federation with Oracle is available https://learn.microsoft.com/en-us/azure/databricks/query-federation/oracle
