Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
Dear all,(migrating for an on-premise Oracle ...)The question is in the subject: "What is the equivalent of Oracle's CLOB in Databricks" ?I saw that the "string" type can go up to 50 thousands characters, which is quite good in most of our cases, but...
Hello;Thanks for the answer.For the concatenation itself, it is not an issue.My question is "is Databricks supporting something bigger than the 'string' data-type" ? Thanks
Hi all,- migrating from an on-premise Oracle -Currently on Oracle, I have a "library" of let's say 300 tables to load, sequentially, based on views (some tables being fed potentially by several views, therefore the number of underlying views is highe...
The common scenario for data processing in Oracle is based on PL/SQL and cursors. In the case of PySpark we don't have such concept as cursors and iteration on data frames can lead to poor performance.I migrated Oracle to Databricks and I learned tha...
I am looking to monitor my SQL Warehouse, especially the 'Running Clusters' metric that is available in the monitoring tab of the warehouse. This shows the start and shut down time as well as the number of running clusters:The issue I have run into i...
Hi. I have a local MongoDB running on an EC2 instance in the same AWS VPC as my Databricks cluster but cannot get Databricks to talk to MongoDB. I've followed the guide at https://docs.databricks.com/aws/en/connect/external-systems/mongodb and have a...
I followed the official Databricks documentation("https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html")to integrate MongoDB Atlas with Spark by setting up the MongoDB Spark Connector and configuring the connection string in my Datab...
Hi @vidya_kothavale ,Could you try to change "spark.mongodb.input.uri" to following?spark.read.format("mongodb").option("spark.mongodb.read.connection.uri"
Hello,I maintain a spark plugin with users who are moving from spark to databricks and want to use shared access mode. My library has a bunch of custom datasources and users are seeing errors when using them. Is there a way to write a custom datasour...
Hello @bcb44, Thanks for your question. Data Source V2 Relations: These are not supported in clusters that are configured with Table ACL or Credential Passthrough. Can you validate if ACL or Passthrough are enabled on the cluster? Also this should w...
Hello communityI am facing an issue when I export a dashboard to an HTML: the problem is that the HTML has white background but the exported tables have black colors, like the example I am showing below Does someone know how to handle this visualizat...
Hi All,Does anyone noticed if you "run all" the first time on the notebook, later if you click "run all below" on a cell, that wouldn't work anymore, and require to click "run all" again.It doesn't happen to me about couple of weeks ago, I used to ru...
Hello @Paddy_chu! This doesn't happen with every notebook, but it's likely due to cell dependencies. When you Run All, the final cell might modify or clear the data. Later, if you use Run All Below from a middle cell, it may not work if the required ...
Hey all,I have encountered a strange issue while validating my dashboard visual against the source table.The visual shows the count(distinct VAL) per day and when I ran a query that does the same calculation I get a difference of 84 (on average) and ...
Hi @yuvala, Are you using same query for the dashboard? could you please share it?
Hi All,How can I maintain 7 years of transactional data in delta table? Can I have log retention of 7 days, but data retention of 7 years?Appreciate your response.Thanks and regards Suhas
I would like to be able to express something like: WHERE columnA = "valueA" OR columnB = "valueB" but on the Widget. I only see the possibility that chaining filters that are processed as AND operator. Am I missing something?
Hello,I'm not sure if it is the correct place to post this, sorry.Migrating from an on-premise Oracle to Databricks, we are wondering about the following functionality:. From the reporting tool in place (currenly, PowerBI), users are able to send bac...
Hello Mantu,Thanks for your answer.This is clear, and as our colleague on PowerBI is already dealing with Power Automate, he should be able to test this.If we will be allowed to use Databricks REST API (our infra guys will tell us), I guess we will b...
We are unable to create a Serverless Warehouse Cluster at our own databricks workspace. The same settings do work on other Azure tenants that I have access to.The workspace is running in Azure on a Premium Plan in West Europe.Features enabled:Automat...
Was this workspace moved to Premium plan recently or have been Premium since creation?
Hi-We have a delta table in our unity catalog called dream_team.stern_portfolio.location_info.We are trying to use row level security to filter our data based on a users group membership. This way when users look at out dashboard they can only see th...
Hey @HeathDG1 I think your function isn’t behaving the way you expect because of how the logic is set up:•If the user is in Stern MA, they only see rows where state = 'MA' (which is good).•BUT for everyone else, the function returns true, meaning th...
I would like to know if the cluster size of a Serverless Warehouse can automatically scale up and down, and what determines the number of workers used when executing queries. Does it use all workers within the cluster size fixedly, or does it use par...
@fishingrod My understanding is that Intelligent Workload Management (IWM) in Serverless SQL Warehouses adjusts the number of clusters, but it does not automatically scale the cluster size.This means that if you need to improve the execution performa...
