Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I'd like to create an SQL table in a notebook that's visible to a specific run (session?) of a notebook. Meaning, even if 2 different users run that notebook at the same time, there should be no conflict. And the table should go away once the noteboo...
For now ‘CREATE TEMPORARY VIEW’ is the way to go. Once you read from it once, the following reads are going to be cached so it won’t be recomputed every time.
Does anyone have any experience migrating a gold layer from Azure Analysis Services (AAS) tabular structure to one centered around the lakehouse architecture? My company currently uses AAS to serve data to all levels of users across the company, but ...
Lakehouseiq looks really cool! Lakehouseiq is a Llm agent that builds sql queries from natural language. Ties into their other products like biz context lineage and previous queries.
Love the idea of creating and sharing apps in the Databricks Marketplace as shown during the keynote speech today, but how do I create them?
How do I time how long does my code run in DBX.
Hi, you can use the standard python timeit function. ''' %timeitprint("hello world")'''
I love Databricks! It’s the best modern data platform solution !!
Today was the first day of summit and learned about how different models can be deployed for different decision points. I'm curious to know what users have found to be model control issues regarding this. Thanks!
With multiple locations for items to exist, duplicate names make it hard to tell which one is which without opening.
@Eric Kieft​ :In Databricks, the Recents view shows the recently accessed notebooks, dashboards, and folders. However, it does not show the exact location of the item. To determine the location of an item in the Recents view, you can try the followin...
We have noticed that users can schedule SQL queries, but currently we haven't found a way to find these scheduled queries (this does not show up in the jobs workplane). Therefore, we don't know that people scheduled this. The only way is to look at t...
@Paulo Rijnberg​ :In Databricks, you can use the following approaches to prevent users from scheduling SQL queries and to receive notifications when such queries are scheduled:Cluster-level permissionsJobs APINotification hooksAudit logs and monitori...
I want to move Databricks Jobs, SQL dasboards, Queries and Alerts from lower environment to higher environment, how we can move?
Hi @Shubham Agagwral​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...
I am building a data pipeline using Delta Live table in Azure Databricks to move data from a raw data table to a feature table and model inference results table. However, I am concerned about the potential for duplication issues in future operations,...
Hi @Chengcheng Guo​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
Hi, I'm currently starting to use SQL Warehouse, and we have most of our lake in a compression different than snappy. How can I set the SQL warehouse to use a compression like gzip, zstd, on CREATE, INSERT, etc? Tried this: set spark.sql.parquet.comp...
Hi @Alejandro Martinez​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
I created the following model: which calls get_identifier_information() which is as follows: This is how I log the model And this is the error I am running into: RuntimeError: It appears that you are attempting to reference SparkContext...
Hi @Nikhil Gajghate​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
Hello, we're receiving an error when running glue jobs to try and connect to and read from a Databricks SQL endpoint. Hello, we're receiving an error when running glue jobs to try and connect to and read from a Databricks SQL endpoint. An error occ...
Hello @Vidula Khanna​ @Debayan Mukherjee​ ,I wanted to give you an update that might be helpful for your future customers, we worked with @Pavan Kumar Chalamcharla​ and through lots of trial and error we figured out a combination that works for SQL e...
I am trying to identify errors coming from Databricks. So I can handle them in my code.Sometimes I get a descriptive error, that points me to the exact problem, but then if I run the exact same test, I sometimes get "request error: invalid operation ...
Can you share what command were you executing ? Also were you currently doing exception handling within your code with try/catch exception ? Can you also check the driver logs during the time error happened ? The driver logs should have more detail...
