- 3572 Views
- 10 replies
- 1 kudos
My DABS CLI Deploy call not generating a .tfstate file
Hi Community,I'm running into an issue, when executing Databricks CLI Bundle Deploy I dont get the Terraform State file (.tfstate). I know that I should get one but even when defining the state_apth on my YAML (.yml) DABS file I still do not get it.D...
- 3572 Views
- 10 replies
- 1 kudos
- 5705 Views
- 4 replies
- 2 kudos
[AWS] How do you replace the Account Admin?
I need to remove an older admin that previously set up the Databricks Account. However, I get an error (even through I am also an Account Admin).How do I replace a prior account admin? Or at least remove their admin status and/or disable the accoun...
- 5705 Views
- 4 replies
- 2 kudos
- 2 kudos
Was this ever fixed as I have got the exact same issue as our original AWS Account Admin has left the business and I now cannot remove his Account Admin role (but I have disabled access etc)!
- 2 kudos
- 3658 Views
- 4 replies
- 0 kudos
Possibilities and Limitations of Delta Live Tables (DLT) with Direct Publish mode
Hi all,I am implementing a DLT table with the new Direct Publish feature which is still in Private Preview.Is it a limitation of DLT with Direct Publish that you can not query the event_log of the DLT?When I use this query:SELECT * FROM event_log('pi...
- 3658 Views
- 4 replies
- 0 kudos
- 0 kudos
If any one is interested how to access the event_log for DLTs that are writing to different schemas:The approach that was described above was not working anymore - at least for us.But we discovered this post:https://stackoverflow.com/questions/794329...
- 0 kudos
- 12685 Views
- 4 replies
- 1 kudos
Resolved! Connect and process Azure analysis services
How do I connect to Azure analysis services from databricks? I need to process the tabular model from databricks. I tried to use adodbapi. while connecting it is failing with error message "windows com error dispatch adodb.connection". please help
- 12685 Views
- 4 replies
- 1 kudos
- 1 kudos
I got another use case now "to run dax against Azure Analysis Services model" from AWS databricks. I tried above suggestion from "Jun Yang" and it is erroring out after 30 seconds with the exception that "Login timeout is expired"
- 1 kudos
- 3781 Views
- 2 replies
- 1 kudos
Multiple DLT pipelines same target table
Is it possible to have multiple DLT pipelines write data concurrently and in append mode to the same Delta table? Because of different data sources, with different data volumes and required processing, we would like to have different pipelines stream...
- 3781 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi I wanted to understand how you set "pipelines.tableManagedByMultiplePipelinesCheck.enabled" to true? I'm not able to figure out how and where you set this param
- 1 kudos
- 4822 Views
- 4 replies
- 0 kudos
Resolved! Legacy Pipeline?
What is this legacy pipeline warning? I cannot find anything related on the internet and related forums. Thanks in advance.
- 4822 Views
- 4 replies
- 0 kudos
- 0 kudos
Ah I've found the issue. There is one more diff:This is the new key-value for target schema: "schema": "dev", This is the old key-value for target schema: "target": "prod" After changing the key from "target" to "schema", the warning has gone....
- 0 kudos
- 4238 Views
- 2 replies
- 1 kudos
Resolved! Configure Soft-Delete Retention Period of Unity Catalog Managed Table
According to the docs,When a managed table is dropped from Unity Catalog, its underlying data is deleted from your cloud tenant within 30 days.Can I configure the retention period? Is executing DELETE FROM my_table followed by VACUUM my_table RETAIN ...
- 4238 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @guiferviz ,Yes, that is correct! The only way is to use DELETE and VACCUM commands. For managed tables, the default retention period is set to 30‑days. Unity Catalog enforces this to provide an additional safety net against accidental data loss, ...
- 1 kudos
- 68872 Views
- 5 replies
- 3 kudos
Resolved! How to set a variable and use it in a SQL query
I want to define a variable and use it in a query, like below: %sql SET database_name = "marketing"; SHOW TABLES in '${database_name}';However, I get the following error:ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near ''''(line 1, pos...
- 68872 Views
- 5 replies
- 3 kudos
- 3 kudos
CJS had the best answer by virtue of it being code-based rather than widget-based. In a notebook where the value of the variable must continually be reset, widgets are suboptimal. That said, the easiest way is to wrap the code in python:%pyvar1 = s...
- 3 kudos
- 18145 Views
- 4 replies
- 0 kudos
Using a Virtual environment
Hi All,We are working on training NHits/TFT (a Pytorch-forecasting implementation) for timeseries forecasting. However, we are having some issues with package dependency conflicts.Is there a way to consistently use a virtual environment across cells ...
- 18145 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello @sshssh Databricks support Notebook scoped environments for databricks runtime (regular and ML) Kindly please check blog post here for ways to install Notebook scoped libraries in databricks Regards
- 0 kudos
- 2290 Views
- 1 replies
- 0 kudos
Update target column with optional source columns
I am trying to load on prem transaction log table and update a databricks table.In Source table,the column reqdetails table hold all the information and expect ProductID remaining columns are dynamic(not all columns exists in the request) all time th...
- 2290 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi RucHow are you doing today?, as per my understanding, Since your source table has a ReqDetails column with dynamic key-value pairs and only ProductID is fixed, you can parse the ReqDetails column dynamically and merge it into the target table usin...
- 0 kudos
- 1299 Views
- 1 replies
- 0 kudos
Facing issue while accessing DLT pipeline event logs.
while accessing event logs created by dlt pipeline, i am getting below error.[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the correspondin...
- 1299 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Pawan,How are you doing today? As per my understanding, This error likely indicates an internal issue with Databricks, Delta Live Tables (DLT), or a Spark plugin. First, check if there are any known issues in Databricks release notes or community ...
- 0 kudos
- 1688 Views
- 1 replies
- 0 kudos
how to upload excel file from power apps to databircks notebook folder
Hi All,User uploads an excel file from power apps interface, excel file has to store in databricks folder and insert the excel data into sql table, storing excel file in databricks and inserting data has to go through power automate. Please suggest ...
- 1688 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sridhark How are you doing today? As per my understanding, You can achieve this by setting up a Power Automate flow that triggers when a user uploads an Excel file in Power Apps, storing it in OneDrive or SharePoint first. Then, use Power Automat...
- 0 kudos
- 5303 Views
- 5 replies
- 2 kudos
User entry via portal/UI
Hello Experts,We have data bricks on Azure. We need to provide a user interface to users so that some customizing table entries can be entered by end users which in turn are saved in the Delta table. Is there any feature in Databricks or tools that w...
- 5303 Views
- 5 replies
- 2 kudos
- 2 kudos
I achieved this using textbox widget and asked user to enter and run dashboard, in python code validating and inserting data in databricks.2nd method , created excel vba application, using odbc connection to insert delete and update data in delta tab...
- 2 kudos
- 1041 Views
- 1 replies
- 0 kudos
Keywords and Functions supported in Vertica SQL but not in Databricks SQL.
I have to convert Vertica queries in Databricks SQLs, so that I can run them in databricks environment. So I want to know the list of all keywords, functions or anything that is different in databricks SQL.
- 1041 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @vidya_kothavale, Please refer to: https://docs.databricks.com/aws/en/sql/language-manual https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-reserved-words
- 0 kudos
- 2706 Views
- 2 replies
- 0 kudos
Databricks Apps - Streamlit Performance Hangups
Hi All,I have a Streamlit app running via Databricks Apps. The app is fairly simple and displays data from a small handful of lightweight database queries (each running in less than 1 second).As of a few days ago, this app was running great. But as o...
- 2706 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @brianr, Have you validated the SQL warehouse you are referring in your code? I would be useful to identify the delayed queries and check it's status from "Query History" in Databricks UI.
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
