- 1598 Views
- 1 replies
- 1 kudos
Resolved! External Table on Databricks using DBT(Data Build Tool) Models
How can we create external tables in Databricks using DBT Models?
- 1598 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @hrushi512 ,You can try to use location_root config parameter, as they did below:https://discourse.getdbt.com/t/add-location-to-create-database-schema-statement-in-databricks-to-enable-creation-of-managed-tables-on-external-storage-accounts/6894
- 1 kudos
- 661 Views
- 2 replies
- 0 kudos
Issue with DLT Pipelines
Hi Team,We are trying to create DLT pipeline. The scenario is this:- We already have catalog in unity catalog and under that we have schema called test in which we have 17 tables. Now we are trying to create DLT pipeline and copy the data from those ...
- 661 Views
- 2 replies
- 0 kudos
- 969 Views
- 2 replies
- 5 kudos
Resolved! How to query existing storage and network configuration (AWS)?
Trying to provision a simple workspace. All the code I can find look something like the following, where credential/storage/network resources are created, then referenced. In my case, it's a separate repo and try to reuse existing configurations. I...
- 969 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @ThisNoName, Based on your description it looks like you're trying to get Databricks account level information for networks and storage configurations. You can easily achieve that through the Databricks account API. Here are the docs: - Get all ...
- 5 kudos
- 620 Views
- 0 replies
- 0 kudos
Accessing DBT Articafts, runs, tests from Databricks workflow using automated script.
I am running dbt on a databricks job. It saves all documentation: manifest.json, run_results.json, etc in "Download Artifacts" in a job. I am not able to find out a way to read those in codes, transform and save on databricks. Tried job API. The arti...
- 620 Views
- 0 replies
- 0 kudos
- 511 Views
- 1 replies
- 0 kudos
Databricks Workflow Error
Hi Community,My Workflows has been running smoothly since it was created but for the past week, I have been getting this errorI have tried different method and Documentations but nothing seems to work. All the different jobs in my workflow gets this ...
- 511 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Jay_Kay Could you please attempt to create a table from this view and re-run your job to verify if it works? Additionally, please ensure that you persist the view dependencies as tables as well. Best Regards, Jéssica Santos
- 0 kudos
- 1234 Views
- 0 replies
- 0 kudos
Cannot use Delta Table columns containing struct with date fields in Power BI
Hi everyone,I have a Delta Table in Databricks with a column of struct type (containing a field of type date) and a column of type date:create table date_struct (s struct<d:date>, d date, s_json string); insert into date_struct (s, d, s_json) values ...
- 1234 Views
- 0 replies
- 0 kudos
- 1545 Views
- 1 replies
- 1 kudos
Resolved! How to distribute pyspark dataframe repartition and row count on Databricks?
Try to compare large datasets for discrepancy. The datasets come from two database tables, each with around 500 million rows. I use Pyspark subtract, joins (leftanti, leftsemi) to sorted out the difference. To distribute the workload, I need to repar...
- 1545 Views
- 1 replies
- 1 kudos
- 1350 Views
- 5 replies
- 1 kudos
Read each cell contains SQL from one notebook and execute it on another notebook and export result
Hi, I'm new to databricks, so, excuse me if the question is silly one. I have a requirement to read cell by cell from one notebook (say notebookA) and execute the contents of the cell in another notebook (say notebookB) using a python script. All the...
- 1350 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @ToReSa, If you just want to execute the notebook, calling another notebook would be easier. You can even exchange some data between the notebooks. But if you specifically want to pick each SQL from one notebook and execute it in another notebook,...
- 1 kudos
- 664 Views
- 3 replies
- 0 kudos
How Troubleshooting in user's env
Hi team, I want to do POC, but here I have a question confusing me is that if your teams engineer need access our data plan env to troubleshooting for us, How do you do can get permission to access our env ? could you help me, thank you very much.
- 664 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Wenhui ,But what's your setup? Which cloud provider? Do you use Unity Catalog?
- 0 kudos
- 2766 Views
- 7 replies
- 0 kudos
Autoloader File Notification Mode not working as expected
I am using this given code to read from a source location in ADLS Gen 2 Azure Storage Container. core_df = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .option("multiLine", "false") .option(...
- 2766 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @ibrahim21124 , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your fe...
- 0 kudos
- 605 Views
- 1 replies
- 0 kudos
facing frequent session expiration in Databricks community edition
facing frequent session expiration in Databricks community edition since 15-Aug. Is it a bug or a any technical update going on.Please confirm.Thanks
- 605 Views
- 1 replies
- 0 kudos
- 0 kudos
Facing the same issue.Is it still going on for you ?
- 0 kudos
- 7888 Views
- 11 replies
- 6 kudos
Resolved! When delta is a streaming source, how can we get the consumer lag?
Hi, I want to keep track of the streaming lag from the source table, which is a delta table. I see that in query progress logs, there is some information about the last version and the last file in the version for the end offset, but this don't give ...
- 7888 Views
- 11 replies
- 6 kudos
- 6 kudos
Hey @Yerachmiel Feltzman​ I hope all is well.Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
- 6 kudos
- 37491 Views
- 7 replies
- 5 kudos
Resolved! How do I pass arguments/variables from widgets to notebooks?
Hello,I am looking for a solution to this problem, which is known since 7 years: https://community.databricks.com/s/question/0D53f00001HKHZfCAP/how-do-i-pass-argumentsvariables-to-notebooksWhat I need is to parametrize my notebooks using widget infor...
- 37491 Views
- 7 replies
- 5 kudos
- 5 kudos
Can we create paginated reports with multiple parameters(one parameter can dynamically change other parameter) or we can pass one variable from one dataset to other dataset like power bi paginated report using Databricks dashboard, please let me know...
- 5 kudos
- 2460 Views
- 3 replies
- 0 kudos
Workflow - How to find the task id at run time in current notebook
There are four tasks in the workflow. How can I get the task ID at the beginning of the notebook, store it after finishing all the code cells in the notebook, and then save it into a table?
- 2460 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @rameshybr,You can capture these as parameters in the task configuration.And from within the notebook you could use the widget utils to get their values.
- 0 kudos
- 859 Views
- 2 replies
- 1 kudos
Apply changes in Delta Live tables
Hi,I have created delta live tables (@dlt), need to capture all CDCs (all insert, update, delete) happening on our source.I tried with by creating streaming live table but still not able to achieve.Does delta live tables (@dlt) does support only appe...
- 859 Views
- 2 replies
- 1 kudos
- 1 kudos
The APPLY CHANGES APIs: Simplify change data capture with Delta Live Tables | Databricks on AWS check with this
- 1 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
