- 4965 Views
- 5 replies
- 1 kudos
Load tables from JDBC in parallel
I have a list of about 80 tables that I need to load from an Oracle database into Databricks via JDBC. I would like to do this in parallel, instead of looping through one table at a time.I have a function defined to ingest the data: def ingest_data...
- 4965 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @ADB0513, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will h...
- 1 kudos
- 3887 Views
- 3 replies
- 1 kudos
Random errors SparkException: Job aborted due to stage failure
Hi, we are trying to run some workflows on a shared cluster, with Databricks runtime version 14.3 LTS, and we randomly receive the error: SparkException: Job aborted due to stage failure: Task 2 in stage 78.0 failed 4 times, most recent failure: Lost...
- 3887 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @l_c_s, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will hel...
- 1 kudos
- 2503 Views
- 4 replies
- 1 kudos
Get metadata of files present in a zip
I have a .zip file present on an ADLS path which contains multiple files of different formats. I want to get metadata of the files like file name, modification time present in it without unzipping it. I have a code which works for smaller zip but run...
- 2503 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @seeker, There are only 2 ways I can think of to do it: Write a UDF.Write customized MapReduce logic instead of using Spark SQL. But they are kind of the same. So I would say UDF is a good solution.
- 1 kudos
- 1501 Views
- 2 replies
- 2 kudos
CSV file and partitions
Hi I want to know whether csv files can be partitioned or not. I find from a book that parque, Avro .. these types of files can be partitioned only. RegardsSubhas
- 1501 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @subhas_1729, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This wi...
- 2 kudos
- 7198 Views
- 3 replies
- 3 kudos
Connect power bi desktop semantic model output to databricks
Hello, I am trying to connect the power bi semantic model output (basically the data that has already been pre processed) to databricks. Does anybody know how to do this? I would like it to be an automated process so I would like to know any way to p...
- 7198 Views
- 3 replies
- 3 kudos
- 4165 Views
- 0 replies
- 0 kudos
Not able to access partner-academy
Hi,I have used my company email to register an account for customer-academy.databricks.com a while back.Now what I need to do is create an account with partner-academy.databricks.com using my company email too.However when I register at partner-acade...
- 4165 Views
- 0 replies
- 0 kudos
- 4152 Views
- 3 replies
- 0 kudos
Extract cluster usage tags from databricks cluster init script
Is it possible we extract cluster usage tags from databricks cluster init script, I am specifically interested in spark.databricks.clusterUsageTags.clusterAllTags.I tried to extract from /databricks/spark/conf/spark.conf and /databricks/spark/conf/sp...
- 4152 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, For reference: https://community.databricks.com/t5/data-engineering/pull-cluster-tags/td-p/19216 , could you please confirm the key expectation here? Extracting as such?
- 0 kudos
- 3931 Views
- 0 replies
- 0 kudos
Free Voucher Worth 200$ in the upcoming events are there Please send me note on it
I need info about any upcoming events that databricks will provide me a free voucher for the Azure platform architect exam , anyone know the time or a hunch about it please ping me the details. i will be stay tuned at that point of time.................
- 3931 Views
- 0 replies
- 0 kudos
- 4825 Views
- 5 replies
- 5 kudos
DLT pipeline - SCD type 2
I created my table using SCD type 2 in SQL. I need to do full refresh to load all of the data. Whenever I update data in my source table, in my new table scd_target I see only the latest record, history is not being saved. CREATE OR REFRESH STREAMING...
- 4825 Views
- 5 replies
- 5 kudos
- 5 kudos
hi @ksenija i got your use case but could you please tell me , what do you mean by "sources_test.sources_test.source"
- 5 kudos
- 1209 Views
- 1 replies
- 1 kudos
Where i can find notebooks for Data Engineering with Databricks
Where i can find notebooks for Data Engineering with Databricks
- 1209 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @michalsadlocha ,You can find them at Slides & DBC section:
- 1 kudos
- 1939 Views
- 3 replies
- 0 kudos
API key deletions
On Thursday the 15th August I noticed that several API keys over all of the accounts I run on Databricks had been removed. They had no expire date and some accounts where on a Azure hosted workspace and some on AWS. I would like to hear if you experi...
- 1939 Views
- 3 replies
- 0 kudos
- 0 kudos
@Retired_mod Yes this is one of the accounts that have had the token removed. I sent an e-mail about the other account.@ChendaZhangDB Thank you for the idea, this could be the case for 2 tokens that were removed. However I also had 2 tokens removed t...
- 0 kudos
- 3805 Views
- 1 replies
- 1 kudos
Resolved! External Table on Databricks using DBT(Data Build Tool) Models
How can we create external tables in Databricks using DBT Models?
- 3805 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @hrushi512 ,You can try to use location_root config parameter, as they did below:https://discourse.getdbt.com/t/add-location-to-create-database-schema-statement-in-databricks-to-enable-creation-of-managed-tables-on-external-storage-accounts/6894
- 1 kudos
- 1547 Views
- 2 replies
- 0 kudos
Issue with DLT Pipelines
Hi Team,We are trying to create DLT pipeline. The scenario is this:- We already have catalog in unity catalog and under that we have schema called test in which we have 17 tables. Now we are trying to create DLT pipeline and copy the data from those ...
- 1547 Views
- 2 replies
- 0 kudos
- 2776 Views
- 2 replies
- 5 kudos
Resolved! How to query existing storage and network configuration (AWS)?
Trying to provision a simple workspace. All the code I can find look something like the following, where credential/storage/network resources are created, then referenced. In my case, it's a separate repo and try to reuse existing configurations. I...
- 2776 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @ThisNoName, Based on your description it looks like you're trying to get Databricks account level information for networks and storage configurations. You can easily achieve that through the Databricks account API. Here are the docs: - Get all ...
- 5 kudos
- 1292 Views
- 1 replies
- 0 kudos
Databricks Workflow Error
Hi Community,My Workflows has been running smoothly since it was created but for the past week, I have been getting this errorI have tried different method and Documentations but nothing seems to work. All the different jobs in my workflow gets this ...
- 1292 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Jay_Kay Could you please attempt to create a table from this view and re-run your job to verify if it works? Additionally, please ensure that you persist the view dependencies as tables as well. Best Regards, Jéssica Santos
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
