- 776 Views
- 1 replies
- 1 kudos
How to use identifier() as reference
Hello,i am trying to build some dynamic sql scripts using this identifier() clause. This works fine when it is used with [ select * from identifier (mytable) ]. However, when I try to use this identifier as a reference to a foreign key table, it does...
- 776 Views
- 1 replies
- 1 kudos
- 1 kudos
Is this something you are working to resolve?We would also like to use IDENTIFIER() with our REFERENCE(s) CONSTRAINT(s)If not - is it possible to get an explaination why the limitation exists?
- 1 kudos
- 7571 Views
- 3 replies
- 2 kudos
Why my notebook I created in a Repo can be opened safe?
I've cloned a Repo during "Get Started with Data Engineering on Databricks".Then I'm trying to run another notebook from a cell with a magic %run command.But I get that the file can't be opened safe.Here my code:notebook_aname = "John" print(f"Hello ...
- 7571 Views
- 3 replies
- 2 kudos
- 2 kudos
It could be that you need to put the %run in a cell all by itself. Suggested here: https://stackoverflow.com/a/72833400/1144966
- 2 kudos
- 2397 Views
- 3 replies
- 0 kudos
Serverless Compute Cost Monitoring (System Tables)
Hello,I have developed a dashboard for monitoring compute costs using system tables, allowing tracking of expenses by Cluster Name (user created name), Job Name, or Warehouse Name. However, with the introduction of the new shared serverless compute, ...
- 2397 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi! We're also facing the same problem. We don't have any materialized views or streaming tables, but we are still seeing PREMIUM_JOBS_SERVERLESS_COMPUTE_EU_NORTH with billing_origin_product = SHARED_SERVERLESS_COMPUTE generated each day at a time wh...
- 0 kudos
- 2073 Views
- 5 replies
- 3 kudos
Best Way to process large number of records from multiple files
Hi,I have input files in S3 with below structure./mnt/<mount_name>/test/<company_id>/sales/file_1.json/mnt/<mount_name>/test/<company_id>/sales/file_2.json/mnt/<mount_name>/test/<company_id>/sales/file_<n>.jsonNumber of companies = 15Number of files ...
- 2073 Views
- 5 replies
- 3 kudos
- 3 kudos
Are the json files compressed? If they are in .gz, this is unsplittable which means you lose some of spark's parallel magic.
- 3 kudos
- 1888 Views
- 5 replies
- 1 kudos
Load tables from JDBC in parallel
I have a list of about 80 tables that I need to load from an Oracle database into Databricks via JDBC. I would like to do this in parallel, instead of looping through one table at a time.I have a function defined to ingest the data: def ingest_data...
- 1888 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @ADB0513, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will h...
- 1 kudos
- 1820 Views
- 3 replies
- 1 kudos
Random errors SparkException: Job aborted due to stage failure
Hi, we are trying to run some workflows on a shared cluster, with Databricks runtime version 14.3 LTS, and we randomly receive the error: SparkException: Job aborted due to stage failure: Task 2 in stage 78.0 failed 4 times, most recent failure: Lost...
- 1820 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @l_c_s, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will hel...
- 1 kudos
- 1062 Views
- 4 replies
- 1 kudos
Get metadata of files present in a zip
I have a .zip file present on an ADLS path which contains multiple files of different formats. I want to get metadata of the files like file name, modification time present in it without unzipping it. I have a code which works for smaller zip but run...
- 1062 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @seeker, There are only 2 ways I can think of to do it: Write a UDF.Write customized MapReduce logic instead of using Spark SQL. But they are kind of the same. So I would say UDF is a good solution.
- 1 kudos
- 617 Views
- 2 replies
- 2 kudos
CSV file and partitions
Hi I want to know whether csv files can be partitioned or not. I find from a book that parque, Avro .. these types of files can be partitioned only. RegardsSubhas
- 617 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @subhas_1729, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This wi...
- 2 kudos
- 4752 Views
- 3 replies
- 3 kudos
Connect power bi desktop semantic model output to databricks
Hello, I am trying to connect the power bi semantic model output (basically the data that has already been pre processed) to databricks. Does anybody know how to do this? I would like it to be an automated process so I would like to know any way to p...
- 4752 Views
- 3 replies
- 3 kudos
- 1253 Views
- 0 replies
- 0 kudos
Not able to access partner-academy
Hi,I have used my company email to register an account for customer-academy.databricks.com a while back.Now what I need to do is create an account with partner-academy.databricks.com using my company email too.However when I register at partner-acade...
- 1253 Views
- 0 replies
- 0 kudos
- 2363 Views
- 3 replies
- 0 kudos
Extract cluster usage tags from databricks cluster init script
Is it possible we extract cluster usage tags from databricks cluster init script, I am specifically interested in spark.databricks.clusterUsageTags.clusterAllTags.I tried to extract from /databricks/spark/conf/spark.conf and /databricks/spark/conf/sp...
- 2363 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, For reference: https://community.databricks.com/t5/data-engineering/pull-cluster-tags/td-p/19216 , could you please confirm the key expectation here? Extracting as such?
- 0 kudos
- 1160 Views
- 0 replies
- 0 kudos
Executing Stored Procedures/update in Federated SQL Server
I have federated Azure SQL DB in my DBX workspace, but I am not able to run update commands or execute a stored procedure, is this still not supported?
- 1160 Views
- 0 replies
- 0 kudos
- 1202 Views
- 0 replies
- 0 kudos
Free Voucher Worth 200$ in the upcoming events are there Please send me note on it
I need info about any upcoming events that databricks will provide me a free voucher for the Azure platform architect exam , anyone know the time or a hunch about it please ping me the details. i will be stay tuned at that point of time.................
- 1202 Views
- 0 replies
- 0 kudos
- 2276 Views
- 5 replies
- 5 kudos
DLT pipeline - SCD type 2
I created my table using SCD type 2 in SQL. I need to do full refresh to load all of the data. Whenever I update data in my source table, in my new table scd_target I see only the latest record, history is not being saved. CREATE OR REFRESH STREAMING...
- 2276 Views
- 5 replies
- 5 kudos
- 5 kudos
hi @ksenija i got your use case but could you please tell me , what do you mean by "sources_test.sources_test.source"
- 5 kudos
- 1156 Views
- 0 replies
- 0 kudos
Upgrade HSM to UC using Ucx tool - workspace to workspace migration
Hello team,I understand that an automatic upgrade to UC utilizing the UCx tool (Databricks Lab project) is now available to complete this migration from HSM to UC in automate way. But does this tool allow workspace to workspace catalog/artifact migra...
- 1156 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
