Hi,We are working on ingesting multiple files from S3. The files name are fixed based on our source system, Files get replaced frequently with full feed. In DLT when we process new file we have to delete the records processed earlier of the same file...
Hi Everyone.I am trying to connect and read data from the Databricks table using SQL Warehouse and return it using Azure API.However, the non-English characters, for example, 'Ä', are present in the response as following: ��.I am using the databricks...
If Databricks support/Product Management following the forum, note that PDF from SIMBA in 2.6.28 does not discuss the name-value pairs in the above solution.Other errata includes PreparedMetadataLimitZero.
If I want to use SSD or GPU in my cluster,is that possible to choose?
Hello,What is the best practice to modify/delete/recreate groups properly ?In order to rename a group, the only mean was to delete/recreate. But after deletion in the account console, the permissions granted to the deleted groups in the tables were i...
Hello,I have exactly the same issue - I am also using terraform.I deleted a group and the catalog permissions are in bad state. I am not able to revoke access to this group using the Databricks UI nor REST API. I also tried to recreate the group wit...
We enabled liquid clustering on one of the large tables (380GBs). This table goes many operations daily, which improved many folds after liquid clustering. However, after enabling liquid clustering and optimizing it number of files are increased.Prev...
Hello,since Asset Bundles is based on GO templating mechanism, I am wondering how it is possible to use IF-ELSE construct within a template file, to define which file content will be set in the generated file ( I want to have that in my custom templa...
Never mind....I figured out the solution I just have to prefix my template file with ".tmpl", then it gets rendered correctly
Hi,Job params override the task params(same name params). Is there a way task params override the job params.Use case:job params: a = "param-1".job has 12 tasks. 10 of them should use job param(a = "param-1").2 of them should override the job param(a...
Hi @ajithgaade,Unfortunately, I don't think it is currently possible. It's clearly stated in documentation that:" Job parameters take precedence over task parameters. If a job parameter and a task parameter have the same key, the job parameter overri...
Hi Everyone,I am happy to be part of this great community.I just determined to be a Data Engineer by profession and I will need a lot of advice on how I can quickly grab it and become a professional.I have Python Programming knowledge and Web develo...
Most import thing, at least at the beginning of your data journey is to grasp a good understanding of SQL. It's cornerstone in data world.Definitely you should familiarize yourself with a concept of data modeling, especially dimensional modeling that...
Hi,I am trying to create a materialized view; however, it failed. I could not find any reason for this issue.Cluster Summary:Cluster Type: SQL warehouse serverless with Unity CatalogWorkers: 1-4 Workers (16-64 GB Memory, 4-16 Cores)Driver: 1 Driver ...
Hey @taimoor123 , could you share the stack trace for the error please ? Also, I find it a bit confusing. You mention you are using a Serverless SQL warehouse to run the query, but you are also providing the details of the configuration of what seem...
Hello TeamI am using DLT. I am able to see the lineage when doing normal process. However as soon as I use 'APPLY_Changes' feature after the lineag ebreaks and I am no more able to see the Data lineage from the catalog after going to table. Is there ...
@Retired_mod : Is there any way this can be achieved or can we expect this problem to be resolved in next releases?
An init script is used to install system libraries, per below.Adding the script to a Personal Compute consistently works. The same script is added to a Workflows job via cluster config, which intermittently fails, as shown in error message below.Both...
Check the cluster event log to see if there is a clue why the script is failing. if the script failed and returned none zero status the cluster wont start
I have the following codePreviously I have a delta table with 180 columns in my_path´, I select a column and try to overwrite columns_to_select = ["one_column"] df_one_column = df.select(*columns_to_select) df_one_column.write.format("de...
ok I get the Issue .option("mergeSchema", "true") Is usefull to add more columns, but if you want to reduce columns in your target delta.Then you need.option("overwriteSchema", "true")
Hi team,We are operating on a single-tenant basis with just one metastore. How can other teams use Databricks Unity Catalog without being granted access to our metadata within single tenantRegards,Janga
You can enable/set workspace catalog isolation for each catalog used by your teams. That way, they'll only be able to see their own catalog and corresponding data assets.
Hi,Is it possible to use the Databricks API to create a connection to a different Azure Databricks workspace?Thanks
My apologies for not being cleared in the first post.Terraform can handle it. The API is probably behind and can't handle the connection to Databricks yet. The best solution would probably be to just move to Unity Catalog...Thanks for your quick repl...
Hello! At the moment, I am considering replacing runtime 10.4 with runtime 11.3 in my ADF. I would like to know how big the differences are between them, and if I can just change the version without worrying about breaking anything?
On thing worth mentioning is the DBR Migration Guide. There you'll find all potential changes between the runtimes.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group| User | Count |
|---|---|
| 1611 | |
| 763 | |
| 345 | |
| 286 | |
| 252 |
