- 7024 Views
- 5 replies
- 2 kudos
Job cluster support in jobs/runs/submit API
We are using jobs/runs/submit API of databricks to create and trigger a one-time run with new_cluster and existing_cluster configuration. We would like to check if there is provision to pass "job_clusters" in this API to reuse the same cluster across...
- 7024 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi, Any update on the above mentioned issue? Unable to submit a one time new job run (api/2.0 or 21/jobs/runs/submit) with shared job cluster or one new cluster has to be used for all TASKs in the job
- 2 kudos
- 2334 Views
- 1 replies
- 1 kudos
Resolved! schema is not enforced when using autoloader
Hi everyone,I am currently trying to enforce the following schema: StructType([ StructField("site", StringType(), True), StructField("meter", StringType(), True), StructField("device_time", StringType(), True), StructField("data", St...
- 2334 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @sakuraDev ,I'm afraid your assumption is wrong. Here you define data field as struct type and the result is as expected. So once you have this column as struct type, you can refer to nested object using dot notation. So if you would like to get e...
- 1 kudos
- 2563 Views
- 2 replies
- 0 kudos
Info on Databricks AWS High Availability during zone selection
Hi Team,During the zone selection in the Databricks environment, there is an option for High Availability (HA), which selects instances from other zones to ensure prolonged uptimes.My question is: Does the HA option only select instances from other a...
- 2563 Views
- 2 replies
- 0 kudos
- 2239 Views
- 4 replies
- 0 kudos
Data Ingestion from GCP
Hi,I'm ingesting data from GCP to Databricks, and I think I've noticed a bug in that any datatables that have a numerical starting character are not ingested at all.Has anyone else experienced this?Please let me know if there is a way around this apa...
- 2239 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Slash, thanks for getting back to meSo for instance - I have data tables such as "20240901_demographics_data_v1" which I'm trying to move from BQOther data tables that don't include a date (or other numerical characters) at the front are being ing...
- 0 kudos
- 7690 Views
- 4 replies
- 1 kudos
Resolved! Not able to set run_as service_principal_name
I'm trying to run: databricks bundle deploy -t prod --profile PROD_Service_Principal My bundle looks: bundle: name: myproject include: - resources/jobs/bundles/*.yml targets: # The 'dev' target, for development purposes. This target is the de...
- 7690 Views
- 4 replies
- 1 kudos
- 1 kudos
Building on this situation, I am seeing if I deploy a job using a service principal this way, I am getting something prepended to the job like `[dev f46583c2_8c9e_499f_8d41_823332bfd4473] `. Is there a different way for me via bundling to change this...
- 1 kudos
- 3645 Views
- 3 replies
- 1 kudos
Data engineering professional exam
Each configuration below is identical in that each cluster has 400 GB total of RAM, 160 total cores, and only one Executor per VM.Given an extremely long-running job for which completion must be guaranteed, which cluster configuration will be able to...
- 3645 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @anshi_t_k ,The key consideration here is fault tolerance. How do you protect against a VM failure? By having more VMs, as the impact of a single VM failure will be the lowest.For example answer C - the crash of the VM is loosing 1/1 so 100% capa...
- 1 kudos
- 6543 Views
- 1 replies
- 0 kudos
Databricks CI/CD Azure Devops
Hi all,I am looking for advice on what would be the best approach when it comes to CI/CD in Databricks and repo in general. What would be the best approach; to have main branch and branch off of it or? How will changes be propagated from dev to qa an...
- 6543 Views
- 1 replies
- 0 kudos
- 5665 Views
- 3 replies
- 2 kudos
Resolved! Sub-Query behavior in sql statements
Hi Team,I have a query with below construct in my projectSELECT count(*) FROM `catalog`.`schema`.`t_table`WHERE _col_check IN (SELECT DISTINCT _col_check FROM `catalog`.`schema`.`t_check_table`)Actually, there is no column "_col_check" in the sub-que...
- 5665 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Harsha777 ,What occurs is called column shadowing.What happens is that the column names in main query and sub query are identica and the databricks engine after not finding it in sub query searches in the main query.The simplest way to avoid the...
- 2 kudos
- 2040 Views
- 2 replies
- 0 kudos
DLT-META requires ddl when using cdc_apply_changes
We are setting up new DLT Pipelines using the DLT-Meta package. Everything is going well in bringing our data in from Landing to our Bronze layer when we keep the onboarding JSON fairly vanilla. However, we are hitting an issue when using the cdc_app...
- 2040 Views
- 2 replies
- 0 kudos
- 0 kudos
Please check these details: https://github.com/databrickslabs/dlt-meta/issues/90
- 0 kudos
- 1676 Views
- 1 replies
- 0 kudos
Unity Catalog: Metastore 3 level Hierarchy
I have data files categorized by application and region. Want to know the best way to load them into the Bronze and Silver layers while maintaining proper segregation.For example, in our landing zone, we have a structure of raw files to be loaded usi...
- 1676 Views
- 1 replies
- 0 kudos
- 0 kudos
If I understand it correctly, you have source files partitioned by application and region in cloud storage that you want to load and would like some suggestions on the Unity Catalog structure. It will definitely depend on how you want the data to be ...
- 0 kudos
- 3766 Views
- 1 replies
- 0 kudos
Where is Data tab and DBFS in Premium Databricks workspace
Currently I can see Catalog tab instead of Data tab in left side navigation. I am unable to find Data tab -> File browser where I would like to upload one sample orders csv file. Later I want to refer that path in Databricks notebooks as /FileStore/t...
- 3766 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sudheer89 ,By default - DBFS tab is disabled. As an admin user, you can manage your users’ ability to browse data in the Databricks File System (DBFS) using the visual browser interface.Go to the admin console.Click the Workspace Settings tab.In ...
- 0 kudos
- 5679 Views
- 1 replies
- 1 kudos
Enforce tags to Jobs
Anyone know how I enforce jobs tags, not the custom tags for cluster. I want to enforce that jobs has certain tags so we can filter our jobs. We are not using Unity Catalog yet.
- 5679 Views
- 1 replies
- 1 kudos
- 1 kudos
Currently, enforcing job tags is not a built-in feature in Databricks. However, you can add tags to your jobs when creating or updating them and filter jobs by these tags on the jobs list page.
- 1 kudos
- 4106 Views
- 1 replies
- 2 kudos
Constructor public org.apache.spark.ml.feature.Bucketizer(java.lang.String) is not whitelisted.
Hi,I am getting the error Constructor public org.apache.spark.ml.feature.Bucketizer(java.lang.String) is not whitelisted. when using a serverless compute cluster. I have seen in some other articles that this is due to high concurrency - does anyone k...
- 4106 Views
- 1 replies
- 2 kudos
- 2 kudos
The error you're encountering, "Constructor public org.apache.spark.ml.feature.Bucketizer(java.lang.String) is not whitelisted", typically arises when using a shared mode cluster. This is because Spark ML is not supported in shared clusters due to se...
- 2 kudos
- 1547 Views
- 1 replies
- 0 kudos
Need Help with SCIM Provisioning URL and Automation
Hi Databricks Community,I’m working on setting up SCIM provisioning and need some assistance:SCIM Provisioning URL:Can anyone confirm the correct process to obtain the SCIM Provisioning URL from the Databricks account console? I need to ensure I'm re...
- 1547 Views
- 1 replies
- 0 kudos
- 0 kudos
Which provider are you using? You can use the doc for Okta provisioning to guide you through the process https://docs.databricks.com/en/admin/users-groups/scim/okta.html
- 0 kudos
- 7062 Views
- 2 replies
- 3 kudos
How to disable users to create personal compute using notebook?
A Databricks account administrator can disable account-wide access to the Personal Compute default policy using the following steps: Navigate to the Databricks Account Console. Click the “Settings” icon. Click the “Feature enablement” tab. Switch the...
- 7062 Views
- 2 replies
- 3 kudos
- 3 kudos
Is there no way to prevent using Personal Compute policy from a notebook?Or does my question make sense? In other words is it by design right/immutable to have this policy when creating a notebook?
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
