- 2112 Views
- 7 replies
- 1 kudos
Resolved! Unable to resolve column error while trying to query the view
I have a federated table from snowflake in data bricks say employee.When i executed print schema i am able to see schema as "employeeid": long,"employeename":stringTried to create a view as create view vw_emp with schema binding as select `"employeei...
- 2112 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @nkrom456 ,Try something like this. If you are using backticks it treats a column name exactly as you type (in this case it treats double quotes as a part of a colum name)create view vw_emp with schema binding as select `employeeid` from employee ...
- 1 kudos
- 6937 Views
- 6 replies
- 1 kudos
Resolved! Databricks asset bundles job and pipeline
Hello, I'm looking to create a job which is linked to a delta live table. Given the job code like this: my_job_name: name: thejobname schedule: quartz_cron_expression: 56 30 12 * * ? timezone_id: UTC pause_stat...
- 6937 Views
- 6 replies
- 1 kudos
- 1 kudos
This ended a frustrating search! Would be great to add this to the documentation instead of "go to portal and copy paste the id"!!!
- 1 kudos
- 596 Views
- 1 replies
- 2 kudos
Machine type for different operations in Azure Databricks
Dear alldo we have a general recommendation for the virtual machine type to be used for different operations in Azure Databricks? we are looking for the below -1. VACUUM 2. OPTIMIZE 3. ANALYZE STATS 4. DESCRIBE TABLE HISTORYI understood at a high lev...
- 596 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @noorbasha534 ,Here's a general recommendation from Databricks. So they're recommending to run OPTIMIZE on compute optimized VMs and VACUUM on general purpose.Comprehensive Guide to Optimize Data Workloads | DatabricksBut as you said, VACCUM is co...
- 2 kudos
- 28605 Views
- 15 replies
- 3 kudos
Does "databricks bundle deploy" clean up old files?
I'm looking at this page (Databricks Asset Bundles development work tasks) in the Databricks documentation.When repo assets are deployed to a databricks workspace, it is not clear if the "databricks bundle deploy" will remove files from the target wo...
- 28605 Views
- 15 replies
- 3 kudos
- 3 kudos
@JamesGraham this issue is related to "databricks bundle deploy" command itself, when run inside ci/cd pipeline, i am still seeing old configs in bundle.tf.json. Ideally it should be updated to changes done from previous run. But i am still seeing er...
- 3 kudos
- 25533 Views
- 4 replies
- 4 kudos
Resolved! Load Data from Sharepoint Site to Delta table in Databricks
Hi New to the community so sorry if my post lacks detail.I am trying to create a connection between databricks and a sharepoint site to read excel files into a delta tableI can see there is a FiveTran partner connection that we can use to get sharepo...
- 25533 Views
- 4 replies
- 4 kudos
- 4 kudos
@Ajay-Pandey can we connect using user ID without using client id and secrets
- 4 kudos
- 883 Views
- 3 replies
- 0 kudos
"with open" not working in single user access mode cluster (no such file or directory found)
Hi fellow engineers,So i was trying to read binary files (.jpg) in a ADLS2 mounted containerBut when im trying to read the file using "with open" i kept getting an error: No such file or directory foundI've read something related to this matter on So...
- 883 Views
- 3 replies
- 0 kudos
- 0 kudos
@rizkyjarr , did you run into the issue with `with open()` using `\dbfs\mnt` paths , while using a non-Unity catalog enabled workspace despite using the single user access mode cluster as you pointed out?
- 0 kudos
- 8452 Views
- 6 replies
- 1 kudos
SqlContext in DBR 14.3
I have a Databricks workspace in GCP and I am using the cluster with the Runtime 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am trying to set the checkpoint directory location using the following command in a notebook:spark.sparkContext.set...
- 8452 Views
- 6 replies
- 1 kudos
- 1 kudos
Has this been resolved? I'm also encountering the same issue with spark.sparkContext.parallelize(). My code is verifiably running on a single user access cluster.
- 1 kudos
- 951 Views
- 3 replies
- 1 kudos
Resolved! Table of Contents Not Visible in Databricks Notebook
Hi everyone,I'm experiencing a strange issue with one of my Databricks notebooks — the Table of Contents (ToC) pane is no longer visible. It used to show up on the left, but now it’s missing only for this specific notebook.What I’ve observed so far:T...
- 951 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @adhi_databricks.,Good day! Please check for the probable errors: It is a Code cell, not markdown - Change to markdown cell, then add headingIncorrect heading syntax - Try using # Heading or #Heading and then refresh the pageNo headings/titles - A...
- 1 kudos
- 1758 Views
- 5 replies
- 1 kudos
Databricks with Airflow
Hi there, I'm trying to understand the advantages of using Airflow operators to orchestrate Databricks notebooks, given that Databricks already offers its own workflow solution. Could someone please explain the benefits?Thanks,Stefan
- 1758 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @sandelic ,If you workload is mainly Databricks-centered then stick to workflows. They are easy to manage and worfklows directly integrate with Databricks notebooks and jobs.But sometimes your workload requires complex orchestration and scheduling...
- 1 kudos
- 1784 Views
- 2 replies
- 0 kudos
Resolved! Reduce Query Latency
While preparing for Databricks Engineer Associate exam, came across a below question.Question: he data analysis team has reported slow Databricks SQL query performance when using their always-on SQL endpoint, especially during periods of high concurr...
- 1784 Views
- 2 replies
- 0 kudos
- 1611 Views
- 3 replies
- 1 kudos
Resolved! incorrect commit timestamp after deep clone.
i have deep cloned a table, then did update but the update time stamp is less than deep clone timestamp version 0.look like there is an issue in the deep clone. here is the output, _commit_timestamp order is not in sync with _commit_versiontimecard_...
- 1611 Views
- 3 replies
- 1 kudos
- 1 kudos
do we need to do time synchronization in serverless. as we are planning to move from job cluster to serverless and using commit_timestamp for CDF it can cause issues. thankssugun
- 1 kudos
- 1577 Views
- 1 replies
- 1 kudos
Not able to call external API when using Databricks free edition
Hello guys , I'm new to Databricks and trying to create ETL pipelines for my personal projects . I'm using the Databricks free Edition but whenever I try to call an API to an external source to extract Data I always get this error .I have attached th...
- 1577 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @AKB_10 This is a common issue with Databricks Community Edition (free tier). Here's what's happening and how to fix it:Root CauseDatabricks Community Edition has network restrictions that block outbound API calls to external services like Spotify...
- 1 kudos
- 7111 Views
- 11 replies
- 1 kudos
SET configuration in SQL DLT pipeline does not work
Hi,I'm trying to set a dynamic value to use in a DLT query, and the code from the example documentation does not work.SET startDate='2020-01-01'; CREATE OR REFRESH LIVE TABLE filtered AS SELECT * FROM my_table WHERE created_at > ${startDate};It is g...
- 7111 Views
- 11 replies
- 1 kudos
- 1 kudos
@smit_tw Have you tried setting it on the "Advanced" tab as my previous suggests?
- 1 kudos
- 794 Views
- 3 replies
- 0 kudos
Compute and SQL Warehouse Permission button has disappeared
Hi,I can no longer see the permission button for all my all-purpose and SQL WH compute options in any of my workspaces. I've tried everything I can think off, but no joy. I'm an account admin.Has anyone else seen this or experiencing the same probl...
- 794 Views
- 3 replies
- 0 kudos
- 0 kudos
Check to see if you are in the Admins group for each workspace you see this issue.
- 0 kudos
- 1086 Views
- 3 replies
- 0 kudos
Error Creating the Workspace
Hi Team, Good Day!Getting the below error message when creating the Databricks Workspace, Moreover databricks is inside the VPC with private subnets.We already have a separate workspace which is using the same private subnet. Now when i try to create...
- 1086 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @gowthamr-TU ,I don't have much experiance with AWS, but the error message is quite straightforward. Check your inbound rules of the Network ACL (acl-0e9b635f05cdd3f9a) associated with the subnets. It looks like this ACL is blocking inbound traffi...
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 534 | |
| 349 | |
| 287 |
