Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
Join discussions on data engineering best practices, architectures, and optimization strategies with...
Join discussions on data governance practices, compliance, and security within the Databricks Commun...
Explore discussions on generative artificial intelligence techniques and applications within the Dat...
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...
Hi,I am using the VM family Lasv3, which incorporate a NVMe SSD. I would like to take advantage of this huge amount of space but I cannot find where this disk is mounted. Does someone know where this disk is mounted and if it can be used as local dri...
Great question; I've been trying to hunt that down also. `/local_disk0` looks like a good candidate, but it has restricted access and I can't confirm or use.Would love to learn a solution someday. This is a big need for hybrid workflows & libraries c...
Created a streaming job with delta table as a target. The table did not have a partition when created earlier, however i would like to add an existing column as a partition column.I am getting the following error.com.databricks.sql.transaction.tahoe...
Hi @Anand4,Delta Lake does not support altering the partitioning of an existing table directly. Therefore, the way forward is to rewrite the entire table with the new partition column
The keyboard shortcut to run selected text (ctrl + shift + enter) has not been working for me since yesterday (10/31/24). Instead of running the selected text, databricks notebooks are treating it like shift + enter and running the entire cell. I hav...
I am trying to run the following chunk of code in the cell of a Databricks notebook (using Databricks runtime 14.3 LTS, Apache spark 3.5.0, scala 2.12): spark.sql("CREATE OR REPLACE table sample_catalog.sample_schema.sample_table_tmp AS SELECT * FROM...
Following. Also having this issue, but within the context of pivoting a DF, then aggregating by *
Something very strange has happened. When trying to login to my databricks community edition account. I'm getting the email with my verification code. but after entring that code, I'm getting the error message : "User is not a member of this workspac...
Hi Walter,Am also facing the same issue...am actively using my work space on regular basis...i had already dropped a mail to feedback@databricks.com and help@databricks.com..but didn'tget any reply till now...I want my notebooks back..please suggest ...
I have a git folder to a Bitbucket repo. Whenever I commit something, the commit uses my Bitbucket username (the unique name) in the field author, making it less readable when I'm reading a list of commits.For example, commits end up like this: commi...
Oh, sorry it is internal. If you have an account executive you work with, they can help you with adding the details of the company etc.
Hi there, I ran into a peculiar case and I'm wondering if anyone else has run into this and can offer an explanation. We have a DLT process to pull CSV files from a landing location and insert (append) them into target tables. We have the setting "cl...
Apologies, that could be the internet or networking issue. So, in DLT you will be able to change the DBR but will have to use custom image, it may be tricky if you have not done it earlier. By default, photon will be used in serverelss. It may be a ...
When using an ODBC data source to connect to a Databricks catalog database via Microsoft Access, the tables are not listing/appearing in the MS Access database for selection.However, when using the same ODBC data source to connect to Microsoft Excel,...
The Trace log info is below. Also, I find it interesting that the log is making reference to hive_metastore (see below), which is not the catalog explicitly stated via the Server Side Properties. MSACCESS 3818-164c ENTER SQLAllocEnvHENV * 0x00000...
Hi community,Since recently (2 days ago) we have been receiving the following error when validating and deploying our DAB (Databricks Asset Bundle):"Error: error downloading Terraform: Get "https://releases.hashicorp.com/terraform/1.5.5/index.json": ...
Some update, we cannot get the FW cleared on time so we need to go for the offline optiion, that is download everything form Terraform and DB templated but it is not as clear or intuitive as describe. Using their Container unfortunately not a option ...
package com.example.databricks; import org.apache.spark.sql.Dataset;import org.apache.spark.sql.Row;import org.apache.spark.sql.SparkSession; public class DatabricksJDBCApp { public static void main(String[] args) { // Initialize Spark Ses...
Can you please try it this way: package com.example.databricks; import org.apache.spark.sql.Dataset; import org.apache.spark.sql.Row; import org.apache.spark.sql.SparkSession; public class DatabricksJDBCApp { public static void main(String[] a...
Hi,I have the following databricks notebook code defined: pyspark_dataframe = create_pyspark_dataframe(some input data)MyUDF = udf(myfunc, StringType())pyspark_dataframe = pyspark_dataframe.withColumn('UDFOutput', DownloadUDF(input data columns))outp...
@pjv Can you please try the following, you'll basically want to have more than a single partition: from pyspark.sql import SparkSession from pyspark.sql.functions import udf from pyspark.sql.types import StringType # Initialize Spark session (if not...
Hi,We are trying to pass the keys to decrypt a file and receiving the above error as in attached.Please help in case we need to change and configuration or set any options to avoid this error. Thanks. Vasu
@VasuKumarT can you provide some more details or context? Feel free to replace sensitive data. Where are you getting this? How are you passing the keys to decrypt a file? Is there a move comprehensive stacktrace apart from this message in the image?
Looking into the followinghttps://docs.databricks.com/en/structured-streaming/delta-lake.html#specify-initial-positionI am unclear as to what is the exact difference (if any) between "startingVersion: The Delta Lake version to start from. Databricks ...
The key difference is that not specifying startingVersion will include the current snapshot of the table in the stream, while setting startingVersion to latest will only process new changes from that point onward.
The trials succeed when the cell in the notebook is executed manually:However, the same process fails when executed as a Workflow: The error simply says that there's an issue with the objective function. However how can that be the case if I'm able t...
I've run in to the same issue using SparkTrials.Runs fine manually.Runs using only Trials in the workflow.Get this error when using SparkTrials.I've tried dropping parallelism right down, making sure there's only one experiment on that cluster.Did yo...
We would like to have the ability to assign an individual and/or group to the "Group: Manager" role, providing them with the ability to add/remove users without the need to be an account or workspace administrator. Ideally this would be an option fo...
thanks @NandiniN , we have looked through that documentation and still have not been able to get anything to work without the user also being an account or workspace admin. The way i'm interpreting the documentation (screenshot) is the API currently...
User | Count |
---|---|
1781 | |
834 | |
463 | |
310 | |
297 |