- 1226 Views
- 3 replies
- 1 kudos
[Memory utilization in Metrics Tab still display after terminate a cluster]
Hi All,Could you guys help me to check this?I run a cluster and then terminate that cluster but when i navigate to the Metrics tab of Cluster still see the Memory utilization show metrics.Thanks
- 1226 Views
- 3 replies
- 1 kudos
- 1587 Views
- 4 replies
- 0 kudos
databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace
databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace. The UI lists `hive_metastore`. It would be better if these two were consistent.
- 1587 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @HakanNordgren, Let’s address the inconsistency between spark_catalog and hive_metastore in Databricks. Here’s what we know: spark_catalog: This catalog is associated with Databricks and is used for managing metadata related to tables, views, ...
- 0 kudos
- 1658 Views
- 3 replies
- 1 kudos
an autoloader in file notification mode to get files from S3 on AWS -Error
I configured an autoloader in file notification mode to get files from S3 on AWS.spark.readStream\.format("cloudFiles")\.option("cloudFiles.format", "json")\.option("cloudFiles.inferColumnTypes", "true")\.option("cloudFiles.schemaLocation", "dbfs:/au...
- 1658 Views
- 3 replies
- 1 kudos
- 1 kudos
In case anyone else stumbles across this, I was able to fix my issue by setting up an instance profile with the file notification permissions and attaching the instance profile to the job cluster. It wasn't clear from the documentation that the file ...
- 1 kudos
- 1165 Views
- 1 replies
- 0 kudos
DataBricks Certification Exam Got Suspended. Require support for the same.
I encountered numerous challenges during my exam, starting with issues related to system compatibility and difficulties with my microphone and other settings. Despite attempting to contact support multiple times, it was not easy to get assistance.Aft...
- 1165 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Shaghil , Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).
- 0 kudos
- 1043 Views
- 1 replies
- 0 kudos
Not able to download Certificate
Hi All,I took the course: Get Started With Data Engineering from below course link https://www.databricks.com/learn/training/getting-started-with-data-engineering#data-videoBut, after completing the Quiz, I am not able to download Certificate. The a...
- 1043 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AbhilashMV, Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours...
- 0 kudos
- 1301 Views
- 1 replies
- 0 kudos
Unable to login to Databricks Community edition
I signed up to Databricks Community Edition with gmail and verified my account and used it to create a notebook as well but having issue in re-logging in even though email and password provided are correct. I did try "Forget Password" step but I am...
- 1301 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Godhuli., Please look at this link related to the Community - Edition, which might solve your problem. I appreciate your interest in sharing your Community-Edition query with us. If you have any more questions or concerns, please don't hesita...
- 0 kudos
- 1823 Views
- 1 replies
- 1 kudos
Resolved! How are Struct type columns stored/accessed (interested in efficiency)?
Hello, I've searched around for awhile and didn't find a similar question here or elsewhere, so thought I'd ask...I'm assessing the storage/access efficiency of Struct type columns in delta tables. I want to know more about how Databricks is storing...
- 1823 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @crowley, Let’s delve into the storage and access efficiency of Struct type columns in Delta tables within the context of Databricks. Structured Data Sources and Efficiency: Structured data sources, such as Parquet and ORC, define a schema on ...
- 1 kudos
- 1591 Views
- 2 replies
- 0 kudos
Resolved! ArcGIS Connection
Hi,I am trying to connect to an ArcGIS instance using Data bricks. Is this possible? After connecting, I am trying to read the data into a Data fame.Please help me with this request. If its not possible to connect , please provide an alternative.Than...
- 1591 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @dbsuersu , Connecting to an ArcGIS instance using Databricks is indeed possible, and I’ll guide you through the process. ArcGIS GeoAnalytics Engine in Databricks: ArcGIS GeoAnalytics Engine (GA Engine) is a powerful plugin for Apache Spark™ t...
- 0 kudos
- 1058 Views
- 1 replies
- 0 kudos
Best practices for working with external locations where many files arrive constantly
I have an Azure Function that receives files (not volumes) and dumps them to cloud storage. One-five files are received approx. per second. I want to create a partitioned table in Databricks to work with. How should I do this? E.g.: register the cont...
- 1058 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pernilak, Since you’re dealing with a high volume of files arriving approximately every second, creating a partitioned table is a good idea. Partitioning helps optimize query performance and manage large datasets efficiently. Here’s how you ca...
- 0 kudos
- 2617 Views
- 1 replies
- 0 kudos
Resolved! New to Spark
Hi all,I am new to Spark, trying to write below code but getting an error.Code:df1 = df.filter(df.col1 > 60 and df.col2 != 'abc') Any suggestion?
- 2617 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Anku_ , use this instead :- df1 = df.filter((df.col1 > 60) & (df.col2 != 'abc'))
- 0 kudos
- 1786 Views
- 1 replies
- 0 kudos
How to use external locations
Hi,I am struggling with truly understanding how to work with external locations. As far as I am able to read, you have:1) Managed catalogs2) Managed schemas3) Managed tables/volumes etc.4) External locations that contains external tables and/or volum...
- 1786 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pernilak, Please refer to the official Databricks documentation on external locations. Let’s say you have an external location for financial data stored in an S3 bucket (s3://depts/finance). Here’s how you can set it up: -- Grant `finance` user ...
- 0 kudos
- 4679 Views
- 12 replies
- 3 kudos
Performance issue while calling mlflow endpoint
Hi,I have pyspark dataframe and pyspark udf which calls mlflow model for each row but its performance is too slow.Here is sample codedef myfunc(input_text): restult = mlflowmodel.predict(input_text) return resultmyfuncUDF = udf(myfunc,StringType(...
- 4679 Views
- 12 replies
- 3 kudos
- 2087 Views
- 1 replies
- 0 kudos
Resolved! Understanding Spark Architecture during Table Creation
Team ,I am trying understand how the parquet files and JSON under the delta log folder stores the data behind the scenesTable Creation:from delta.tables import *DeltaTable.create(spark) \.tableName("employee") \.addColumn("id", "INT") \.addColumn("na...
- 2087 Views
- 1 replies
- 0 kudos
- 0 kudos
@Ramakrishnan83 - Kindly go through the blog post - https://www.databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html which discuss in detail on delta's transaction log.
- 0 kudos
- 1790 Views
- 2 replies
- 1 kudos
Corrupted Python installation on Python restart on DBR 13.3
Hey there, we're using DBR 13.3 (no Docker) as general purpose cluster and init the cluster using the following init script:```#!/usr/bin/env bashexport DEBIAN_FRONTEND=noninteractiveset -euxo pipefailif [[ $DB_IS_DRIVER = "TRUE" ]]; thenecho "I am d...
- 1790 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ivanychev , Let me get some of our experts here at Databricks to answer your question. Please bear with us until then.
- 1 kudos
- 2273 Views
- 1 replies
- 0 kudos
Resolved! can not set permission in table
In databricks database table I was able to set permissions to groups but Now I get this error when using a cluster:Error getting permissionssummary: SparkException: Trying to perform permission action on Hive Metastore /CATALOG/`hive_metastore`/DATAB...
- 2273 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @arkiboys, It seems you’re encountering an issue related to permissions and table access control in Databricks. Let’s troubleshoot this together. Table Access Control Not Enabled: The error message indicates that Table Access Control is not en...
- 0 kudos
