- 2381 Views
- 0 replies
- 0 kudos
Both `hive_metastore` and `spark_catalog`?
HiIs it possible for a workspace to have both a `hive_metastore` catalog and a `spark_catalog` catalog?
- 2381 Views
- 0 replies
- 0 kudos
Forum Posts
- 3398 Views
- 0 replies
- 0 kudos
New to Spark
Hi all,I am new to Spark, trying to write below code but getting an error.Code:df1 = df.filter(df.col1 > 60 and df.col2 != 'abc') Any suggestion?
- 3398 Views
- 0 replies
- 0 kudos
- 10543 Views
- 3 replies
- 4 kudos
Resolved! Error not a delta table for Unity Catalog table
Is anyone able to advise why I am getting the error not a delta table? The table was created in Unity Catalog. I've also tried DeltaTable.forName and also using 13.3 LTS and 14.3 LTS clusters. Any advice would be much appreciated
- 10543 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
@StogponI believe if you are using DeltaTable.forPath then you have to pass the path where the table is. You can get this path from the Catalog. It is available in the details tab of the table.Example:delta_table_path = "dbfs:/user/hive/warehouse/xyz...
- 4 kudos
- 2197 Views
- 0 replies
- 0 kudos
Best practices for working with external locations where many files arrive constantly
I have an Azure Function that receives files (not volumes) and dumps them to cloud storage. One-five files are received approx. per second. I want to create a partitioned table in Databricks to work with. How should I do this? E.g.: register the cont...
- 2197 Views
- 0 replies
- 0 kudos
- 9676 Views
- 9 replies
- 0 kudos
Performance issue while calling mlflow endpoint
Hi,I have pyspark dataframe and pyspark udf which calls mlflow model for each row but its performance is too slow.Here is sample codedef myfunc(input_text): restult = mlflowmodel.predict(input_text) return resultmyfuncUDF = udf(myfunc,StringType(...
- 9676 Views
- 9 replies
- 0 kudos
Latest Reply
- 3368 Views
- 1 replies
- 0 kudos
Resolved! Understanding Spark Architecture during Table Creation
Team ,I am trying understand how the parquet files and JSON under the delta log folder stores the data behind the scenesTable Creation:from delta.tables import *DeltaTable.create(spark) \.tableName("employee") \.addColumn("id", "INT") \.addColumn("na...
- 3368 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@Ramakrishnan83 - Kindly go through the blog post - https://www.databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html which discuss in detail on delta's transaction log.
- 0 kudos
- 3718 Views
- 0 replies
- 0 kudos
How to use external locations
Hi,I am struggling with truly understanding how to work with external locations. As far as I am able to read, you have:1) Managed catalogs2) Managed schemas3) Managed tables/volumes etc.4) External locations that contains external tables and/or volum...
- 3718 Views
- 0 replies
- 0 kudos
- 9688 Views
- 1 replies
- 0 kudos
Databricks and streamlit and fast API combination
hello friends ! i have project where i need databricks to train eval model then put it to productioni trained model & eval in databricks i used mlflow everything is good now i have another two steps that i have zeroclue how they should be done : usag...
- 9688 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
This repo has examples that you can use in your Databricks workspace for FastAPI and Streamlit. I recommend only using these for development or lightweight use cases.
- 0 kudos
- 3707 Views
- 0 replies
- 0 kudos
can not set permission in table
In databricks database table I was able to set permissions to groups but Now I get this error when using a cluster:Error getting permissionssummary: SparkException: Trying to perform permission action on Hive Metastore /CATALOG/`hive_metastore`/DATAB...
- 3707 Views
- 0 replies
- 0 kudos
- 4044 Views
- 4 replies
- 3 kudos
BAD_REQUEST: ExperimentIds cannot be empty when checking ACLs in bulk
I was going through this tutorial https://mlflow.org/docs/latest/getting-started/tracking-server-overview/index.html#method-2-start-your-own-mlflow-server, I ran the whole script and when I try to open the experiment on the databricks website I get t...
- 4044 Views
- 4 replies
- 3 kudos
Latest Reply
- 1388 Views
- 0 replies
- 0 kudos
Using com.databricks:databricks-jdbc:2.6.36 inside oracle stored proc
Hi dear Databricks community,We tried to use databricks-jdbc inside oracle store procedure to load something from hive. However Oracle marked databricks-jdbc invalid because some classes (for example com.databricks.client.jdbc42.internal.io.netty.ut...
- 1388 Views
- 0 replies
- 0 kudos
- 3054 Views
- 2 replies
- 1 kudos
Databricks Repos
Hi everyone!I've set up an Azure cloud environment for the analytical team that I am part of and everythings is working wonderfully except Databricks Repos. Whenever we open Databricks, we find ourselves in the branch that the most recent person work...
- 3054 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
use a separate a Databricks Git folder mapped to a remote Git repo for each user who works in their own development branch .Run Git operations on Databricks Repos | Databricks on AWS
- 1 kudos
- 4117 Views
- 4 replies
- 1 kudos
Not loading csv files with ".c000.csv" in the name
Yesterday I created a ton of csv files via joined_df.write.partitionBy("PartitionColumn").mode("overwrite").csv( output_path, header=True )Today, when working with them I realized, that they were not loaded. Upon investigation I saw...
- 4117 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
Then removing the "_commited_" file stops spark form reading in the other files
- 1 kudos
- 3478 Views
- 1 replies
- 1 kudos
Resolved! Is it possible to get Azure Databricks cluster metrics using REST API thru pyspark code
Am trying to get azure databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API by writing pyspark code. Its showing always cpu utilization & memory usage as N/A where as data...
- 3478 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @databricksdev You can use System tables for Azure Databricks cluster metrics.Please refer below blog for the same -Compute system tables reference | Databricks on AWS
- 1 kudos
- 4957 Views
- 0 replies
- 0 kudos
TASK_WRITE_FAILED when trying to write on the table, Databricks (Scala)
Hello,I have a code on Databricks (Scala) that constructs a df and then write it to a Database table. It is working fine for almost all of the tables, but there is a table with a problem. It says No module named 'delta.connect' - TASK_WRITE_FAILED.In...
- 4957 Views
- 0 replies
- 0 kudos
