- 898 Views
- 3 replies
- 0 kudos
Resolved! java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMeta
Guys please help me too solve this issue, I need it very urgent basis.
- 898 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @balwantsingh24 Internal Metastore:- Internal metastores are managed by Databricks and are typically used to store metadata about databases, tables, views, and user-defined functions (UDFs). This metadata is essential for operations like the SHOW...
- 0 kudos
- 618 Views
- 4 replies
- 0 kudos
Delta live tables multiple .csv diff schemas
Hi all, I have a fairly straight-forward task whereby I am looking to ingest six .csv file all with different names, schema's and blob locations into individual tables on one bronze schema. I have the files in my landing zone under different fol...
- 618 Views
- 4 replies
- 0 kudos
- 0 kudos
The code follows similar pattern below to load the different tables. import dltimport reimport pyspark.sql.functions as Flanding_zone = '/Volumes/bronze_dev/landing_zone/'source = 'addresses'@Dlt.table(comment="addresses snapshot",name="addresses")de...
- 0 kudos
- 7808 Views
- 4 replies
- 3 kudos
Resolved! cluster creation - access mode option
I am a bit lazy and trying to manually recreate a cluster I have in one workspace into another one. The cluster was created some time ago. Looking at the configuration, the access mode field is "custom": When trying to create a new cluster, I do not...
- 7808 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi All - I am new to databricks and trying to create my first workflow. For some reason, the cluster created is of type -"custom". I have not mentioned it anywhere in my asset bundle.Due to this - I cannot create get the Unity Catalog feature. Could ...
- 3 kudos
- 299 Views
- 1 replies
- 0 kudos
Getting error "Serverless Generic Compute Cluster Not Supported For External Creators."
Getting the above mentioned error while creating serverless compute. this is the request curl --location 'https://adb.azuredatabricks.net/api/2.0/clusters/create' \--header 'Content-Type: application/json' \--header 'Authorization: ••••••' \--data '{...
- 299 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @tonyd Thank you for reaching out to the Databricks Community. You are trying to create a Serverless Generic Compute Cluster which is not supported. You cannot create a Serverless compute Cluster. As per the below link, if you observe, there is no...
- 0 kudos
- 508 Views
- 2 replies
- 0 kudos
Unable to set shuffle partitions on DLT pipeline
Hello,We are using a 5 worker node DLT job compute for a continuous mode streaming pipeline. The worker configuration is Standard_D4ads_v5 i.e. 4 cores so total cores across 5 workers is 20 cores.We have wide transformation at some places in the pipe...
- 508 Views
- 2 replies
- 0 kudos
- 17984 Views
- 6 replies
- 3 kudos
Resolved! How to prevent my cluster to shut down after inactivity
Currently, I am running a cluster that is set to terminate after 60 minutes of inactivity. However, in one of my notebooks, one of the cells is still running. How can I prevent this from happening, if want my notebook to run overnight without monito...
- 17984 Views
- 6 replies
- 3 kudos
- 3 kudos
If a cell is already running ( I assume it's a streaming operation), then I think it doesn't mean that the cluster is inactive. The cluster should be running if a cell is running on it.On the other hand, if you want to keep running your clusters for ...
- 3 kudos
- 2098 Views
- 1 replies
- 0 kudos
How to call azure databricks api from azure api management
Hi,Has anyone successfully configure azure apim to access databricks rest api ? If yes, appreciate he can provide the setup guide for me as I am stuck at this point. Thanks.
- 2098 Views
- 1 replies
- 0 kudos
- 0 kudos
@Check Did you manage to complete the mosaic gateway URL in the Azure APIM .@Retired_mod Do we have any other generic link like databricks document which we can refer .Thanks
- 0 kudos
- 611 Views
- 2 replies
- 0 kudos
how to git integration multiple gitlab repo
I have 3 GitLab repos in my Databricks workspace.I have also generated personal tokens for these 3 repos. However, it seems that Databricks can only use one repo token at a time for Git integration.For example: I am currently using the token for repo...
- 611 Views
- 2 replies
- 0 kudos
- 0 kudos
Unfortunately this is expected behavior. We only support 1 Git credential at a time per user in the workspace. We are adding a sample notebook in this section to help you swap Git credentials more easily. https://docs.databricks.com/en/repos/repos-s...
- 0 kudos
- 831 Views
- 1 replies
- 0 kudos
Errors using Databricks Extension for VS Code on Windows
Hi - I am trying to get my VS Code (running on Windows) to work with the Databricks extension for VS Code. It seems like I can almost get this to work. Here is my setup:1. Using Databricks Extension v2.4.02. Connecting to Databricks cluster with ru...
- 831 Views
- 1 replies
- 0 kudos
- 0 kudos
So I found my problem(s). I had a local environment variable called "DATABRICKS_HOST" that was set to the wrong URL.My Databricks runtime version and the databricks-connect version were not the same. When I made them both 15.4.x, everything works a...
- 0 kudos
- 527 Views
- 2 replies
- 0 kudos
Resolved! Serverless Compute no support for Caching data frames
Can anyone please tell me why df.cache() and df.persist() are not supported in Serevrless compute?Many Thanks
- 527 Views
- 2 replies
- 0 kudos
- 0 kudos
Global caching functionality (and other global states used on classic clusters) is conceptually hard to represent on serverless computing. Serverless spark cluster optimizes the cache than the user.
- 0 kudos
- 370 Views
- 1 replies
- 1 kudos
Connecting to redshift from databricks notebooks without password using IAM
Hello all,Have you ever tried to connect to redshift from databricks notebooks without password using IAM. Pradeesh M P
- 370 Views
- 1 replies
- 1 kudos
- 1 kudos
you should create an IAM assumed role, adding the principal as the aws account that hosts your databricks env.
- 1 kudos
- 536 Views
- 3 replies
- 0 kudos
Resolved! fs ls lists files that i cannot navigate to an view
Hi Community - I have an issue and I did not find any effective solution so hoping someone can help here. When I use %fs ls "dbfs:/Workspace/Shared/" I see 2 folders but when i navigate to the folder I only see 1. Can someone help me with this issue....
- 536 Views
- 3 replies
- 0 kudos
- 0 kudos
Possible UI bug. Submit a Case with the Engineering team. https://help.databricks.com/s/ (Right top SUBMIT CASE)
- 0 kudos
- 298 Views
- 1 replies
- 0 kudos
org.rocksdb.RocksDBException: No space left on device
Structured stream is failing intermittently with following message. org.rocksdb.RocksDBException: No space left on device .Do we have a settings on databricks to assigned limited diskpace to the checkpoint tracking?Appreciate any help with resolving ...
- 298 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you share details like the DBR version, and are you using anyForEachBatch?
- 0 kudos
- 13888 Views
- 9 replies
- 6 kudos
Resolved! dropping a managed table does not remove the underlying files
the documentation states that "drop table":Deletes the table and removes the directory associated with the table from the file system if the table is not EXTERNAL table. An exception is thrown if the table does not exist.In case of an external table...
- 13888 Views
- 9 replies
- 6 kudos
- 6 kudos
Hi,There is a way to force delete files after drop the table and don't wait 30 days to see size in S3 decrease?Tables that I dropped related to the dev and staging, I don't want to keep there files for 30 days
- 6 kudos
- 1359 Views
- 2 replies
- 2 kudos
DLT Direct Publishing Mode
Hello,I'm working on a DLT pipeline and have a block of SQL that runs...USE CATALOG catalog_a; USE SCHEMA schema_a; CREATE OR REFRESH MATERIALIZED VIEW table_a AS SELECT ... FROM catalog_b.schema_b.table_b;Executing this block returns the following.....
- 1359 Views
- 2 replies
- 2 kudos
- 2 kudos
Ah, thank you! I couldn't find anything about Direct Publishing Mode through my Google search (in fact, if you search it now, this post is the top result).I believe I'll be waiting until it goes into public preview/GA. Removing USE CATALOG and USE SC...
- 2 kudos