- 15071 Views
- 4 replies
- 3 kudos
Resolved! cluster creation - access mode option
I am a bit lazy and trying to manually recreate a cluster I have in one workspace into another one. The cluster was created some time ago. Looking at the configuration, the access mode field is "custom": When trying to create a new cluster, I do not...
- 15071 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi All - I am new to databricks and trying to create my first workflow. For some reason, the cluster created is of type -"custom". I have not mentioned it anywhere in my asset bundle.Due to this - I cannot create get the Unity Catalog feature. Could ...
- 3 kudos
- 1529 Views
- 1 replies
- 0 kudos
Getting error "Serverless Generic Compute Cluster Not Supported For External Creators."
Getting the above mentioned error while creating serverless compute. this is the request curl --location 'https://adb.azuredatabricks.net/api/2.0/clusters/create' \--header 'Content-Type: application/json' \--header 'Authorization: ••••••' \--data '{...
- 1529 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @tonyd Thank you for reaching out to the Databricks Community. You are trying to create a Serverless Generic Compute Cluster which is not supported. You cannot create a Serverless compute Cluster. As per the below link, if you observe, there is no...
- 0 kudos
- 4089 Views
- 2 replies
- 0 kudos
Unable to set shuffle partitions on DLT pipeline
Hello,We are using a 5 worker node DLT job compute for a continuous mode streaming pipeline. The worker configuration is Standard_D4ads_v5 i.e. 4 cores so total cores across 5 workers is 20 cores.We have wide transformation at some places in the pipe...
- 4089 Views
- 2 replies
- 0 kudos
- 30596 Views
- 6 replies
- 3 kudos
Resolved! How to prevent my cluster to shut down after inactivity
Currently, I am running a cluster that is set to terminate after 60 minutes of inactivity. However, in one of my notebooks, one of the cells is still running. How can I prevent this from happening, if want my notebook to run overnight without monito...
- 30596 Views
- 6 replies
- 3 kudos
- 3 kudos
If a cell is already running ( I assume it's a streaming operation), then I think it doesn't mean that the cluster is inactive. The cluster should be running if a cell is running on it.On the other hand, if you want to keep running your clusters for ...
- 3 kudos
- 4209 Views
- 1 replies
- 0 kudos
How to call azure databricks api from azure api management
Hi,Has anyone successfully configure azure apim to access databricks rest api ? If yes, appreciate he can provide the setup guide for me as I am stuck at this point. Thanks.
- 4209 Views
- 1 replies
- 0 kudos
- 0 kudos
@Check Did you manage to complete the mosaic gateway URL in the Azure APIM .@Retired_mod Do we have any other generic link like databricks document which we can refer .Thanks
- 0 kudos
- 2826 Views
- 2 replies
- 0 kudos
how to git integration multiple gitlab repo
I have 3 GitLab repos in my Databricks workspace.I have also generated personal tokens for these 3 repos. However, it seems that Databricks can only use one repo token at a time for Git integration.For example: I am currently using the token for repo...
- 2826 Views
- 2 replies
- 0 kudos
- 0 kudos
Unfortunately this is expected behavior. We only support 1 Git credential at a time per user in the workspace. We are adding a sample notebook in this section to help you swap Git credentials more easily. https://docs.databricks.com/en/repos/repos-s...
- 0 kudos
- 1524 Views
- 1 replies
- 1 kudos
Connecting to redshift from databricks notebooks without password using IAM
Hello all,Have you ever tried to connect to redshift from databricks notebooks without password using IAM. Pradeesh M P
- 1524 Views
- 1 replies
- 1 kudos
- 1 kudos
you should create an IAM assumed role, adding the principal as the aws account that hosts your databricks env.
- 1 kudos
- 2284 Views
- 3 replies
- 0 kudos
Resolved! fs ls lists files that i cannot navigate to an view
Hi Community - I have an issue and I did not find any effective solution so hoping someone can help here. When I use %fs ls "dbfs:/Workspace/Shared/" I see 2 folders but when i navigate to the folder I only see 1. Can someone help me with this issue....
- 2284 Views
- 3 replies
- 0 kudos
- 0 kudos
Possible UI bug. Submit a Case with the Engineering team. https://help.databricks.com/s/ (Right top SUBMIT CASE)
- 0 kudos
- 1588 Views
- 1 replies
- 0 kudos
org.rocksdb.RocksDBException: No space left on device
Structured stream is failing intermittently with following message. org.rocksdb.RocksDBException: No space left on device .Do we have a settings on databricks to assigned limited diskpace to the checkpoint tracking?Appreciate any help with resolving ...
- 1588 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you share details like the DBR version, and are you using anyForEachBatch?
- 0 kudos
- 29089 Views
- 9 replies
- 7 kudos
Resolved! dropping a managed table does not remove the underlying files
the documentation states that "drop table":Deletes the table and removes the directory associated with the table from the file system if the table is not EXTERNAL table. An exception is thrown if the table does not exist.In case of an external table...
- 29089 Views
- 9 replies
- 7 kudos
- 7 kudos
Hi,There is a way to force delete files after drop the table and don't wait 30 days to see size in S3 decrease?Tables that I dropped related to the dev and staging, I don't want to keep there files for 30 days
- 7 kudos
- 20526 Views
- 4 replies
- 1 kudos
Resolved! custom upsert for delta live tables apply_changes()
Hello community :).I am currently implementing some pipelines using DLT. They are working great for my medalion architecture for landed json in bronze -> silver (using apply_changes) then materialized gold views ontop.However, I am attempting to crea...
- 20526 Views
- 4 replies
- 1 kudos
- 1 kudos
Is it possible to have custom upserts for streaming tables in delta live tables?Im getting the error:pyspark.errors.exceptions.captured.AnalysisException: `blusmart_poc.information_schema.sessions` is not a Delta table.
- 1 kudos
- 7095 Views
- 2 replies
- 1 kudos
Error while Resetting my Community Edition Password
I recently tried to create my account with Databricks Community Edition. I have singed up for it and received verification email. After that I have to reset my password. But while doing so I am always getting the following error. Can someone help me ...
- 7095 Views
- 2 replies
- 1 kudos
- 1 kudos
Receiving the same error when creating a new account - "An error has occurred. Please try again later."
- 1 kudos
- 3338 Views
- 1 replies
- 0 kudos
PySparkRuntimeError: [CONTEXT_ONLY_VALID_ON_DRIVER] It appears that you are attempting to reference
Getting The above error for this lineresult_df.rdd.foreachPartition(self.process_partition)
- 3338 Views
- 1 replies
- 0 kudos
- 0 kudos
The error message "CONTEXT_ONLY_VALID_ON_DRIVER" indicates that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that runs on workers. This is ...
- 0 kudos
- 1680 Views
- 2 replies
- 0 kudos
Resolved! Unable to find data engineer learning path database file
I'm unable to find the database file to download for learning
- 1680 Views
- 2 replies
- 0 kudos
- 11802 Views
- 4 replies
- 0 kudos
Reading JSON from Databricks Workspace
I am using second example from Databricks` official document here: Work with workspace files. But I'm getting following error:Question: What could be a cause of the error, and how can we fix it?ERROR: Since Spark 2.3, the queries from raw JSON/CSV fi...
- 11802 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Saf4Databricks ,As you said, you probably need to add multiline options to make it work. You can use this option when creating temporary view or using pyspark api. Below is example of creating temporary view: CREATE TEMPORARY VIEW multilineJson U...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
