- 828 Views
- 2 replies
- 0 kudos
Single task job that runs SQL notebook, can't retrieve results
Hello,We are integrating databricks and I need to run a job with single task that will run notebok with SQL query in it. I Can only use SQL warehouse and no cluster, I need to retrieve a result of the the notebook task but I can't see the results. Is...
- 828 Views
- 2 replies
- 0 kudos
- 0 kudos
> I need to retrieve a result of the the notebook taskIf you want to know if the task run has succeeded or not, you can enable the "lakeflow" system schema and you'll find the logs of jobs and task runs.You could then use the above info to execute a...
- 0 kudos
- 547 Views
- 1 replies
- 0 kudos
AWS Role of NACL outbound 3306 port
When using databricks for AWSI need to open 3306 to the NACL outbound port of the subnet where the endpoint is locatedI understand this is to communicate with the meta store of Databricks on the instanceAm I right to understand?If not, please let me ...
- 547 Views
- 1 replies
- 0 kudos
- 0 kudos
You are correct, the intention of this port is to connect to the Hive metastore
- 0 kudos
- 3887 Views
- 1 replies
- 0 kudos
How databricks assign memory and cores
Hi team,We are using job cluster with node type 128G memory+16cores for a workflow. From document we know one worker is one node and is one executor. From Spark UI env tab we can see the spark.executor.memory is 24G, and from metrics we can see the m...
- 3887 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks allocates resources to executors on a node based on several factors, and it appears that your cluster configuration is using default settings since no specific Spark configurations were provided. Executor Memory Allocation: The spark.exec...
- 0 kudos
- 1331 Views
- 1 replies
- 0 kudos
Asset Bundles from Workspace for CI/CD
Hello there,I am exploring the possibilities for CI/CD from a DEV-Workspace to PROD. Besides the Notebooks (which can easily be handled by the GIT-provider) I am mainly interested in the Deployment o Jobs/Clusters/DDl...I can nowhere find a tutorial ...
- 1331 Views
- 1 replies
- 0 kudos
- 0 kudos
i think the dab mlop stack template is pretty helpful re: how to bundle, schedule and trigger custom jobshttps://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.htmlyou can bundle init it locally and it should give you the skeleton of how to bu...
- 0 kudos
- 2911 Views
- 3 replies
- 0 kudos
Resolved! java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMeta
Guys please help me too solve this issue, I need it very urgent basis.
- 2911 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @balwantsingh24 Internal Metastore:- Internal metastores are managed by Databricks and are typically used to store metadata about databases, tables, views, and user-defined functions (UDFs). This metadata is essential for operations like the SHOW...
- 0 kudos
- 1216 Views
- 4 replies
- 0 kudos
Delta live tables multiple .csv diff schemas
Hi all, I have a fairly straight-forward task whereby I am looking to ingest six .csv file all with different names, schema's and blob locations into individual tables on one bronze schema. I have the files in my landing zone under different fol...
- 1216 Views
- 4 replies
- 0 kudos
- 0 kudos
The code follows similar pattern below to load the different tables. import dltimport reimport pyspark.sql.functions as Flanding_zone = '/Volumes/bronze_dev/landing_zone/'source = 'addresses'@Dlt.table(comment="addresses snapshot",name="addresses")de...
- 0 kudos
- 10722 Views
- 4 replies
- 3 kudos
Resolved! cluster creation - access mode option
I am a bit lazy and trying to manually recreate a cluster I have in one workspace into another one. The cluster was created some time ago. Looking at the configuration, the access mode field is "custom": When trying to create a new cluster, I do not...
- 10722 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi All - I am new to databricks and trying to create my first workflow. For some reason, the cluster created is of type -"custom". I have not mentioned it anywhere in my asset bundle.Due to this - I cannot create get the Unity Catalog feature. Could ...
- 3 kudos
- 685 Views
- 1 replies
- 0 kudos
Getting error "Serverless Generic Compute Cluster Not Supported For External Creators."
Getting the above mentioned error while creating serverless compute. this is the request curl --location 'https://adb.azuredatabricks.net/api/2.0/clusters/create' \--header 'Content-Type: application/json' \--header 'Authorization: ••••••' \--data '{...
- 685 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @tonyd Thank you for reaching out to the Databricks Community. You are trying to create a Serverless Generic Compute Cluster which is not supported. You cannot create a Serverless compute Cluster. As per the below link, if you observe, there is no...
- 0 kudos
- 1217 Views
- 2 replies
- 0 kudos
Unable to set shuffle partitions on DLT pipeline
Hello,We are using a 5 worker node DLT job compute for a continuous mode streaming pipeline. The worker configuration is Standard_D4ads_v5 i.e. 4 cores so total cores across 5 workers is 20 cores.We have wide transformation at some places in the pipe...
- 1217 Views
- 2 replies
- 0 kudos
- 23112 Views
- 6 replies
- 3 kudos
Resolved! How to prevent my cluster to shut down after inactivity
Currently, I am running a cluster that is set to terminate after 60 minutes of inactivity. However, in one of my notebooks, one of the cells is still running. How can I prevent this from happening, if want my notebook to run overnight without monito...
- 23112 Views
- 6 replies
- 3 kudos
- 3 kudos
If a cell is already running ( I assume it's a streaming operation), then I think it doesn't mean that the cluster is inactive. The cluster should be running if a cell is running on it.On the other hand, if you want to keep running your clusters for ...
- 3 kudos
- 3050 Views
- 1 replies
- 0 kudos
How to call azure databricks api from azure api management
Hi,Has anyone successfully configure azure apim to access databricks rest api ? If yes, appreciate he can provide the setup guide for me as I am stuck at this point. Thanks.
- 3050 Views
- 1 replies
- 0 kudos
- 0 kudos
@Check Did you manage to complete the mosaic gateway URL in the Azure APIM .@Retired_mod Do we have any other generic link like databricks document which we can refer .Thanks
- 0 kudos
- 1408 Views
- 2 replies
- 0 kudos
how to git integration multiple gitlab repo
I have 3 GitLab repos in my Databricks workspace.I have also generated personal tokens for these 3 repos. However, it seems that Databricks can only use one repo token at a time for Git integration.For example: I am currently using the token for repo...
- 1408 Views
- 2 replies
- 0 kudos
- 0 kudos
Unfortunately this is expected behavior. We only support 1 Git credential at a time per user in the workspace. We are adding a sample notebook in this section to help you swap Git credentials more easily. https://docs.databricks.com/en/repos/repos-s...
- 0 kudos
- 3181 Views
- 1 replies
- 0 kudos
Errors using Databricks Extension for VS Code on Windows
Hi - I am trying to get my VS Code (running on Windows) to work with the Databricks extension for VS Code. It seems like I can almost get this to work. Here is my setup:1. Using Databricks Extension v2.4.02. Connecting to Databricks cluster with ru...
- 3181 Views
- 1 replies
- 0 kudos
- 0 kudos
So I found my problem(s). I had a local environment variable called "DATABRICKS_HOST" that was set to the wrong URL.My Databricks runtime version and the databricks-connect version were not the same. When I made them both 15.4.x, everything works a...
- 0 kudos
- 684 Views
- 1 replies
- 1 kudos
Connecting to redshift from databricks notebooks without password using IAM
Hello all,Have you ever tried to connect to redshift from databricks notebooks without password using IAM. Pradeesh M P
- 684 Views
- 1 replies
- 1 kudos
- 1 kudos
you should create an IAM assumed role, adding the principal as the aws account that hosts your databricks env.
- 1 kudos
- 1085 Views
- 3 replies
- 0 kudos
Resolved! fs ls lists files that i cannot navigate to an view
Hi Community - I have an issue and I did not find any effective solution so hoping someone can help here. When I use %fs ls "dbfs:/Workspace/Shared/" I see 2 folders but when i navigate to the folder I only see 1. Can someone help me with this issue....
- 1085 Views
- 3 replies
- 0 kudos
- 0 kudos
Possible UI bug. Submit a Case with the Engineering team. https://help.databricks.com/s/ (Right top SUBMIT CASE)
- 0 kudos
| User | Count |
|---|---|
| 1615 | |
| 777 | |
| 349 | |
| 287 | |
| 260 |
