Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
724 Posts
Activity in Databricks Platform Discussions
- 2 Views
- 0 replies
- 0 kudos
Data Governance
If I grant all privileges in my schema does that automatically give access to users for all underlying objects? Or should I give access seperately for all the objects?
- 2 Views
- 0 replies
- 0 kudos
- 14 Views
- 2 replies
- 0 kudos
Equivalent of Oracle's CLOB in Databricks
Dear all,(migrating for an on-premise Oracle ...)The question is in the subject: "What is the equivalent of Oracle's CLOB in Databricks" ?I saw that the "string" type can go up to 50 thousands characters, which is quite good in most of our cases, but...
- 14 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello;Thanks for the answer.For the concatenation itself, it is not an issue.My question is "is Databricks supporting something bigger than the 'string' data-type" ? Thanks
- 0 kudos
- 9 Views
- 0 replies
- 0 kudos
Partition Size:
HiI have chosen the default partition size 128 MB. I am reading a 3.8 GB file and checking the size of partition using df.rdd.getNumPartitions() as given below. I find the partition size: 159 MB. Why the partition size after reading the file differ ?...
- 9 Views
- 0 replies
- 0 kudos
- 4304 Views
- 3 replies
- 3 kudos
Resolved! Retrieve task name within workflow task (notebook, python)?
Using workflows, is there a way to obtain the task name from within a task?EX: I have a workflow with a notebook task. From within that notebook task I would like to retrieve the task name so I can use it for a variety of purposes.Currently, we're re...
- 4304 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi @EWhitley,Would {{task.name}} help in getting the current task name?https://docs.databricks.com/en/workflows/jobs/parameter-value-references.htmlPass context about job runs into job t
- 3 kudos
- 10 Views
- 0 replies
- 0 kudos
Query text truncated for queries longer than 153,596 characters
Hi, When using the `query_history.list` function of the python SDK workspace client the queries that have more than 153,596 characters are truncated.I could not find anywhere in the documentation this limit so I wanted to know if this is documented s...
- 10 Views
- 0 replies
- 0 kudos
- 58 Views
- 5 replies
- 0 kudos
Performance issue when reading data from a view.
Hi All,We are facing some performance issue and I need your help to know what could be the best approach to follow here.Existing: For each region, we have view(Reg1_View,Reg2_View..) to pull data from table( we don't have direct access to table).And ...
- 58 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Does any table hold data of all region 1. if yes. Get a Materialized view created (replacing all_reg_view)2. i see you already tried creating a staging table replacing the all_reg_view. Try creating cluster key along with partition.Cluster key on the...
- 0 kudos
- 56 Views
- 4 replies
- 0 kudos
How to add shared libs
Hi team,I want to add some shared libs which might be used by many repos, e.g. some util functions which might be used by any repos.1. What is the recommended way to add those libs? E.g. create a separate repo and reference it in another repo?2. How ...
- 56 Views
- 4 replies
- 0 kudos
- 1908 Views
- 2 replies
- 0 kudos
Unable to upload a wheel file in Azure DevOps pipeline
Hi, I am trying to upload a wheel file to Databricks workspace using Azure DevOps release pipeline to use it in the interactive cluster. I tried "databricks workspace import" command, but looks like it does not support .whl files. Hence, I tried to u...
- 1908 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @vvk - The HTTP 403 error typically indicates a permissions issue. Ensure that the SP has the necessary permissions to perform the fs cp operation on the specified path. Verify that the path specified in the fs cp command is correct and that the v...
- 0 kudos
- 1714 Views
- 1 replies
- 0 kudos
How to mount AWS EFS via NFS on a Databricks Cluster
I'm trying to read in ~500 million small json files into an spark autoloader pipeline, and I seem to be slowed down massively by S3 request limits, so I want to explore using AWS EFS instead. I found this blog post: https://www.databricks.com/blog/20...
- 1714 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @stvayers Please refer to this doc. https://docs.databricks.com/api/workspace/clusters/create It has instructions on how to mount using EFS.
- 0 kudos
- 243 Views
- 5 replies
- 2 kudos
(Big) Problem with SQL Warehouse Auto stop
Long story short, I'm not sure if this is an already known problem, but the Auto Stop feature on SQL Warehouses after minutes of inactivity is not working properly.We started using SQL Warehouses more aggressively this December when we scaled up one ...
- 243 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
Is this still being investigated by Databricks? I'm seeing similar behavior that's costing us a lot of money.
- 2 kudos
- 288 Views
- 4 replies
- 0 kudos
Resolved! select 1 query not finishing
Hello,I have the issue that even a query like "select 1" is not finishing. The sql warehouse runs infinite. I have no idea where to look for any issues because in the SPARK UI I cant see any error.What is intresting is that also allpurpose clusters (...
- 288 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Bepposbeste1993, Do you have the case ID raised for this issue?
- 0 kudos
- 16 Views
- 0 replies
- 0 kudos
Catalogue global UDF's
The current UDF implementation stores UDFs in a catalogue.schema location. This requires reference/call to said udf location; example `select my_catalogue.my_schema.my_udf()`. Or have the sql execute from that schema.In Snowflake, UDFs are globally a...
- 16 Views
- 0 replies
- 0 kudos
- 28 Views
- 0 replies
- 0 kudos
Drive memory utilization cleanup
Issue Summary:When running multiple jobs on the same compute cluster, over time, I see an increase in memory utilization that is seemingly never fully released, even when jobs finish. This eventually leads to some jobs stalling out as memory hits the...
- 28 Views
- 0 replies
- 0 kudos
- 35 Views
- 1 replies
- 0 kudos
Getting error while creating external delta table in Databricks
I am getting below error while creating external delta table in Databricks, even there is a external location created.[NO_PARENT_EXTERNAL_LOCATION_FOR_PATH] No parent external location was found for path 'abfss://destination@datalakeprojectsid.dfs.co...
- 35 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@Siddalinga If the path specified during table creation is outside the scope of the external location, you may encounter the [NO_PARENT_EXTERNAL_LOCATION_FOR_PATH] error.Is the external location correctly defined to scope the directory, such as abfss...
- 0 kudos
- 6960 Views
- 6 replies
- 4 kudos
Setup UCX (Databricks CLI) from Databricks web terminal
Setup UCX from Databricks web terminal For cases when your desktop or laptop will not support the databricks labs install ucx technical requirements such as Python 3.10, administrative access, python package access or network access but your Databric...
- 6960 Views
- 6 replies
- 4 kudos
Latest Reply
- 4 kudos
I’ve found that using DBR 15.4, Personal Compute, and a Single Node is the most effective setup for installing UCX via web terminal. Note: The Databricks CLI is the recommended and most efficient option for managing the UCX installation. Always try t...
- 4 kudos
| User | Count |
|---|---|
| 1786 | |
| 841 | |
| 464 | |
| 311 | |
| 298 |
