- 2159 Views
- 3 replies
- 3 kudos
Databricks Job API: The job must have exactly one owner
Hi Team,I'm trying to set the Job Permission using the Databricks Job API but getting the below error.{"error_code": "INVALID_PARAMETER_VALUE","message": "The job must have exactly one owner."} I first tried to get the job permission using the below ...
- 2159 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi , I have tried the same approach but it ddint work for me. Iam using api/2.0 with PUT Request
- 3 kudos
- 8430 Views
- 2 replies
- 0 kudos
Pivot in Databricks SQL
Hi Team,I have a table that has a key column (column name) and value column (value of the column name). These values are generated dynamically and wanted to pivot the table.Question 1: Is there a way that we can do this without specifying all the col...
- 8430 Views
- 2 replies
- 0 kudos
- 0 kudos
PySpark can help to list the available columns .Please find the demo snippets as below Image 1. Image 2
- 0 kudos
- 1802 Views
- 4 replies
- 1 kudos
Procedure of retrieving archived data from delta table
Hi all,I am currently researching on the archive support features in Databricks. https://docs.databricks.com/en/optimizations/archive-delta.htmlLet say I have enabled archive support and configured the data to be archived after 5 years and I also con...
- 1802 Views
- 4 replies
- 1 kudos
- 1 kudos
@Walter_C Thank you for your reply. However, there are some part that might need your further clarification.Assume I already set the delta.timeUntilArchived to 1825days (5years) and I have configured the lifecycle policy align with databricks setting...
- 1 kudos
- 1660 Views
- 3 replies
- 0 kudos
How to bring in databricks dbacademy courseware
I have created an account in dbacademy and signed up for advanced data engineering with databricks course. Also, I have subscribed to Vocareum lab as well. During the demo, tutor/trainer opened 'ADE 1.1 - Follow Along Demo - Reading from a Streaming ...
- 1660 Views
- 3 replies
- 0 kudos
- 0 kudos
So, it appears that we no longer make the notebooks available with self-paced training. They are not available for download.
- 0 kudos
- 2892 Views
- 2 replies
- 0 kudos
Deploying Overwatch on Databricks (AWS) with System Tables as the Data Source
IntroductionOverwatch is a powerful tool for monitoring and analyzing your Databricks environment, providing insights into resource utilization, cost management, and system performance. By leveraging system tables as the data source, you can gain a c...
- 2892 Views
- 2 replies
- 0 kudos
- 0 kudos
hi @jiteshraut20, Thanks for your post. From my set up, validation seems to work.Wrote 32 bytes. Validation report has been saved to dbfs:/mnt/overwatch_global/multi_ow_dep/report/validationReport Validation report details Total validation count: 35 ...
- 0 kudos
- 2398 Views
- 6 replies
- 0 kudos
Hive metastore federation, internal and external unable to connect
I enabled the internal hive on the metastore federation using this query commandCREATE CONNECTION IF NOT EXISTS internal-hive TYPE hive_metastoreOPTIONS (builtin true);But I can't get a password or username to access the JDBC URL.
- 2398 Views
- 6 replies
- 0 kudos
- 0 kudos
Not really, what I want to achieve is connecting to an external hive but I do want to configure the external hive on our server to be able to interact with the Databricks cluster in such a way that I could have access to thrift protocol.
- 0 kudos
- 4017 Views
- 4 replies
- 0 kudos
Databricks regression test suite
Hi, I am new to Databricks and setting up the non-prod environment. I am wanted to know, IS there any way by which I can run a regression suite so that existing setup should not break in case of any feature addition and also how can I make available ...
- 4017 Views
- 4 replies
- 0 kudos
- 0 kudos
Regression testing after code changes can be automated easily. Once you’ve created test cases with Pytest or Great Expectations, you can set up a CI/CD pipeline using tools like Jenkins or GitHub Actions. For a non-prod setup, Docker is great for rep...
- 0 kudos
- 1134 Views
- 3 replies
- 1 kudos
Optimize Cluster Uptime by Avoiding Unwanted Library or Jar Installations
Whenever we discuss clusters or nodes in any service, we need to address the cluster bootstrap process. Traditionally, this involves configuring each node using a startup script (startup.sh).In this context, installing libraries in the cluster is par...
- 1134 Views
- 3 replies
- 1 kudos
- 1177 Views
- 1 replies
- 0 kudos
How to set environment (client) on notebook via API/Terraform provider?
I am deploying a job with a notebook task via the Terraform provider. I want to set the client version to 2. I do NOT need to install any dependencies. I just want to use the new client version for the serverless compute. How do I do this with the Te...
- 1177 Views
- 1 replies
- 0 kudos
- 0 kudos
Unfortunately, there is no direct way to set the client version for a notebook task via the Terraform provider or the API without using the UI. The error message suggests that the %pip magic command is the recommended approach for installing dependen...
- 0 kudos
- 1203 Views
- 4 replies
- 2 kudos
model_serving_endpoints in DAB updates every time
Love the model_serving_endpoints in the dab, but now it takes over 6 minutes to deploy resources when they already exist. It says (updating) in the serving tab in the side bar even if nothing has changed.Is there a way to not update the endpoints aft...
- 1203 Views
- 4 replies
- 2 kudos
- 2 kudos
I have created an internal feature request for this behavior: DB-I-13108
- 2 kudos
- 1856 Views
- 0 replies
- 2 kudos
Databricks UniForm - Bridging Delta Lake and Iceberg
Databricks UniForm, enables seamless integration between Delta Lake and Iceberg formats. Databricks UniForm key features include:Interoperability: Read Delta tables with Iceberg clients without rewriting data.Automatic Metadata Generation: Asynchrono...
- 1856 Views
- 0 replies
- 2 kudos
- 4922 Views
- 2 replies
- 0 kudos
Jobs indefinitely pending with libraries install
I think I found a bug where you get Pending indefinitely on jobs that has a library requirement and the user of the job does not have Manage permission on the cluster.In my case I was trying to start a dbt job with dbt-databricks=1.8.5 as library. Th...
- 4922 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for your feedback! Just checking is this still an issue for you? would you share more details? if I wanted to reproduce this for example.
- 0 kudos
- 9556 Views
- 10 replies
- 0 kudos
Resolved! Py4JJavaError: An error occurred while calling o552.count()
Hey! I'm new to the forums but not Databricks, trying to get some help with this question:The error also is also fickle since it only appears what seems to be random. Like when running the same code it works then on the next run with a new set of dat...
- 9556 Views
- 10 replies
- 0 kudos
- 0 kudos
@ls Agree, it doesn't seem to be fixed. Maybe on DBR 16 memory management is better optimized, hence I'd like to suggest going through the methods mentioned earlier in this post: Memory Profiling: Try freezing the dataset that reproduces the problem...
- 0 kudos
- 1814 Views
- 1 replies
- 0 kudos
How Dedicated Access mode work ?
I have a question about the dedicated access modehttps://docs.databricks.com/en/compute/group-access.htmlIt is stated that:"Dedicated access mode is the latest version of single user access mode. With dedicated access, a compute resource can be assig...
- 1814 Views
- 1 replies
- 0 kudos
- 0 kudos
The dedicated access mode allows multiple users within the assigned group to access and use the compute resource simultaneously. This is different from the traditional single-user access mode, as it enables secure sharing of the resource among group ...
- 0 kudos
- 3353 Views
- 1 replies
- 1 kudos
Resolved! referencing external locations in python notebooks
How can I refrence external lcoations in python notebook. I got the docs for referencing it in python : https://docs.databricks.com/en/sql/language-manual/sql-ref-external-locations.html.But how to do it in python. I am not able to understand. Do we ...
- 3353 Views
- 1 replies
- 1 kudos
- 1 kudos
@ashraf1395 ,Referencing external locations in a Databricks Python notebook, particularly for environments like Azure DevOps with different paths for development (dev) and production (prod), can be effectively managed using parameterized variables. H...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
