- 4549 Views
- 1 replies
- 1 kudos
Catalog Volume Preview for PDF files
Hi,Does anyone know if and when Databricks will enable the Preview option to work for PDF files stored in a Catalog Volume?Thanks!Nick
- 4549 Views
- 1 replies
- 1 kudos
- 1 kudos
this capability has not been explicitly mentioned or confirmed in the available documentation. Further clarification or inquiries with Databricks product teams may be useful to confirm if such functionality is planned for future releases.
- 1 kudos
- 4840 Views
- 1 replies
- 0 kudos
Access control on view
We've created a view containing PII mapping values in Unity Catalog. We need to grant other users the ability to query this view and access the mapping values, but we must prevent them from seeing the view's definition. Is it possible to grant "exec...
- 4840 Views
- 1 replies
- 0 kudos
- 0 kudos
In Unity Catalog, enabling users to query a view while hiding its definition is currently not supported directly. Unity Catalog requires view definitions to be visible during query execution for metadata access purposes. While querying, users typical...
- 0 kudos
- 5891 Views
- 1 replies
- 0 kudos
Making dynamic tasks like in Airflow, but in Databricks?
I've used Airflow which allows us to create a DAG with dynamic tasks, for example we can have a list of items (such as table names), loop through an operator that accepts a table name and create a task for each table without having to create a new n...
- 5891 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, it is possible to create dynamic tasks in Databricks workflows, similar to the approach using Apache Airflow, by leveraging Databricks' job orchestration capabilities. However, the implementation may differ from Airflow's dynamic DAG creation. D...
- 0 kudos
- 3666 Views
- 3 replies
- 1 kudos
Resolved! Unable to import a library before restarting the cluster
Hi everyone,Anyone has problem when import a library in Databricks notebook? I found it failed to import. Then I restarted the cluster, run it again and it successfully imported the library.My concern here is that: I scheduled to run the notebook at ...
- 3666 Views
- 3 replies
- 1 kudos
- 3523 Views
- 5 replies
- 1 kudos
Resolved! Getting an error "You cannot use dbutils within a spark job"
I am using mlflow to register my custom model using a simple code below. The DatabricksParams extracts all the params from dbutils and sets the params dictionary and dbutils is not used anywhere else within the rest of my code base. The code fails wh...
- 3523 Views
- 5 replies
- 1 kudos
- 3355 Views
- 3 replies
- 0 kudos
Resolved! Auto Loader for copying files on s3
Hey community,I have a folder on s3 with around 5 million small files. On a daily basis new files are added. I would like to simply copy those new files to another folder on s3. My approach is to use an Auto Loader of which I attached the code below....
- 3355 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey LRALVAThe first time running your code I got the error: PicklingError: Could not serialize object: Exception: You cannot use dbutils within a spark job You cannot use dbutils within a spark job or otherwise pickle it.So I changed the copy_single...
- 0 kudos
- 2354 Views
- 1 replies
- 0 kudos
Resolved! Is SSL cert needed for Azure Databricks API calls?
Hi, I'm using Azure Databricks and making API calls to the endpoints with verify=False Eg.response = requests.get('https://%s/api/2.0/clusters/list' % (databricks_domain),headers=request_headers,verify=False)Security scanners are flagging the use of ...
- 2354 Views
- 1 replies
- 0 kudos
- 0 kudos
In API communication with Azure Databricks, it is not necessary to manually install or configure an SSL certificate on the client side because Azure Databricks endpoints inherently use TLS (Transport Layer Security) for secure communication. To addre...
- 0 kudos
- 5546 Views
- 2 replies
- 1 kudos
Docker image fails to initalize - Databricks Container Service
I am trying to use a public docker image hosted in Dockerhub for my job. I have not included public_auth field as per this guide since the image type is public and when the cluster is trying to initialize it fails with the error: Invalid argument: C...
- 5546 Views
- 2 replies
- 1 kudos
- 1 kudos
Did you figure this issue out? I am having the exact same problem - I am also having trouble finding thorough documentation/examples on the use of DCS, so if you have stumbled upon any links in that regard, i'd also appreciate it.
- 1 kudos
- 5103 Views
- 8 replies
- 2 kudos
Resolved! dbutils.notebooks.exit() is not returning results to the variable only on some notebooks
dbutils.notebooks.exit() is not returning results to the variable only on some notebooks issue is Parent child notebook communication and need some solution
- 5103 Views
- 8 replies
- 2 kudos
- 2 kudos
@SteveW @lorenzoscandola Its working without any issue from Yesterday. Thank you guys for the support.
- 2 kudos
- 18967 Views
- 8 replies
- 8 kudos
Resolved! connect databricks to teradata
hey i want to know can we connect databricks to the teradata database and if yes what will be the procedure ??? help would be appreciated
- 18967 Views
- 8 replies
- 8 kudos
- 8 kudos
There are two main ways to connect to Teradata from Databricks using Python.Way 1: Using Python Libraries (e.g., sqlalchemy, pyjdbc, pyodbc, jaydebeapi, and so on)Pros: Provides a comprehensive solution, allowing us to: Query data, Trigger stored pro...
- 8 kudos
- 5708 Views
- 3 replies
- 1 kudos
Issue when trying to create a Foreign Catalog to a On Prem SQL Server Instance
Hi,We are creating a lakehouse federated connection to our 2016 On Prem SQL Server. This has an instance in place, so we only want and need to connect to this instance. From this connection, we want to create a foreign catalog of a database on the ...
- 5708 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Nick_Pacey Thank you I already did that using in the codejdbc_url = "jdbc:sqlserver://999.99.999.99\\instance:7777;encrypt=true;trustServerCertificate=true;database=mydatabase"jdbc_username = "myusername"jdbc_password = "mypassword"jdbc_driver =...
- 1 kudos
- 42895 Views
- 6 replies
- 1 kudos
Resolved! Importing CA certificate into a Databricks cluster
Hi!I was following guide outlined here:https://kb.databricks.com/en_US/python/import-custom-ca-cert(also tried this: https://stackoverflow.com/questions/73043589/configuring-tls-ca-on-databricks)to add ca root certificate into Databricks cluster, but...
- 42895 Views
- 6 replies
- 1 kudos
- 1 kudos
@Debayan One question - Will same approach work for JKS file containing private key certificate for X.509 authentication to Mongo Atlas database.Usual way of adding below spark config's is not working. spark.driver.extraJavaOptions -Djavax.net.ssl.ke...
- 1 kudos
- 2266 Views
- 3 replies
- 0 kudos
DLT Direct Publish Mode does not Handle Constraint Dependencies
I'm having some issues with the direct publish mode when defining a DLT workflow that includes tables where their schema defines foreign key constraints. When the foreign constraints reference tables that are not directly defined in any joins of the ...
- 2266 Views
- 3 replies
- 0 kudos
- 0 kudos
@Stentone This is a tricky situation where you want to leverage the metadata benefits (like the ERD visualization) without running into execution dependencies. Let me help you solve this issue.The error suggests that DLT is trying to validate the for...
- 0 kudos
- 4059 Views
- 3 replies
- 2 kudos
Custom container doesn't launch systemd
Quite soon after moving from VMs to containers, I started crafting my own images. That way notebooks have all the necessary libraries already there and no need to do any Pipping/installing in the notebook.As requirements get more complex, now I'm at ...
- 4059 Views
- 3 replies
- 2 kudos
- 2 kudos
Are there any updates on this? I'm also very interested in having full-fledged Databricks system services integrated into DCS.
- 2 kudos
- 5937 Views
- 2 replies
- 3 kudos
Resolved! when to activate photon and when not to ?
Photon appears as an option to check and uncheck as appropriate.The use of Photon leads to higher consumption of DBUs and higher costs.At what point does it pay off and when not to enable it.More costs for the use of photon, but at the same time less...
- 5937 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi All,Based on the discussion, can we load a huge flat file (csv) 10 GB using photon accelerated Runtime .Just dump that file into delta from cloud storage like S3 or BlobStorage.Is this one of those ideal usecases for using photon where it will be ...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
