- 3459 Views
- 1 replies
- 0 kudos
How to allow Table deletion without requiring ownership on table? Problem Description In DBR 6 (and earlier), a non-admin user can delete a table that...
How to allow Table deletion without requiring ownership on table?Problem DescriptionIn DBR 6 (and earlier), a non-admin user can delete a table that the user doesn't own, as long as the user has ownership on the table's parent database (perhaps throu...
- 3459 Views
- 1 replies
- 0 kudos
- 0 kudos
I am having the same issue but on Python 3.10.12.I need to be able to have another user have "manage" access to a table in the unity catalog. We both have write access to the schema.
- 0 kudos
- 29142 Views
- 11 replies
- 7 kudos
PGP Encryption / Decryption in Databricks
Is there a way to Decrypt / Encrypt Blob files in Databricks using Key stored in Key Vault. What libraries need to be used? Any code snippets? Links?
- 29142 Views
- 11 replies
- 7 kudos
- 29323 Views
- 21 replies
- 13 kudos
Community Edition signup issues
I am trying to sign up for the community edition (https://databricks.com/try-databricks) for use with a databricks academy course. However, I am unable to signup and I receive the following error (image attached). On going to login page (link in ora...
- 29323 Views
- 21 replies
- 13 kudos
- 13 kudos
I am facing issue like you have reached maximum number of account associated with this databricks account and also you are not the member of any workspace and also i can't delete my existing account associated with my email.And also i can't open my c...
- 13 kudos
- 6151 Views
- 6 replies
- 7 kudos
Resolved! Auto Loader for Shape File
Hello: As you can see from below link, that it support 7 file formats. I am dealing with GeoSpatial Shape files and I want to know if Auto Loader can support Shape Files ???Any help on this is greatly appreciated. Thanks. https://docs.microsoft.com/...
- 6151 Views
- 6 replies
- 7 kudos
- 7 kudos
You could try to use the binary file type. But the disadvantage of this is that the content of the shape files will be put into a column, that might not be what you want.If you absolutely want to use the autoloader, maybe some thinking outside the b...
- 7 kudos
- 1856 Views
- 2 replies
- 1 kudos
Does Databricks have a data processing agreement?
Does Databricks have a data processing agreement?
- 1856 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes, Databricks typically provides a Data Processing Agreement (DPA) to comply with data protection regulations like GDPR. It's important for businesses to thoroughly review these agreements to ensure alignment with their data privacy policies. You c...
- 1 kudos
- 23134 Views
- 18 replies
- 18 kudos
Resolved! How to connect Neo4j aura to a cluster
Please help resolve this error :org.neo4j.driver.exceptions.SecurityException: Failed to establish secured connection with the serverThis occurs when I want to establish a connection to neo4j aura to my cluster .Thank you.
- 23134 Views
- 18 replies
- 18 kudos
- 18 kudos
I have added the init script on cluster start up in Spark->config->init.scripts. But my clsuter won't start after this.Cluster scoped init script /Volumes/xxx/xxx/neo4j/neo4j-init.sh failed: Script exit status is non-zero. Could you please help me or...
- 18 kudos
- 4047 Views
- 3 replies
- 5 kudos
Resolved! How do I use databricks-cli without manual configuration
I want to use databricks cli:databricks clusters listbut this requires a manual step that requires interactive work with the user:databricks configure --tokenIs there a way to use databricks cli without manual intervention so that you can run it as p...
- 4047 Views
- 3 replies
- 5 kudos
- 5 kudos
You can set two environment variables: DATABRICKS_HOST and DATABRICKS_TOKEN, and databricks-cli will use them. See the example of that in the DevOps pipelinesee the full list of environment variables at the end of the Authentication section of docume...
- 5 kudos
- 6443 Views
- 3 replies
- 1 kudos
How to migrate power bi reports to databricks
I have a sample set of power bi(.pbix) reports with all dropdowns, tables, filters etc. Now I would like to migrate this reports to data bricks. whatever visuals are created in power bi, I would like to create same in data bricks from scratch. I wou...
- 6443 Views
- 3 replies
- 1 kudos
- 1 kudos
We must ensure you are sure that the Databricks cluster is operational. These are the steps needed for integration between Azure Databricks into Power BI Desktop.1. Constructing the URL for the connectionConnect to the cluster, and click the Advanced...
- 1 kudos
- 3705 Views
- 4 replies
- 1 kudos
Can you use the Databricks API from a notebook?
I want to test out different APIs directly from a Databricks notebook instead of using Postman or CURL. Is this possible?
- 3705 Views
- 4 replies
- 1 kudos
- 1 kudos
@Panda There is no REST API for databricks. "RE" in REST stands for Ready Everywhere. You cannot connect to the API in workspace 1, from a notebook in workspace 2. Therefor it is Not Ready Everywhere. Workspace 1 cannot resolve the hostname for Works...
- 1 kudos
- 24587 Views
- 3 replies
- 3 kudos
How can I make Databricks API calls from notebook?
Access to Databricks APIs require the user to authenticate. This usually means creating a PAT (Personal Access Token) token. Conveniently, a token is readily available to you when you are using a Databricks notebook.databricksURL = dbutils.notebook....
- 24587 Views
- 3 replies
- 3 kudos
- 3 kudos
@User16752245312 You can use Databricks Secret Scope to manage sensitive data such as personal access tokens (PATs) securely. Storing your token in a secret scope ensures you don’t hard-code credentials in your notebook, making it more secure.For mo...
- 3 kudos
- 60723 Views
- 9 replies
- 5 kudos
Resolved! How to get the total directory size using dbutils
Is there a way to get the directory size in ADLS(gen2) using dbutils in databricks? If I run this dbutils.fs.ls("/mnt/abc/xyz") I get the file sizes inside the xyz folder( there are about 5000 files), I want to get the size of the XYZ folder how ca...
- 60723 Views
- 9 replies
- 5 kudos
- 5 kudos
File size is only specified for files. So, if you specify a directory as your source, you have to iterate through the directory. The below snippet should work (and should be faster than the other solutions).import glob def get_directory_size_in_byt...
- 5 kudos
- 5019 Views
- 2 replies
- 4 kudos
- 5019 Views
- 2 replies
- 4 kudos
- 4 kudos
Yes it does, !!https://databricks.com/session/secured-kerberos-based-spark-notebook-for-data-science
- 4 kudos
- 14931 Views
- 3 replies
- 6 kudos
Resolved! How to add I custom logging in Databricks
I want to add custom logs that redirect in the Spark driver logs. Can I use the existing logger classes to have my application logs or progress message in the Spark driver logs.
- 14931 Views
- 3 replies
- 6 kudos
- 6 kudos
1) Is it possible to save all the custom logging to its own file? Currently it is being logging with all other cluster logs (see image) 2) Also Databricks it seems like a lot of blank files are also being created for this. Is this a bug? this include...
- 6 kudos
- 19084 Views
- 3 replies
- 4 kudos
Resolved! AttributeError: 'DataFrame' object has no attribute 'rename'
Hello, I am doing the Data Science and Machine Learning course. The Boston housing has unintuitive column names. I want to rename them, e.g. so 'zn' becomes 'Zoning'. When I run this command: df_bostonLegible = df_boston.rename({'zn':'Zoning'}, axi...
- 19084 Views
- 3 replies
- 4 kudos
- 4 kudos
If df_boston is a DataFrame, but you still face issues, try an alternative syntax: df_boston = df_boston.rename(columns={'zn': 'Zoning'}).Make sure df_boston is a proper DataFrame and you're using a recent version of Pandas.
- 4 kudos
- 6393 Views
- 3 replies
- 2 kudos
TPC -DS test On databricks
If I want to run TPC-DS test on databricks what are the steps involved, do we have already daya available on databricks file system or I have to download or create from somewhere.
- 6393 Views
- 3 replies
- 2 kudos
- 2 kudos
See the spark-sql-perf repo for details on how to run benchmark tests using TPC-DS - https://github.com/databricks/spark-sql-perf
- 2 kudos