- 4861 Views
- 2 replies
- 3 kudos
Resolved! issue with rounding selected column in "for in" loop
This must be trivial, but I must have missed something.I have a dataframe (test1) and want to round all the columns listed in list of columns (col_list)here is the code I am running:col_list = ['measure1', 'measure2', 'measure3'] for i in col_list:...
- 4861 Views
- 2 replies
- 3 kudos
- 7322 Views
- 2 replies
- 3 kudos
Resolved! Running vacuum on each table
Hi, in line with my question about optimize, this is the next step, with a retention of 7 days I could execute vacuum on all tables once a week, is this a recommended procedure?How can I know if I'll be getting any benefit from vacuum, without DRY RU...
- 7322 Views
- 2 replies
- 3 kudos
- 3 kudos
Ideally 7 days is recommended, but discuss with data stakeholders to identify what's suitable? 7/14/28 days. To use VACCUM, first run some analytics on behaviour of your data.Identify % of operations that perform updates and deletes vs insert operati...
- 3 kudos
- 9823 Views
- 2 replies
- 0 kudos
Removing new line character from spark dataframe column
I have to remove new line character from entire column of a dataframe , I tried with regex_replace but its not working.Help me on this.​
- 9823 Views
- 2 replies
- 0 kudos
- 0 kudos
Could you please provide example of your data and the code you've tried?
- 0 kudos
- 2568 Views
- 1 replies
- 0 kudos
Resolved! Databricks PAT (personal access token) with access to databases selectively
Hi all,I am establishing a connection to databricks from Collibra through Spark driver. Collibra expects these details for the connection (for token based):personal access token (pat)server/workspace namehttpPathUpon successful connection, Collibra d...
- 2568 Views
- 1 replies
- 0 kudos
- 0 kudos
PAT token is integrated with the workspace, So it will get access of all hive. Is there anyway you can filter out with Collibra?
- 0 kudos
- 6031 Views
- 4 replies
- 0 kudos
Resolved! Hive version after Upgrade Azure Databricks from 6.4 (Spark 2) to 9.1 (Spark 3)
I have upgraded the Azure Databricks from 6.4 to 9.1 which enable me to use Spark3. As far as I know, the Hive version has to be upgraded to 2.3.7 as well as discussed in: https://community.databricks.com/s/question/0D53f00001HKHy2CAH/how-to-upgrade-...
- 6031 Views
- 4 replies
- 0 kudos
- 0 kudos
I'm asking about Datatricks version 9.1. I've follow the url given (https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore). Do you mind letting me know where in the table is mentioning the supported hive version fo...
- 0 kudos
- 6927 Views
- 8 replies
- 6 kudos
Resolved! Compile all the scripts under the workspace folder
In workspace one folder I have around 100+ pyspark scripts, all these scripts need to be compiled before running the main program. In order to compile all these files, we are using the %run magic command like %run ../prod/netSales. Since we have 100+...
- 6927 Views
- 8 replies
- 6 kudos
- 6 kudos
Problem is that you can list all files in workspace only via API call and than you can run every one of them using:dbutils.notebook.run()This is the script to list files from workspace (probably you need to add some filterning):import requests ctx = ...
- 6 kudos
- 6568 Views
- 4 replies
- 3 kudos
Resolved! Deploy tar.gz package from private git hub
We created Python package (.tar.gz) and kept it under private git.We can able to connect to that git (using PAT) from the Azure databricks notebook.Our requirement is to install that package from .tar.gz file for that notebook"pip install https://USE...
- 6568 Views
- 4 replies
- 3 kudos
- 3 kudos
For installing the package using pip you need to package the repo using setup.py. check this link for more details https://packaging.python.org/en/latest/tutorials/packaging-projects/alternatively you can pass the tar.gz using --py-files while submi...
- 3 kudos
- 1120 Views
- 0 replies
- 0 kudos
www.scent-sational-waxmelts.co.uk
Ignite your senses with distinctive and fm world shop delightful fragrances for your home, Discover scents to set the mood and inspire fragrant memories
- 1120 Views
- 0 replies
- 0 kudos
- 996 Views
- 0 replies
- 0 kudos
www.scent-sational-waxmelts.co.uk
Ignite your senses with distinctive and soy wax melts delightful fragrances for your home, Discover scents to set the mood and inspire fragrant memories
- 996 Views
- 0 replies
- 0 kudos
- 6543 Views
- 5 replies
- 1 kudos
Resolved! Notebook level automated pipeline monitoring or failure notif
Hi, is there any way other than adf monitoring where in automated way we can get notebook level execution details without getting to go to each pipeline and checking
- 6543 Views
- 5 replies
- 1 kudos
- 1 kudos
@Vibhor Sethi​ - Would you be happy to mark @Werner Stinckens​' answer as best if it resolved your question?
- 1 kudos
- 2641 Views
- 1 replies
- 0 kudos
Databricks to Databricks connection
How do you connect to Azure Databricks instance from another Databricks instance? I needed to access (database) Views created in a Databricks instance from a Pyspark notebook running in another Databricks instance. Appreciate if anyone has any samp...
- 2641 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi there, @Venkata Ramakrishna Alvakonda​! My name is Piper, and I'm a moderator for the community. Thank you for your great question! Let's give the community a chance to respond first, and then we'll circle back around. If the community's response ...
- 0 kudos
- 8717 Views
- 1 replies
- 0 kudos
- 8717 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi there,My name is Piper, and I'm one of the moderators for Databricks. Thank you for coming to us with this. Let's give our members a chance to respond first, then we'll come back to see how things went.
- 0 kudos
- 8237 Views
- 1 replies
- 1 kudos
How to configure Databricks Connect to 'Assume Role' when accessing file from an AWS S3 bucket?
I have a Databricks cluster configured with an instance profile to assume role when accessing an AWS S3 bucket.Accessing the bucket from the notebook using the cluster works properly (the instance profile can assume role to access the bucket).However...
- 8237 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello, @lsoewito​ - My name is Piper, and I'm a moderator for the Databricks community. Welcome and thank you for coming to us with your question. I'm sorry to hear that you're having trouble. Let's give your peers a chance to answer your question. W...
- 1 kudos
- 9943 Views
- 2 replies
- 3 kudos
Resolved! How to reset an autoloader?
Hii'm using an autoloader with Azure Databricks:df = (spark.readStream.format("cloudFiles") .options(**cloudfile) .load("abfss://dev@std******.dfs.core.windows.net/**/*****)) at my target checkpointLocation folder there are some files and subdirs...
- 9943 Views
- 2 replies
- 3 kudos
- 3 kudos
@Aman Sehgal​ - My name is Piper, and I'm one of the moderators for Databricks. I wanted to jump in real quick to thank you for being so generous with your knowledge.
- 3 kudos
- 2208 Views
- 0 replies
- 2 kudos
Can not log in to https://community.cloud.databricks.com/login.html
Hi , I created the community cloud account , even I got a mail for resetting password . But once I try to log in to https://community.cloud.databricks.com/login.html , it does not give error , but simple hanging for some time and again login screen ...
- 2208 Views
- 0 replies
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
