- 11779 Views
- 6 replies
- 5 kudos
Resolved! Is there a way that admins can restrict users to install libraries on clusters and notebooks?
- 11779 Views
- 6 replies
- 5 kudos
- 5 kudos
one way to manage is make the cluster permission only to can restart and then use an init script to install libraries on start up so that users wont install libraries on the fly.
- 5 kudos
- 2832 Views
- 1 replies
- 3 kudos
Delta Optimized Write vs Reparation, Which is recommended?
When streaming to a Delta table, both repartitioning on the partition column and optimized write can help to avoid small files.Which is recommended between Delta Optimized Write vs Repartitioning?
- 2832 Views
- 1 replies
- 3 kudos
- 3 kudos
Optimized write is recommended over repartitioning for the below reasons.* The key part of Optimized Writes is that it is an adaptive shuffle. If you have a streaming ingest use case and input data rates change over time, the adaptive shuffle will a...
- 3 kudos
- 4690 Views
- 0 replies
- 2 kudos
Accelerating row-wise Python UDF functions without using Pandas UDFÂ ProblemSpark will not automatically parallelize UDF operations on smaller/medium d...
Accelerating row-wise Python UDF functions without using Pandas UDFProblemSpark will not automatically parallelize UDF operations on smaller/medium dataframes. As a result, spark will process the UDF as a single non parallelized task. For row-wise op...
- 4690 Views
- 0 replies
- 2 kudos
- 4027 Views
- 1 replies
- 0 kudos
Resolved! Creating Pandas Data Frame of Features After Applying Variance Reduction
I am building a classification model using the following data frame of 120,000 records (sample of 5 records shown):Using this data, I have built the following model:from sklearn.model_selection import train_test_split from sklearn.feature_extraction....
- 4027 Views
- 1 replies
- 0 kudos
- 0 kudos
This is more of a scikit-learn question than a Databricks question. But poking around I think VT_reduced.get_support() is probably what you are looking for:https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold....
- 0 kudos
- 3310 Views
- 2 replies
- 1 kudos
how to include a third-party Maven package in MLflow model serving job cluster in Azure Databricks
We try to use MLflow Model Serving, this service will enable realtime model serving behind a REST API interface; it will launch a single-node cluster that will host our model. The issue happens when the single-node cluster try to get the environment...
- 3310 Views
- 2 replies
- 1 kudos
- 1 kudos
Unfortunately we came across this same issue. We were trying to use MLFlow Serve to produce an API that could take text input and pass it through some NLP. In this instance we had installed a maven package on the cluster, so the experiment would run ...
- 1 kudos
- 3638 Views
- 3 replies
- 19 kudos
Resolved! Welcome back! Please introduce yourself to the community. :)
Hello everyone! My name is Piper and I'm one of the community moderators for Databricks. I'd like to take this opportunity to welcome you to the new Databricks community! I'd also like to ask you to introduce yourself in this thread. We are here to h...
- 3638 Views
- 3 replies
- 19 kudos
- 19 kudos
I work mostly with health and medical data, on a contract or project basis. I am located in Bedford MA and Ogunquit Maine. I formerly worked at Blue Metal / Insight, which is where I got my start on Databricks.Languages -- Python, PySpark, Koalashttp...
- 19 kudos
- 2065 Views
- 0 replies
- 0 kudos
Can Admins enable Table Download on Sample but not on Full Dataset?
Is it possible to allow for Table download on a sampled dataset but not the full dataset? In the configuration settings it seems like you have to allow both?Not withstanding the fact people could loop through the sample download, it seems like a prud...
- 2065 Views
- 0 replies
- 0 kudos
- 4060 Views
- 1 replies
- 1 kudos
Need guidance on migrating power bi reports to databricks
Hi All, I want to import an existing database/tables (say AdventureWorks) to databricks. And after importing tables, I want to develop reports on top I need guidance on this. Can someone give me resources that could help me in doing things end to en...
- 4060 Views
- 1 replies
- 1 kudos
- 1 kudos
@sania fatimi​ There are several different ways to do this and it's really going to depend on what your current need is. You could for example load the data into the databricks delta lake and use the databricks powerbi connecter to query the data fr...
- 1 kudos
- 3193 Views
- 3 replies
- 0 kudos
Resolved! Libraries in Databricks Runtimes
Is it possible to easily determine what libraries and which version are included in a specific DBR Version?
- 3193 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello. My name is Piper and I'm one of the community moderators. One of the team members sent this information to me.This should be the correct path to check libraries installed with DBRs.https://docs.databricks.com/release-notes/runtime/8.3ml.html?_...
- 0 kudos
- 6011 Views
- 3 replies
- 4 kudos
Resolved! Upload CSV files on Databricks by code (note UI)
Hello everyone.I have a process on databricks when I need to upload a CSV file everyday manually.I would like to know if there is a way to import this data (as panda in python, for example) with no necessary to upload this file everyday manually util...
- 6011 Views
- 3 replies
- 4 kudos
- 4 kudos
Autoloader is indeed a valid option,or use of some kind of ETL tool which fetches the file and put it somewhere on your cloud provider, like Azure Data Factory or AWS Glue etc.
- 4 kudos
- 7034 Views
- 2 replies
- 3 kudos
Resolved! ssh onto Cluster as root
Hello, I'm following the instructions here:https://docs.databricks.com/clusters/configure.html?_ga=2.17611385.1712747127.1631209439-1615211488.1629573963#ssh-access-to-clustersto ssh onto the Driver node, and it's working perfectly when I ssh on as `...
- 7034 Views
- 2 replies
- 3 kudos
- 3 kudos
I am 99% sure that logging into a Databricks node as root will not be allowed.
- 3 kudos
- 2681 Views
- 2 replies
- 0 kudos
Resolved! What are the advantages of using Delta if I am using MLflow? How is Delta useful for DS/ML use cases?
I am already using MLflow. What benefit would Delta provide me since I am not really working on Data engineering workloads
- 2681 Views
- 2 replies
- 0 kudos
- 0 kudos
The most important aspect is your experiment can track the version of the data table. So during audits you will be able to trace back why a specific prediction was made.
- 0 kudos
- 3825 Views
- 2 replies
- 3 kudos
Resolved! What is the best file format for a temporary table?
As part of my ETL process, I create intermediate/staging temporary tables. These tables created are read at a later point in the ETL and finally cleaned up. Should I use Delta? Using Delta creates the overhead of running optimize jobs, which would de...
- 3825 Views
- 2 replies
- 3 kudos
- 3 kudos
Agree.. the intermediate delta tables helps since it brings reliability to the pipeline.
- 3 kudos
- 1421 Views
- 0 replies
- 0 kudos
How to connect Neo4j aura to Databricks connection Error
I get this error: org.neo4j.driver.exceptions.SecurityException: Failed to establish secured connection with the serverI have tried to read through the documentation and tried the solution suggested but I can't seem to hack this problem.Kindly help. ...
- 1421 Views
- 0 replies
- 0 kudos
- 780 Views
- 0 replies
- 0 kudos
Hello World Databricks Community. My first post about nothing
Hello World Databricks Community.My first post about nothing
- 780 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
