- 3013 Views
- 1 replies
- 2 kudos
Resolved! Azure synapse writing data with Databricks using polybase error
We are writing the data from Delta tables to Azure synapse using Azure Databricks. While loading the data into the synapse staging table getting an error " HdfsBridge::CreateRecordReader - Unexpected error encountered creating the record reader: abf...
- 3013 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi SivakrishnaSunkara,You will find more information and example in the following link url https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/synapse-analytics please follow the steps fro the authentication section to avoid thi...
- 2 kudos
- 6149 Views
- 1 replies
- 2 kudos
- 6149 Views
- 1 replies
- 2 kudos
- 2 kudos
Just like B-tree indices in the traditional EDW world, Z-order indexing can be used on high-cardinality columns like Primary Key columns and high-cardinality joins like facts and dimension tables joins. Z-order indexes can be created only on the ...
- 2 kudos
- 2396 Views
- 1 replies
- 4 kudos
- 2396 Views
- 1 replies
- 4 kudos
- 4 kudos
There are various methods like using uuid , monotonically_increasing_id(), using row_number() OVER (ORDER BY NULL) AS SK, using md5() or sha() hashing functions etc. Detailed discussion of various options and the pros/cons can be found in this youtu...
- 4 kudos
- 4520 Views
- 1 replies
- 2 kudos
Resolved! I am looking for best practices in implementing Ranger type of Access control in Databricks ?
Need this to do auditing and numbers of users accessing databases and tables created in databricks
- 4520 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi Sravan, Apache Ranger is commonly used for fine-grained access controls. In your description, it sounds like you might be able to leverage Databricks audit logs, which would allow you to see user-level actions: https://docs.databricks.com/administ...
- 2 kudos
- 12378 Views
- 6 replies
- 5 kudos
Resolved! Is there a way that admins can restrict users to install libraries on clusters and notebooks?
- 12378 Views
- 6 replies
- 5 kudos
- 5 kudos
one way to manage is make the cluster permission only to can restart and then use an init script to install libraries on start up so that users wont install libraries on the fly.
- 5 kudos
- 3094 Views
- 1 replies
- 3 kudos
Delta Optimized Write vs Reparation, Which is recommended?
When streaming to a Delta table, both repartitioning on the partition column and optimized write can help to avoid small files.Which is recommended between Delta Optimized Write vs Repartitioning?
- 3094 Views
- 1 replies
- 3 kudos
- 3 kudos
Optimized write is recommended over repartitioning for the below reasons.* The key part of Optimized Writes is that it is an adaptive shuffle. If you have a streaming ingest use case and input data rates change over time, the adaptive shuffle will a...
- 3 kudos
- 5027 Views
- 0 replies
- 2 kudos
Accelerating row-wise Python UDF functions without using Pandas UDFÂ ProblemSpark will not automatically parallelize UDF operations on smaller/medium d...
Accelerating row-wise Python UDF functions without using Pandas UDFProblemSpark will not automatically parallelize UDF operations on smaller/medium dataframes. As a result, spark will process the UDF as a single non parallelized task. For row-wise op...
- 5027 Views
- 0 replies
- 2 kudos
- 4423 Views
- 1 replies
- 0 kudos
Resolved! Creating Pandas Data Frame of Features After Applying Variance Reduction
I am building a classification model using the following data frame of 120,000 records (sample of 5 records shown):Using this data, I have built the following model:from sklearn.model_selection import train_test_split from sklearn.feature_extraction....
- 4423 Views
- 1 replies
- 0 kudos
- 0 kudos
This is more of a scikit-learn question than a Databricks question. But poking around I think VT_reduced.get_support() is probably what you are looking for:https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold....
- 0 kudos
- 3507 Views
- 2 replies
- 1 kudos
how to include a third-party Maven package in MLflow model serving job cluster in Azure Databricks
We try to use MLflow Model Serving, this service will enable realtime model serving behind a REST API interface; it will launch a single-node cluster that will host our model. The issue happens when the single-node cluster try to get the environment...
- 3507 Views
- 2 replies
- 1 kudos
- 1 kudos
Unfortunately we came across this same issue. We were trying to use MLFlow Serve to produce an API that could take text input and pass it through some NLP. In this instance we had installed a maven package on the cluster, so the experiment would run ...
- 1 kudos
- 3921 Views
- 3 replies
- 19 kudos
Resolved! Welcome back! Please introduce yourself to the community. :)
Hello everyone! My name is Piper and I'm one of the community moderators for Databricks. I'd like to take this opportunity to welcome you to the new Databricks community! I'd also like to ask you to introduce yourself in this thread. We are here to h...
- 3921 Views
- 3 replies
- 19 kudos
- 19 kudos
I work mostly with health and medical data, on a contract or project basis. I am located in Bedford MA and Ogunquit Maine. I formerly worked at Blue Metal / Insight, which is where I got my start on Databricks.Languages -- Python, PySpark, Koalashttp...
- 19 kudos
- 2177 Views
- 0 replies
- 0 kudos
Can Admins enable Table Download on Sample but not on Full Dataset?
Is it possible to allow for Table download on a sampled dataset but not the full dataset? In the configuration settings it seems like you have to allow both?Not withstanding the fact people could loop through the sample download, it seems like a prud...
- 2177 Views
- 0 replies
- 0 kudos
- 4231 Views
- 1 replies
- 1 kudos
Need guidance on migrating power bi reports to databricks
Hi All, I want to import an existing database/tables (say AdventureWorks) to databricks. And after importing tables, I want to develop reports on top I need guidance on this. Can someone give me resources that could help me in doing things end to en...
- 4231 Views
- 1 replies
- 1 kudos
- 1 kudos
@sania fatimi​ There are several different ways to do this and it's really going to depend on what your current need is. You could for example load the data into the databricks delta lake and use the databricks powerbi connecter to query the data fr...
- 1 kudos
- 4110 Views
- 3 replies
- 0 kudos
Resolved! Libraries in Databricks Runtimes
Is it possible to easily determine what libraries and which version are included in a specific DBR Version?
- 4110 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello. My name is Piper and I'm one of the community moderators. One of the team members sent this information to me.This should be the correct path to check libraries installed with DBRs.https://docs.databricks.com/release-notes/runtime/8.3ml.html?_...
- 0 kudos
- 6365 Views
- 3 replies
- 4 kudos
Resolved! Upload CSV files on Databricks by code (note UI)
Hello everyone.I have a process on databricks when I need to upload a CSV file everyday manually.I would like to know if there is a way to import this data (as panda in python, for example) with no necessary to upload this file everyday manually util...
- 6365 Views
- 3 replies
- 4 kudos
- 4 kudos
Autoloader is indeed a valid option,or use of some kind of ETL tool which fetches the file and put it somewhere on your cloud provider, like Azure Data Factory or AWS Glue etc.
- 4 kudos
- 7354 Views
- 2 replies
- 3 kudos
Resolved! ssh onto Cluster as root
Hello, I'm following the instructions here:https://docs.databricks.com/clusters/configure.html?_ga=2.17611385.1712747127.1631209439-1615211488.1629573963#ssh-access-to-clustersto ssh onto the Driver node, and it's working perfectly when I ssh on as `...
- 7354 Views
- 2 replies
- 3 kudos
- 3 kudos
I am 99% sure that logging into a Databricks node as root will not be allowed.
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 585 | |
| 349 | |
| 287 |
