- 1513 Views
- 1 replies
- 1 kudos
Will enabling Unity Catalog affect existing user access and jobs in production?
Hi, at my company, we are using Databricks with AWS IAM identity center as single sign on, I was looking into Unity catalog which seems to offer centralized access but I wanted to know if there will be any downside like loss of existing user profile ...
- 1513 Views
- 1 replies
- 1 kudos
- 1 kudos
You can look into this doc https://docs.databricks.com/en/data-governance/unity-catalog/migrate.html which have some details about your question here.
- 1 kudos
- 10124 Views
- 7 replies
- 2 kudos
Bad performance UDFs functions
Hello,I am contacting you because I am having a problem with the performance of my notebooks on databricks.My notebook is written in python (pypark) in it I read a delta table that I copy to a dataframe and do several transformations and create sever...
- 10124 Views
- 7 replies
- 2 kudos
- 2 kudos
looping over records is a performance killer. To be avoided at all costs.beware the for-loop (databricks.com)
- 2 kudos
- 4356 Views
- 2 replies
- 1 kudos
Resolved! Custom Library's(Unity Catalog Enabled Clusters)
I'm trying to use a custom library that I created from a .whl file in the workspace/shared location. The library attaches to the cluster without any issues and I can it when I list the modules using pip. When I try to call the module I get an error t...
- 4356 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello Guys,I am working on the project where we need to use spark-excel library (Maven) in order to ingest data from excel files. As those 3rd party library are not allowed on shared cluster, do you have any workaround other then using pandas for exa...
- 1 kudos
- 6042 Views
- 4 replies
- 1 kudos
- 6042 Views
- 4 replies
- 1 kudos
- 1 kudos
Yes, it is possible to connect databricks to a kerberized hbase cluster. The attached article explains the steps. It consists of setting up a kerberos client using a keytab in the cluster nodes, installing the hbase-spark integration library, and set...
- 1 kudos
- 5131 Views
- 1 replies
- 0 kudos
Reading bigquery data using a query
To read Bigquery data using spark.read, i'm using a query. This query executes and creates a table on the materializationDataset. df = spark.read.format("bigquery") \.option("query", query) \.option("materializationProject", materializationProject) \...
- 5131 Views
- 1 replies
- 0 kudos
- 1746 Views
- 2 replies
- 2 kudos
Appropriate storage account type for reference data (Azure)
Hello,We are using a reference dataset for our Production applications. We would like to create a delta table for this dataset to be used from our applications. Currently, manual updates will occur on this dataset through a script on a weekly basis. ...
- 1746 Views
- 2 replies
- 2 kudos
- 2 kudos
+1 for ADLS. Hierarchical storage, hot/cold/premium storage, things not possible in blob storage
- 2 kudos
- 5535 Views
- 4 replies
- 1 kudos
Recommended Hive metastore pattern for Trino integration
Hi, i have several questions regarding Trino integration:Is it recommended to use an external Hive metastore or leverage on the databricks-maintained Hive metastore when it comes to enabling external query engines such as Trino?When I tried to use ex...
- 5535 Views
- 4 replies
- 1 kudos
- 1 kudos
> Is it recommended to use an external Hive metastore or leverage on the databricks-maintained Hive metastore when it comes to enabling external query engines such as Trino?Databricks maintained hive metastore is not suggested to be used externally. ...
- 1 kudos
- 6986 Views
- 3 replies
- 3 kudos
Update destination table when using Spark Structured Streaming and Delta tables
I’m trying to implement a streaming pipeline that will run hourly using Spark Structured Streaming, Scala and Delta tables. The pipeline will process different items with their details.The source are delta tables that already exists, written hourly u...
- 6986 Views
- 3 replies
- 3 kudos
- 3 kudos
@Agus1 Could you try using CDC in delta. You could use readChangeFeed to read only the changes that got applied on the source table. This is also explained here.https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed
- 3 kudos
- 3615 Views
- 2 replies
- 1 kudos
Unity Catalog Table/View Column Data Type Changes
When changing a delta table column data type in Unity Catalog, we noticed a view that is referencing that table did not automatically update to reflect the new data type.Is there a way to update the delta table column data type so that it also update...
- 3615 Views
- 2 replies
- 1 kudos
- 1 kudos
Can you try refreshing the view by running the command: REFRESH TABLE <viewname>
- 1 kudos
- 2142 Views
- 1 replies
- 0 kudos
Soft stop a Streaming Job
I have an auto loader streaming job with the continuous run, I want to stop that job on weekend for some time and restart again.
- 2142 Views
- 1 replies
- 0 kudos
- 6143 Views
- 5 replies
- 4 kudos
Resolved! Cluster Performance
Facing an issue with cluster performance, in event log can see - cluster is not responsive likely due to GC. Number of pipeline (databricks notebooks) running and cluster configuration is same as it used to be before but started seeing this issue sin...
- 6143 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @Vibhor Sethi​ ,Do you see any other error messages? did you data volume increase? what kind of job are you running?
- 4 kudos
- 2685 Views
- 1 replies
- 1 kudos
what is the best way to optimize historical data tables in databricks?
Hi There,Looking for recommendations on how to optimize history data tables in databricks?
- 2685 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks Kaniz!
- 1 kudos
- 25892 Views
- 5 replies
- 10 kudos
Resolved! SFTP Connect
How I can connect sftp server from databricks. So I can write files into tables directly?
- 25892 Views
- 5 replies
- 10 kudos
- 10 kudos
The classic solution is to copy data from FTP to ADLS storage using Azure Data Factory, and after the copy is done in the ADF pipeline, trigger the databricks notebook.
- 10 kudos
- 2566 Views
- 1 replies
- 0 kudos
Resolved! Databricks lakehouse fundamental certification page is not working
While going to give Databricks lakehouse fundamental certification quizIt is showing that you don't have permission to access this. Could you please guide me on that or can provide a link through which I can get permission to access this and complet...
- 2566 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Souvikng, Here is the link for the Accrediition.
- 0 kudos
- 5082 Views
- 1 replies
- 0 kudos
How to Get spark history server logs using python REST API
Hello,I am trying to load logs from spark job in remote location using Python Rest API.I want to collect those logs for particular job runs using runID field, that log should contain Errors, Exceptions, and print details.I have tried "/api/2.1/jobs/r...
- 5082 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello,How can I access endpoint (http URL) of spark history server for a particular cluster? Thanks,Parimal
- 0 kudos
| User | Count |
|---|---|
| 1623 | |
| 790 | |
| 501 | |
| 349 | |
| 287 |
