- 13695 Views
- 7 replies
- 2 kudos
Bad performance UDFs functions
Hello,I am contacting you because I am having a problem with the performance of my notebooks on databricks.My notebook is written in python (pypark) in it I read a delta table that I copy to a dataframe and do several transformations and create sever...
- 13695 Views
- 7 replies
- 2 kudos
- 2 kudos
looping over records is a performance killer. To be avoided at all costs.beware the for-loop (databricks.com)
- 2 kudos
- 5301 Views
- 2 replies
- 1 kudos
Resolved! Custom Library's(Unity Catalog Enabled Clusters)
I'm trying to use a custom library that I created from a .whl file in the workspace/shared location. The library attaches to the cluster without any issues and I can it when I list the modules using pip. When I try to call the module I get an error t...
- 5301 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello Guys,I am working on the project where we need to use spark-excel library (Maven) in order to ingest data from excel files. As those 3rd party library are not allowed on shared cluster, do you have any workaround other then using pandas for exa...
- 1 kudos
- 7220 Views
- 4 replies
- 1 kudos
- 7220 Views
- 4 replies
- 1 kudos
- 1 kudos
Yes, it is possible to connect databricks to a kerberized hbase cluster. The attached article explains the steps. It consists of setting up a kerberos client using a keytab in the cluster nodes, installing the hbase-spark integration library, and set...
- 1 kudos
- 6217 Views
- 1 replies
- 0 kudos
Reading bigquery data using a query
To read Bigquery data using spark.read, i'm using a query. This query executes and creates a table on the materializationDataset. df = spark.read.format("bigquery") \.option("query", query) \.option("materializationProject", materializationProject) \...
- 6217 Views
- 1 replies
- 0 kudos
- 2184 Views
- 2 replies
- 2 kudos
Appropriate storage account type for reference data (Azure)
Hello,We are using a reference dataset for our Production applications. We would like to create a delta table for this dataset to be used from our applications. Currently, manual updates will occur on this dataset through a script on a weekly basis. ...
- 2184 Views
- 2 replies
- 2 kudos
- 2 kudos
+1 for ADLS. Hierarchical storage, hot/cold/premium storage, things not possible in blob storage
- 2 kudos
- 6715 Views
- 4 replies
- 1 kudos
Recommended Hive metastore pattern for Trino integration
Hi, i have several questions regarding Trino integration:Is it recommended to use an external Hive metastore or leverage on the databricks-maintained Hive metastore when it comes to enabling external query engines such as Trino?When I tried to use ex...
- 6715 Views
- 4 replies
- 1 kudos
- 1 kudos
> Is it recommended to use an external Hive metastore or leverage on the databricks-maintained Hive metastore when it comes to enabling external query engines such as Trino?Databricks maintained hive metastore is not suggested to be used externally. ...
- 1 kudos
- 8240 Views
- 3 replies
- 3 kudos
Update destination table when using Spark Structured Streaming and Delta tables
I’m trying to implement a streaming pipeline that will run hourly using Spark Structured Streaming, Scala and Delta tables. The pipeline will process different items with their details.The source are delta tables that already exists, written hourly u...
- 8240 Views
- 3 replies
- 3 kudos
- 3 kudos
@Agus1 Could you try using CDC in delta. You could use readChangeFeed to read only the changes that got applied on the source table. This is also explained here.https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed
- 3 kudos
- 4379 Views
- 2 replies
- 1 kudos
Unity Catalog Table/View Column Data Type Changes
When changing a delta table column data type in Unity Catalog, we noticed a view that is referencing that table did not automatically update to reflect the new data type.Is there a way to update the delta table column data type so that it also update...
- 4379 Views
- 2 replies
- 1 kudos
- 1 kudos
Can you try refreshing the view by running the command: REFRESH TABLE <viewname>
- 1 kudos
- 2547 Views
- 1 replies
- 0 kudos
Soft stop a Streaming Job
I have an auto loader streaming job with the continuous run, I want to stop that job on weekend for some time and restart again.
- 2547 Views
- 1 replies
- 0 kudos
- 7703 Views
- 5 replies
- 4 kudos
Resolved! Cluster Performance
Facing an issue with cluster performance, in event log can see - cluster is not responsive likely due to GC. Number of pipeline (databricks notebooks) running and cluster configuration is same as it used to be before but started seeing this issue sin...
- 7703 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @Vibhor Sethi​ ,Do you see any other error messages? did you data volume increase? what kind of job are you running?
- 4 kudos
- 3020 Views
- 1 replies
- 1 kudos
what is the best way to optimize historical data tables in databricks?
Hi There,Looking for recommendations on how to optimize history data tables in databricks?
- 3020 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks Kaniz!
- 1 kudos
- 28040 Views
- 5 replies
- 10 kudos
Resolved! SFTP Connect
How I can connect sftp server from databricks. So I can write files into tables directly?
- 28040 Views
- 5 replies
- 10 kudos
- 10 kudos
The classic solution is to copy data from FTP to ADLS storage using Azure Data Factory, and after the copy is done in the ADF pipeline, trigger the databricks notebook.
- 10 kudos
- 3089 Views
- 1 replies
- 0 kudos
Resolved! Databricks lakehouse fundamental certification page is not working
While going to give Databricks lakehouse fundamental certification quizIt is showing that you don't have permission to access this. Could you please guide me on that or can provide a link through which I can get permission to access this and complet...
- 3089 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Souvikng, Here is the link for the Accrediition.
- 0 kudos
- 5664 Views
- 1 replies
- 0 kudos
How to Get spark history server logs using python REST API
Hello,I am trying to load logs from spark job in remote location using Python Rest API.I want to collect those logs for particular job runs using runID field, that log should contain Errors, Exceptions, and print details.I have tried "/api/2.1/jobs/r...
- 5664 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello,How can I access endpoint (http URL) of spark history server for a particular cluster? Thanks,Parimal
- 0 kudos
- 10104 Views
- 2 replies
- 2 kudos
Resolved! Cannot delete file in dbfs
Hello, I am trying to delete a folder in /dbfs/mnt, but I am unable to do so. The folder was an old mounted storage account, which was deleted. The folder contains a single file 'mount.err'.In command line I tried rm -rf my_folder, with and without s...
- 10104 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @erigaud The issue you're facing might be due to the fact that the folder was a mounted storage account that has been deleted. This could cause some inconsistencies in the file system view and hence, you're unable to delete the folder.You can try...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
