- 2457 Views
- 2 replies
- 3 kudos
Resolved! Data lineage graph now working
I created the below tables but when I click the lineage graph not able to see the upstream or downstream table .... the + sign goes away after a few sec but not able to click it ... anyone else having this issue?CREATE TABLE IF NOT EXISTS lineage_d...
- 2457 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Raj Sharma​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 2072 Views
- 2 replies
- 1 kudos
Resolved! How to load multiple xlsx that are storage in different folders with the same name in a blob storage in a delta table ?
Hi i have a blob storage with multile unzip folders with the same suffix folder_report_name_01_2023_01_02 -> file_name_2023_01_02.xlsxBut i want to load all of this data using pandas or pyspark and insert in my delta table.I'm trying to using widget...
- 2072 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Fernando Vázquez​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 1 kudos
- 2568 Views
- 2 replies
- 2 kudos
Why Azure Databricks needs to store data in temp storage in Azure before writing to the synapse.
I was following the tutorial about data transformation with azure databricks, and it says before loading data into azure synapse analytics, the data transformed by azure databricks would be saved on temp storage in azure blob storage first before loa...
- 2568 Views
- 2 replies
- 2 kudos
- 2 kudos
@Ajay Pandey​ Saving the transformed data to temporary storage in Azure Blob Storage before loading into Azure Synapse Analytics provides a number of benefits to ensure that the data is accurate, optimized, and performs well in the target environmen...
- 2 kudos
- 1311 Views
- 1 replies
- 0 kudos
What is the upper bound limit for dataSkippingNumIndexedCols, to keeps stats in delta log file?
Is there an upper bound of number that i can assign to delta.dataSkippingNumIndexedCols for computing statistics. Is there some tradeoff benchmark available for increasing this number beyond 32.
- 1311 Views
- 1 replies
- 0 kudos
- 0 kudos
@Chhavi Bansal​ :The delta.dataSkippingNumIndexedCols configuration property controls the maximum number of columns that Delta Lake will build statistics on during data skipping. By default, this value is set to 32. There is no hard upper bound on th...
- 0 kudos
- 6527 Views
- 1 replies
- 1 kudos
how can i export my dashboard en format html using databriks api
hi everyone, i would like to export my dashbord in html format and embed it in my body of my email in order to send it to my teamso there is my code python for the databriks api and i got this error and when i put my htm in the body of my message i...
- 6527 Views
- 1 replies
- 1 kudos
- 1 kudos
@mathild noun​ :import databricks.workspace as workspace_api import requests # set up your Databricks workspace credentials domain = "<your Databricks workspace domain>" token = "<your Databricks API token>" # set up the workspace client workspac...
- 1 kudos
- 10574 Views
- 1 replies
- 2 kudos
How does hash shuffle join work in Spark?
Hi All, I am trying to understand the internals shuffle hash join. I want to check if my understanding of it is correct. Let’s say I have two tables t1 and t2 joined on column country (8 distinct values). If I set the number of shuffle partitions as ...
- 10574 Views
- 1 replies
- 2 kudos
- 2 kudos
@Vinay Emmadi​ : In Spark, a hash shuffle join is a type of join that is used when joining two data sets on a common key. The data is first partitioned based on the join key, and then each partition is shuffled and sent to a node in the cluster. The ...
- 2 kudos
- 4120 Views
- 1 replies
- 1 kudos
Save Spark DataFrame to shape file (.shp format)
Hello,I know how to create .shp file from Geopandas dataframe using code similar to this, also mentioned on SO:gpd_df = geopandas.GeoDataFrame(pandas_df, geometry='geom') gpd_df .to_file("username/nh.shp")However I have .parquet files that I can load...
- 4120 Views
- 1 replies
- 1 kudos
- 1 kudos
@Bartosz Maciejewski​ :Spark does not have native support for writing Shapefiles directly. However, you can use a third-party library such as GeoPandas or PyShp to write your Spark DataFrame to a Shapefile.Here's an example of how to use GeoPandas to...
- 1 kudos
- 7050 Views
- 1 replies
- 4 kudos
Resolved! Query related to Storage account authentication
Use Case: Copy data from SharePoint List to Blob using Power AutomateShort Description:To Access the blob storage account from Power Automate. There are three authentication type:1. Access Key2. Service Principal3. Azure AD IntegratedWhich authentica...
- 7050 Views
- 1 replies
- 4 kudos
- 4 kudos
@KVNARK .​ :It's recommended to use the Azure AD Integrated authentication type. This authentication type allows you to use Azure Active Directory (AD) to authenticate and manage access to Blob Storage resources at the folder or container level using...
- 4 kudos
- 1309 Views
- 0 replies
- 2 kudos
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Ma...
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to shar...
- 1309 Views
- 0 replies
- 2 kudos
- 2802 Views
- 2 replies
- 1 kudos
How to download a file in DBFS that contains multibyte characters in the file path?
I would like to download a file in DBFS using the FileStore Endpoint.If the file or folder name contains multibyte characters, the file path cannot be specified due to URL encoding and an error occurs.Question 1: If a file or folder name contains mul...
- 2802 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi,Databricks CLI can be used to download a file from DBFS. https://docs.databricks.com/dev-tools/cli/index.htmlAlso, you can refer to https://stackoverflow.com/questions/49019706/databricks-download-a-dbfs-filestore-file-to-my-local-machine , which ...
- 1 kudos
- 2030 Views
- 0 replies
- 4 kudos
lnkd.in
Databricks has introduced a new feature that allows users to send SQL statements to their database via REST API. Users can easily integrate this feature with any tool by simply posting their queries to the /api/2.0/sql/statements/ endpoint. With this...
- 2030 Views
- 0 replies
- 4 kudos
- 3255 Views
- 2 replies
- 5 kudos
Resolved! Databricks SQL External Connections
Lakehouse architectures seem enticing, especially from the standpoint of querying the data lake directly as it sits (as opposed to first migrating the data to an external data warehouse). While documentation and support seems pretty clear regarding ...
- 3255 Views
- 2 replies
- 5 kudos
- 2939 Views
- 1 replies
- 1 kudos
Resolved! python 3.10 custom docker image
Goal: To use Python 3.10.4+Why: We have Python repos that are not backward compatible we want to use.What: I have created an image from Databricks example experimental containers already with Ubuntu 22.04 (2 major versions newer than the curre...
- 2939 Views
- 1 replies
- 1 kudos
- 1 kudos
After searching for an hour, I realized what I needed to look for. It's the importing the Iterable from collections, which is deprecated in 3.10. I guess Databricks hasn't migrated code, yet. In which case, I'm at a cross-roads. Databricks 3.9, local...
- 1 kudos
- 2184 Views
- 2 replies
- 1 kudos
Resolved! How do I reduce the size of a hive table's S3 bucket
I have a hive table in Delta format with over 1B rows, when I check the Data Explorer in the SQL section of Databricks it notes that the table size is 139.3GiB with 401 files but when I check the S3 bucket where the files are located (dbfs:/user/hive...
- 2184 Views
- 2 replies
- 1 kudos
- 1 kudos
When you run updates, deletes etc on a delta table, new files are created. However, the old files are not automatically deleted. This is to allow for features like time travel on the Delta tables. In order to delete older files for a delta table, you...
- 1 kudos
- 1166 Views
- 1 replies
- 5 kudos
Exciting news for Databricks users! #databricks launched a new feature that allows users to run job workflows continuously. Setting up a continuous jo...
Exciting news for Databricks users! #databricks launched a new feature that allows users to run job workflows continuously. Setting up a continuous job workflow is straightforward: create a job and select the continuous trigger option in the scheduli...
- 1166 Views
- 1 replies
- 5 kudos
- 5 kudos
Thank you for sharing!!!
- 5 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
