- 4036 Views
- 6 replies
- 0 kudos
GeneratedAlwaysAs' along with dataframe.write
Is it possible to use a calculated column (as like in the delta table using generatedAlwaysAs) definition while writing the data frame as a delta file like df.write.format("delta").Any options are there with the dataframe.write method to achieve this...
- 4036 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Thushar R​ ,This option is not a part of Dataframe write API as GeneratedAlwaysAs feature is only applicable to Delta format and df.write is a common API to handle writes for all formats. If you to achieve this programmatically, you can still use...
- 0 kudos
- 5985 Views
- 4 replies
- 7 kudos
Resolved! Delta Live Table Schema Comment
I predefined my schema for a Delta Live Table Autoload. This included comments for some attributes. When performing a standard readStream, my comments appear, but when in Delta Live Tables I get no comments. Is there anything I need to do get comment...
- 5985 Views
- 4 replies
- 7 kudos
- 7 kudos
You need to add your schema to dlt declaration:@dlt.table( name="test_bronze", comment = "test account data incrementally ingested from S3 Raw landing zone", table_properties={ "quality": "bronze" }, schema=schema)
- 7 kudos
- 31392 Views
- 4 replies
- 1 kudos
Resolved! How to get rid of "Command result size exceeds limit"
I am working on Databricks Notebook and trying to display a map using Floium and I keep getting this error > Command result size exceeds limit: Exceeded 20971520 bytes (current = 20973510)How can I get increase the memory limit?I already reduced the...
- 31392 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi, I have the same problem with keplergl, and the save to disk option, whilst helpful isn't super practical... So how does one plot large datasets in kepler?Any thought welcome
- 1 kudos
- 2546 Views
- 2 replies
- 3 kudos
Resolved! Data lineage graph now working
I created the below tables but when I click the lineage graph not able to see the upstream or downstream table .... the + sign goes away after a few sec but not able to click it ... anyone else having this issue?CREATE TABLE IF NOT EXISTS lineage_d...
- 2546 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Raj Sharma​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 2254 Views
- 2 replies
- 1 kudos
Resolved! How to load multiple xlsx that are storage in different folders with the same name in a blob storage in a delta table ?
Hi i have a blob storage with multile unzip folders with the same suffix folder_report_name_01_2023_01_02 -> file_name_2023_01_02.xlsxBut i want to load all of this data using pandas or pyspark and insert in my delta table.I'm trying to using widget...
- 2254 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Fernando Vázquez​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 1 kudos
- 2688 Views
- 2 replies
- 2 kudos
Why Azure Databricks needs to store data in temp storage in Azure before writing to the synapse.
I was following the tutorial about data transformation with azure databricks, and it says before loading data into azure synapse analytics, the data transformed by azure databricks would be saved on temp storage in azure blob storage first before loa...
- 2688 Views
- 2 replies
- 2 kudos
- 2 kudos
@Ajay Pandey​ Saving the transformed data to temporary storage in Azure Blob Storage before loading into Azure Synapse Analytics provides a number of benefits to ensure that the data is accurate, optimized, and performs well in the target environmen...
- 2 kudos
- 1354 Views
- 1 replies
- 0 kudos
What is the upper bound limit for dataSkippingNumIndexedCols, to keeps stats in delta log file?
Is there an upper bound of number that i can assign to delta.dataSkippingNumIndexedCols for computing statistics. Is there some tradeoff benchmark available for increasing this number beyond 32.
- 1354 Views
- 1 replies
- 0 kudos
- 0 kudos
@Chhavi Bansal​ :The delta.dataSkippingNumIndexedCols configuration property controls the maximum number of columns that Delta Lake will build statistics on during data skipping. By default, this value is set to 32. There is no hard upper bound on th...
- 0 kudos
- 6766 Views
- 1 replies
- 1 kudos
how can i export my dashboard en format html using databriks api
hi everyone, i would like to export my dashbord in html format and embed it in my body of my email in order to send it to my teamso there is my code python for the databriks api and i got this error and when i put my htm in the body of my message i...
- 6766 Views
- 1 replies
- 1 kudos
- 1 kudos
@mathild noun​ :import databricks.workspace as workspace_api import requests # set up your Databricks workspace credentials domain = "<your Databricks workspace domain>" token = "<your Databricks API token>" # set up the workspace client workspac...
- 1 kudos
- 11262 Views
- 1 replies
- 2 kudos
How does hash shuffle join work in Spark?
Hi All, I am trying to understand the internals shuffle hash join. I want to check if my understanding of it is correct. Let’s say I have two tables t1 and t2 joined on column country (8 distinct values). If I set the number of shuffle partitions as ...
- 11262 Views
- 1 replies
- 2 kudos
- 2 kudos
@Vinay Emmadi​ : In Spark, a hash shuffle join is a type of join that is used when joining two data sets on a common key. The data is first partitioned based on the join key, and then each partition is shuffled and sent to a node in the cluster. The ...
- 2 kudos
- 4259 Views
- 1 replies
- 1 kudos
Save Spark DataFrame to shape file (.shp format)
Hello,I know how to create .shp file from Geopandas dataframe using code similar to this, also mentioned on SO:gpd_df = geopandas.GeoDataFrame(pandas_df, geometry='geom') gpd_df .to_file("username/nh.shp")However I have .parquet files that I can load...
- 4259 Views
- 1 replies
- 1 kudos
- 1 kudos
@Bartosz Maciejewski​ :Spark does not have native support for writing Shapefiles directly. However, you can use a third-party library such as GeoPandas or PyShp to write your Spark DataFrame to a Shapefile.Here's an example of how to use GeoPandas to...
- 1 kudos
- 7105 Views
- 1 replies
- 4 kudos
Resolved! Query related to Storage account authentication
Use Case: Copy data from SharePoint List to Blob using Power AutomateShort Description:To Access the blob storage account from Power Automate. There are three authentication type:1. Access Key2. Service Principal3. Azure AD IntegratedWhich authentica...
- 7105 Views
- 1 replies
- 4 kudos
- 4 kudos
@KVNARK .​ :It's recommended to use the Azure AD Integrated authentication type. This authentication type allows you to use Azure Active Directory (AD) to authenticate and manage access to Blob Storage resources at the folder or container level using...
- 4 kudos
- 1345 Views
- 0 replies
- 2 kudos
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Ma...
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to shar...
- 1345 Views
- 0 replies
- 2 kudos
- 2884 Views
- 2 replies
- 1 kudos
How to download a file in DBFS that contains multibyte characters in the file path?
I would like to download a file in DBFS using the FileStore Endpoint.If the file or folder name contains multibyte characters, the file path cannot be specified due to URL encoding and an error occurs.Question 1: If a file or folder name contains mul...
- 2884 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi,Databricks CLI can be used to download a file from DBFS. https://docs.databricks.com/dev-tools/cli/index.htmlAlso, you can refer to https://stackoverflow.com/questions/49019706/databricks-download-a-dbfs-filestore-file-to-my-local-machine , which ...
- 1 kudos
- 2326 Views
- 0 replies
- 4 kudos
lnkd.in
Databricks has introduced a new feature that allows users to send SQL statements to their database via REST API. Users can easily integrate this feature with any tool by simply posting their queries to the /api/2.0/sql/statements/ endpoint. With this...
- 2326 Views
- 0 replies
- 4 kudos
- 3440 Views
- 2 replies
- 5 kudos
Resolved! Databricks SQL External Connections
Lakehouse architectures seem enticing, especially from the standpoint of querying the data lake directly as it sits (as opposed to first migrating the data to an external data warehouse). While documentation and support seems pretty clear regarding ...
- 3440 Views
- 2 replies
- 5 kudos
| User | Count |
|---|---|
| 1615 | |
| 779 | |
| 349 | |
| 287 | |
| 267 |
