- 2421 Views
- 1 replies
- 1 kudos
DLT x UC x Auto Loader
Now that the Directory Listing Mode of Auto Loader is officially deprecated, is there a solution for using File Notification Mode in a DLT pipeline writing to a UC-managed table? My understanding is that File Notification Mode is only available on si...
- 2421 Views
- 1 replies
- 1 kudos
- 1 kudos
I am having the same concern and am reaching out to our Solutions Architect to better understand how AutoLoader &DLT can be used. DLT and AutoLoader should go hand in hand especially when using file notification mode.
- 1 kudos
- 34273 Views
- 8 replies
- 8 kudos
Resolved! How to merge small parquet files into a single parquet file?
I have thousands of parquet files having same schema and each has 1 or more records. But reading with spark these files is very very slow. I want to know if there is any solution how to merge the files before reading them with spark? Or is there any ...
- 34273 Views
- 8 replies
- 8 kudos
- 8 kudos
We can combine all these small parquet files into single file using optimize command..Optimize delta_table_name
- 8 kudos
- 1128 Views
- 3 replies
- 7 kudos
Cant find DBFS File Browser option in community edition settings
Hi,I was able to access DBFS, but I can no longer find the DBFS File Browser option in the settings. I used it two days ago, but now it's missing.Please help.Thanks,Shashi
- 1128 Views
- 3 replies
- 7 kudos
- 7 kudos
Thanks for the update @szymon_dybczak . It's reassuring to know that others are experiencing the same issue. I'll keep an eye out for any official announcements regarding the change.
- 7 kudos
- 1536 Views
- 5 replies
- 0 kudos
Internal Errors in Databricks Queries via Go Library.
While executing a query directly from the Databricks editor for the entire table, no errors are observed. However, when querying through the Go library, we encounter an internal error.Your guidance on this issue would be greatly appreciated.
- 1536 Views
- 5 replies
- 0 kudos
- 0 kudos
@Vetrivel wrote:While executing a query directly from the Databricks editor for the entire table, no errors are observed. However, when querying through the Go library, we encounter an internal error.Your guidance on this issue would be greatly appre...
- 0 kudos
- 1116 Views
- 2 replies
- 1 kudos
Connect a SFTP to Databricks
What would be the best way to set up a connection to a sftp server from Databricks? In Jupyter Lab, this can done from terminal. Of coruse, there are alternatives like using paramiko library. But is there perhaps a more Databricks-ish solution?
- 1116 Views
- 2 replies
- 1 kudos
- 790 Views
- 2 replies
- 0 kudos
Databricks Tables in Excel
I attended the Databricks World Tour in 2023 (not the most recent one) where I remember there was mention of a feature in development that was an add-in (or similar) for Excel that would allow for native connection and importation of a Databricks tab...
- 790 Views
- 2 replies
- 0 kudos
- 0 kudos
@haydenLiQ -Power BI has native support for Databricks connections via the Azure Databricks connector. You can pull data from Databricks into Power BI, create datasets, and then use Power BI’s Excel integration to export these datasets into Excel. C...
- 0 kudos
- 15140 Views
- 4 replies
- 2 kudos
Resolved! Why is mounts = dbutils.fs.mounts () not available now?
Hi there,I am currently using cluster version 15.4 LTS with UC enabled . Azure Data Lake Storage Gen2 has enabled hierarchical namespaces.I tried the following three ways to mount external storage and all got an errorMount point via the ADLS Gen2 acc...
- 15140 Views
- 4 replies
- 2 kudos
- 2 kudos
@zmsoft Unity Catalog (UC) enforces strict access control policies, and traditional mounting techniques—such as using access keys or the dbutils.fs.mount command—are not recommended. Best practices for DBFS and Unity Catalog.Databricks advises agains...
- 2 kudos
- 1429 Views
- 3 replies
- 1 kudos
Databricks overwrite didn't delete previous data
Hi databricks, we met an issue like below picture shows:we use pyspark api to store data into ADLS :df.write.partitionBy("xx").option("partitionOverwriteMode","dynamic").mode("overwrite").parquet(xx)However, not sure why the second time we overwrite ...
- 1429 Views
- 3 replies
- 1 kudos
- 1 kudos
@阳光彩虹小白马The issue you're encountering seems to involve inconsistent behavior in partition overwrites using PySpark with ADLS.Can you validate the below along with what @Himanshu6 mentioned.Force Spark to refresh the metadata of the data lake director...
- 1 kudos
- 3163 Views
- 5 replies
- 1 kudos
Resolved! Databricks Workspace import api size limitation
As libraries from dbfs is deprecated and support is going to be removed. We are moving libs from dbfs to workspace files.But while uploading libraries using https://docs.databricks.com/api/azure/workspace/workspace/import api there seems to be limit ...
- 3163 Views
- 5 replies
- 1 kudos
- 1 kudos
Can we use this API to upload files to /workspace/ location? Any sample curl request ? I am using -X PUT https://<HOST_NAME>/api/2.0/fs/files/Workspace/Shared/jars/all.jar and getting { "error_code" : "BAD_REQUEST", "message" : "Invalid path:", "d...
- 1 kudos
- 1689 Views
- 2 replies
- 3 kudos
Read Geojson file using Sedona Context in Databricks
Hi Everyone,I am trying to read a geojson file in Databricks using the following syntax as mentioned in the apache sedona official docs Load Geojson Data I am using Sedon 1.6.1 Version which supports this feature but i am getting an error as mentione...
- 1689 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Kunal_Mishra ,The error you are experiencing with Sedona when trying to read a GeoJSON file in Databricks (java.lang.NoSuchMethodError) often indicates a compatibility issue between the Spark version you're using and the Sedona library.Sedona has...
- 3 kudos
- 722 Views
- 2 replies
- 0 kudos
How to load single line mode json file?
Hi there,The activity log store in adls gen2 container is a single line mode json file.How to load single line mode json file, save data to delta table? Thanks & Regards,zmsoft
- 722 Views
- 2 replies
- 0 kudos
- 0 kudos
@zmsoft Since the JSON is a single-line file, ensure it is being read correctly. Try setting the multiLine option to false (it defaults to false, but explicitly setting it ensures correct handling). stageDf = ( spark.read.format("json") .opti...
- 0 kudos
- 1822 Views
- 2 replies
- 1 kudos
Resolved! what api calls does autoloader make on s3?
Hey guys I'm trying to find the estimate for an ingestion pipeline that uses autoloader on an S3 bucket every 2 minutes.I found the pricing for s3 bucket api consumption but I am not certain what api calls will autoloader make.Talking to chatGPT it t...
- 1822 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @SakuraDev1 ,LIST and GET make sense.How autoloader works is monitors a specified location and then if the new file is discovered, it is being processed to bronze table.So a LIST request is needed to check the files in the source directory, and th...
- 1 kudos
- 845 Views
- 1 replies
- 0 kudos
Exploring Postman Alternatives for API Testing in VSCode?
Has anyone here explored Postman alternatives within VSCode? I’ve seen mentions of Thunder Client and Apidog. Would love to know if they offer a smoother integration or better functionality.
- 845 Views
- 1 replies
- 0 kudos
- 0 kudos
HiI'm using ThunderClient as VS Code extension: https://www.thunderclient.com/ The functionality in the Free version is okay. If you want more features, there are plans for some bucks.
- 0 kudos
- 615 Views
- 1 replies
- 0 kudos
Mongo server to Delta Live Tables
I want to fetch data from mongodb server and then show that in delta live table in gcp.
- 615 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Rutuja_3641 Have a look here: https://docs.databricks.com/en/connect/external-systems/mongodb.htmlI think you can easily adapt the code to DLT
- 0 kudos
- 1547 Views
- 2 replies
- 0 kudos
Can't create table in unity catalog.
Hi all.I have created a Databricks workspace on AWS. I can log into the workspace and successfully perform select operations on files in S3, but I am unable to create tables.The error when creating the table is as follows. "Your request failed with s...
- 1547 Views
- 2 replies
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
