- 3717 Views
- 5 replies
- 0 kudos
Internal Errors in Databricks Queries via Go Library.
While executing a query directly from the Databricks editor for the entire table, no errors are observed. However, when querying through the Go library, we encounter an internal error.Your guidance on this issue would be greatly appreciated.
- 3717 Views
- 5 replies
- 0 kudos
- 0 kudos
@Vetrivel wrote:While executing a query directly from the Databricks editor for the entire table, no errors are observed. However, when querying through the Go library, we encounter an internal error.Your guidance on this issue would be greatly appre...
- 0 kudos
- 2163 Views
- 2 replies
- 1 kudos
Connect a SFTP to Databricks
What would be the best way to set up a connection to a sftp server from Databricks? In Jupyter Lab, this can done from terminal. Of coruse, there are alternatives like using paramiko library. But is there perhaps a more Databricks-ish solution?
- 2163 Views
- 2 replies
- 1 kudos
- 2137 Views
- 2 replies
- 0 kudos
Databricks Tables in Excel
I attended the Databricks World Tour in 2023 (not the most recent one) where I remember there was mention of a feature in development that was an add-in (or similar) for Excel that would allow for native connection and importation of a Databricks tab...
- 2137 Views
- 2 replies
- 0 kudos
- 0 kudos
@haydenLiQ -Power BI has native support for Databricks connections via the Azure Databricks connector. You can pull data from Databricks into Power BI, create datasets, and then use Power BI’s Excel integration to export these datasets into Excel. C...
- 0 kudos
- 20509 Views
- 4 replies
- 2 kudos
Resolved! Why is mounts = dbutils.fs.mounts () not available now?
Hi there,I am currently using cluster version 15.4 LTS with UC enabled . Azure Data Lake Storage Gen2 has enabled hierarchical namespaces.I tried the following three ways to mount external storage and all got an errorMount point via the ADLS Gen2 acc...
- 20509 Views
- 4 replies
- 2 kudos
- 2 kudos
@zmsoft Unity Catalog (UC) enforces strict access control policies, and traditional mounting techniques—such as using access keys or the dbutils.fs.mount command—are not recommended. Best practices for DBFS and Unity Catalog.Databricks advises agains...
- 2 kudos
- 2878 Views
- 3 replies
- 1 kudos
Databricks overwrite didn't delete previous data
Hi databricks, we met an issue like below picture shows:we use pyspark api to store data into ADLS :df.write.partitionBy("xx").option("partitionOverwriteMode","dynamic").mode("overwrite").parquet(xx)However, not sure why the second time we overwrite ...
- 2878 Views
- 3 replies
- 1 kudos
- 1 kudos
@阳光彩虹小白马The issue you're encountering seems to involve inconsistent behavior in partition overwrites using PySpark with ADLS.Can you validate the below along with what @Himanshu6 mentioned.Force Spark to refresh the metadata of the data lake director...
- 1 kudos
- 6958 Views
- 5 replies
- 1 kudos
Resolved! Databricks Workspace import api size limitation
As libraries from dbfs is deprecated and support is going to be removed. We are moving libs from dbfs to workspace files.But while uploading libraries using https://docs.databricks.com/api/azure/workspace/workspace/import api there seems to be limit ...
- 6958 Views
- 5 replies
- 1 kudos
- 1 kudos
Can we use this API to upload files to /workspace/ location? Any sample curl request ? I am using -X PUT https://<HOST_NAME>/api/2.0/fs/files/Workspace/Shared/jars/all.jar and getting { "error_code" : "BAD_REQUEST", "message" : "Invalid path:", "d...
- 1 kudos
- 3474 Views
- 2 replies
- 4 kudos
Read Geojson file using Sedona Context in Databricks
Hi Everyone,I am trying to read a geojson file in Databricks using the following syntax as mentioned in the apache sedona official docs Load Geojson Data I am using Sedon 1.6.1 Version which supports this feature but i am getting an error as mentione...
- 3474 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Kunal_Mishra ,The error you are experiencing with Sedona when trying to read a GeoJSON file in Databricks (java.lang.NoSuchMethodError) often indicates a compatibility issue between the Spark version you're using and the Sedona library.Sedona has...
- 4 kudos
- 1480 Views
- 2 replies
- 0 kudos
How to load single line mode json file?
Hi there,The activity log store in adls gen2 container is a single line mode json file.How to load single line mode json file, save data to delta table? Thanks & Regards,zmsoft
- 1480 Views
- 2 replies
- 0 kudos
- 0 kudos
@zmsoft Since the JSON is a single-line file, ensure it is being read correctly. Try setting the multiLine option to false (it defaults to false, but explicitly setting it ensures correct handling). stageDf = ( spark.read.format("json") .opti...
- 0 kudos
- 2798 Views
- 2 replies
- 1 kudos
Resolved! what api calls does autoloader make on s3?
Hey guys I'm trying to find the estimate for an ingestion pipeline that uses autoloader on an S3 bucket every 2 minutes.I found the pricing for s3 bucket api consumption but I am not certain what api calls will autoloader make.Talking to chatGPT it t...
- 2798 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @SakuraDev1 ,LIST and GET make sense.How autoloader works is monitors a specified location and then if the new file is discovered, it is being processed to bronze table.So a LIST request is needed to check the files in the source directory, and th...
- 1 kudos
- 1255 Views
- 1 replies
- 0 kudos
Mongo server to Delta Live Tables
I want to fetch data from mongodb server and then show that in delta live table in gcp.
- 1255 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Rutuja_3641 Have a look here: https://docs.databricks.com/en/connect/external-systems/mongodb.htmlI think you can easily adapt the code to DLT
- 0 kudos
- 2704 Views
- 2 replies
- 0 kudos
Can't create table in unity catalog.
Hi all.I have created a Databricks workspace on AWS. I can log into the workspace and successfully perform select operations on files in S3, but I am unable to create tables.The error when creating the table is as follows. "Your request failed with s...
- 2704 Views
- 2 replies
- 0 kudos
- 8018 Views
- 5 replies
- 4 kudos
Read and saving Blob data from oracle to databricks S3 is slow
I am trying to import a table from oracle which has around 1.3 mill rows and one of the column is a Blob, the total size of data on oracle is around 250+ GB. read and save to S3 as delta table is taking around 60 min. I tried with parallel(200 thread...
- 8018 Views
- 5 replies
- 4 kudos
- 4 kudos
Any update on this topic what should be the best option to read from oracle and write in ADLS.
- 4 kudos
- 1584 Views
- 1 replies
- 0 kudos
Lakehouse monitoring
I created snapshot and time series lake house monitoring on 2 different tables. After execution, metrics table got created and dashboard also got created but monitoring data is not populating in dashboard and metrics tables even after refresh of moni...
- 1584 Views
- 1 replies
- 0 kudos
- 0 kudos
Do you refreshed the metrics?Table -> Quality -> Refresh metrics
- 0 kudos
- 8877 Views
- 5 replies
- 0 kudos
Structured Streaming: How to handle Schema Changes in source and target Delta Table
Hey Community,we have a streaming pipeline, starting with autoloader to ingest data into a bronze table and this data gets then picked up by another streaming job that transforms this data and writes into a silver table.Now there are some schema chan...
- 8877 Views
- 5 replies
- 0 kudos
- 0 kudos
I adjusted the schema in both bronze and silver, such that I do not need schema evolution. The problem is that the DataStreamReader does not pick up the schema changes in bronze.I already figured out that it has something to do with providing also a ...
- 0 kudos
- 2799 Views
- 1 replies
- 0 kudos
Resolved! With the JDBC Driver, ResultSet metadata gives the column type as column name
Hello,Using the JDBC Driver , when I check the metadata of a result set, when retrieving the column name, it's the column type that is given . Here what it looks like in the meta data . Using version 2.6.40, with the query SELECT "id" string, "year" ...
- 2799 Views
- 1 replies
- 0 kudos
- 0 kudos
forget about it, the select clause is wrong of course .. too tired I guess sorry
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
