To read Bigquery data using spark.read, i'm using a query. This query executes and creates a table on the materializationDataset. df = spark.read.format("bigquery") \.option("query", query) \.option("materializationProject", materializationProject) \...
Hello,We are using a reference dataset for our Production applications. We would like to create a delta table for this dataset to be used from our applications. Currently, manual updates will occur on this dataset through a script on a weekly basis. ...
+1 for ADLS. Hierarchical storage, hot/cold/premium storage, things not possible in blob storage
Hi, i have several questions regarding Trino integration:Is it recommended to use an external Hive metastore or leverage on the databricks-maintained Hive metastore when it comes to enabling external query engines such as Trino?When I tried to use ex...
> Is it recommended to use an external Hive metastore or leverage on the databricks-maintained Hive metastore when it comes to enabling external query engines such as Trino?Databricks maintained hive metastore is not suggested to be used externally. ...
I’m trying to implement a streaming pipeline that will run hourly using Spark Structured Streaming, Scala and Delta tables. The pipeline will process different items with their details.The source are delta tables that already exists, written hourly u...
@Agus1 Could you try using CDC in delta. You could use readChangeFeed to read only the changes that got applied on the source table. This is also explained here.https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed
When changing a delta table column data type in Unity Catalog, we noticed a view that is referencing that table did not automatically update to reflect the new data type.Is there a way to update the delta table column data type so that it also update...
Can you try refreshing the view by running the command: REFRESH TABLE <viewname>
I have an auto loader streaming job with the continuous run, I want to stop that job on weekend for some time and restart again.
Facing an issue with cluster performance, in event log can see - cluster is not responsive likely due to GC. Number of pipeline (databricks notebooks) running and cluster configuration is same as it used to be before but started seeing this issue sin...
Hi @Vibhor Sethi​ ,Do you see any other error messages? did you data volume increase? what kind of job are you running?
Hi There,Looking for recommendations on how to optimize history data tables in databricks?
Thanks Kaniz!
How I can connect sftp server from databricks. So I can write files into tables directly?
The classic solution is to copy data from FTP to ADLS storage using Azure Data Factory, and after the copy is done in the ADF pipeline, trigger the databricks notebook.
While going to give Databricks lakehouse fundamental certification quizIt is showing that you don't have permission to access this. Could you please guide me on that or can provide a link through which I can get permission to access this and complet...
Hi @Souvikng, Here is the link for the Accrediition.
Hello,I am trying to load logs from spark job in remote location using Python Rest API.I want to collect those logs for particular job runs using runID field, that log should contain Errors, Exceptions, and print details.I have tried "/api/2.1/jobs/r...
Hello,How can I access endpoint (http URL) of spark history server for a particular cluster? Thanks,Parimal
Hello, I am trying to delete a folder in /dbfs/mnt, but I am unable to do so. The folder was an old mounted storage account, which was deleted. The folder contains a single file 'mount.err'.In command line I tried rm -rf my_folder, with and without s...
Hi @erigaud The issue you're facing might be due to the fact that the folder was a mounted storage account that has been deleted. This could cause some inconsistencies in the file system view and hence, you're unable to delete the folder.You can try...
Is it possible to run a DBT job using Jobs Compute? If not, is there a reason for that, and is there any plans to change that?
Hello @brendanc19 ,Running DBT with Jobs compute cluster is actually supportedYou will need to create a job task with type DBTKindly check documentation here for step by step on how to run DBT transformation Databricks jobsRegards
I have this error when trying to clone a repo, using aos sparse checkout mode. Error creating repoGit operation is terminated because of out of memory or CPU, possibly the repository is too big. Need help? See our Repos Limitations and FAQs docs How...
From the error message, you could exceed below limit :Working branches are limited to 200 MB.Individual files are limited to 200 MB.Files larger than 10 MB can’t be viewed in the Databricks UI.Databricks recommends that in a repo:The total number of ...
I have an continuous running streaming Job, I would like to stop this over weekend and start again on Monday. Here is my streaming job code.(spark.readStream.format("delta").load(input_path).writeStream.option("checkpointLocation", input_checkpoint_p...
@sanjay Any luck on that, I am also looking for the solution for the same issue
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
