I am following this guide on writing data to the BigQuery table.Right now, I have an error when I try to write data using Databricks Secret instead of the JSON credential file and setting the GOOGLE_APPLICATION_CREDENTIALS environment variable. java....
This is partly a question, partly a feature request: How do you guys handle streaming checkpoints in combination with unity catalog managed tables?It seems like the only way is to create a volume, and manually specify paths in it as streaming checkpo...
For Structured Streaming Applications, this would be a nice feature. Delta Live Tables manages checkpoints for you out of the box - you don't even have to reason about checkpoints at all, would recommend checking it out!
Hi Team,I have a requirement to trigger a databricks job more than once daily, may be twice or thrice daily.I have checked the workflows but I couldn't find any option in the UI.Please advice
Ak chcete nájsť informácie o najlepšie zahraničné online hry bonusy bez vkladu, odporúčam navštíviť stránku s recenziami online hry . Tento zdroj poskytuje podrobné recenzie najlepších zahraničných online hry vrátane informácií o bonusoch bez vkladu,...
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
Hi everyone,I’m trying to integrate Spotify API to get an authentication token, but I’m encountering an issue with the error message: "raw_input was called, but this frontend does not support input requests." Has anyone else experienced this while wo...
Hi Community,I have successfully run a job through the API but would need to be able to pass parameters (configuration) to the DLT workflow via the APII have tried passing JSON in this format:{ "full_refresh": "true", "configuration": [ ...
You cannot pass parameters from a Databricks job to a DLT pipeline. Atleast not yet. You can see from the DLT rest API that there is no option for it to accept any parameters.But there is a workaround.But there is a workaround.With the assumption tha...
Hello,I have the issue that even a query like "select 1" is not finishing. The sql warehouse runs infinite. I have no idea where to look for any issues because in the SPARK UI I cant see any error.What is intresting is that also allpurpose clusters (...
Hi!I have been carrying out a POC, so I created the CSV file in my workspace and tried to read the content using the techniques below in a Python notebook, but did not work.Option1:repo_file = "/Workspace/Users/u1@org.com/csv files/f1.csv"tmp_file_na...
for unity catalog enabled clusters, with the default security permmissions, I think we cannot access like this anymore.
Can i set column level and table level tags programmatically in a dlt pipeline. I tried it the normal way using spark.sql(f"alter table and set tags (key=value)") using this syntax I found it out in one of the databricks community post. But can we do...
@ashraf1395 wrote:Can i set column level and table level tags programmatically in a dlt pipeline. I tried it the normal way using spark.sql(f"alter table and set tags (key=value)") using this syntax I found it out in one of the databricks community p...
Hi time travel gurus,I am investigating creating a reporting solution with an AsOf functionality. Users will be able to create a report based on the current data or on the data AsOf some time ago. Due to the nature of our data this AsOf feature is qu...
In Databricks, you can use time travel to access historical versions of a table using the versionAsOf or timestampAsOf options in a SELECT query. To retrieve all changes made to a table, you would typically query the table's historical versions, spec...
I want to create empty streaming tables in dlt with only schema specified. Is it possible ?I want to do it in dlt python.
it is possible to create empty streaming tables in Delta Live Tables (DLT) with only the schema specified using the DLT Python API. Here is how you can do it: Import the DLT module:import dlt Define the schema:You can specify the schema using a P...
I'm trying to connect to NetSuite2.com using Pyspark from a Databricks Notebook utilizing a JDBC driver.I was successful in setting up my DBVisualizer connection by installing the JDBC Driver (JAR) and generating the password with the one-time hashin...
It seems like the issue might be related to the password generation or the JDBC URL configuration. Here are a few things you can check: NextCareurgentcare1. Password Generation: Ensure that the generate_tba_password function is correctly implemented ...
Dear All,Is it possible to create Materialized view on view and table (Joining view and table)?I suspect it is not possible. Please suggest.Also please provide best way to schedule the refresh of Materialized view. Regards,Surya
Hello @sboxi Can you try with this, as it looks it is possible, also about scheduling, please refer to below documentation you can use SCHEDULE clause. CREATE MATERIALIZED VIEW IF NOT EXISTS my_materialized_view AS SELECT t.column1, v.co...
Hello I am trying to send multiple post requests to an endpoint, i have a spark dataframe and each column of this dataframe is sent through the payload of the post request.However when i run this in my notebook, no exception is raised. I'm guessing i...
package com.example.databricks; import org.apache.spark.sql.Dataset;import org.apache.spark.sql.Row;import org.apache.spark.sql.SparkSession; public class DatabricksJDBCApp { public static void main(String[] args) { // Initialize Spark Ses...
After reading through the Driver documentation I've finally found a solution that appears to work for me. I've added .option("UseNativeQuery", 0) to my JDBC connection. The query that was being passed from the Databricks Driver to the Databricks Clus...
I want to migrate my data as well as the transformations i have to convert my raw data into BI data. Is there a way i can move those transformation to databricks? Some of the transformations use native snowflake functions.Thanks in advance.
@ayush_273 use Travinto Technologies tool that may help you. if you want to migrate etl, database and sql to databrick or any , their tool is very good.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group| User | Count |
|---|---|
| 1606 | |
| 761 | |
| 345 | |
| 285 | |
| 251 |
