- 542 Views
- 7 replies
- 0 kudos
jdbc integration returning header as data for read operation
package com.example.databricks; import org.apache.spark.sql.Dataset;import org.apache.spark.sql.Row;import org.apache.spark.sql.SparkSession; public class DatabricksJDBCApp { public static void main(String[] args) { // Initialize Spark Ses...
- 542 Views
- 7 replies
- 0 kudos
- 0 kudos
After reading through the Driver documentation I've finally found a solution that appears to work for me. I've added .option("UseNativeQuery", 0) to my JDBC connection. The query that was being passed from the Databricks Driver to the Databricks Clus...
- 0 kudos
- 853 Views
- 1 replies
- 0 kudos
Migrating from Snowflake to Databricks
How are you offloading data from Snowflake & repointing raw S3 staging dumps to Parquet/Delta?
- 853 Views
- 1 replies
- 0 kudos
- 0 kudos
@Tmm35 use Travinto Technologies tool that may help you. if you want to migrate etl, database and sql to databrick or any , their tool is very good.
- 0 kudos
- 1866 Views
- 2 replies
- 2 kudos
Migration Teradata to Databricks
We are actually working in the migration of Teradata to Databricks.And i want to know how can i replace the Teradata connector in Datastage, to send the data to azure datalake using mft ? Thank you for your help.
- 1866 Views
- 2 replies
- 2 kudos
- 2 kudos
@OthmaneH use Travinto Technologies tool that may help you. we have done using their tool more that 100+ sources to databricks migration with etl, database and sql.
- 2 kudos
- 1111 Views
- 1 replies
- 0 kudos
Data Migration
I am looking for a solution for Data Migration.Want to connect SFTP Server to Databricks to New Platform(Don't Want ADF in between)Is it possible?
- 1111 Views
- 1 replies
- 0 kudos
- 0 kudos
@Jay_rockstar use Travinto Technologies tool.
- 0 kudos
- 161 Views
- 3 replies
- 0 kudos
using VSCode extension to interact with Databricks
Hello community, I want to understand if it is possible to use Databricks Connect inside VSCode IDE to interact with Notebooks in local interactively like in Databricks Notebook, Is it possible? Because now I can only use the cluster and wait after t...
- 161 Views
- 3 replies
- 0 kudos
- 157 Views
- 2 replies
- 0 kudos
How to Set Expiration Time for Delta Sharing URL in Databricks Using Terraform?
I am automating Delta Sharing from Databricks to non-Databricks recipients using Terraform. I can successfully create shares and recipients with my Terraform code, retrieve the sharing URL after creating the recipient, and see that the URL gets a de...
- 157 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, I did it yesterday through account console (idk if you can do it using terraform).If you are an admin at higher level you can go in that window and enable your metastore to set a token with an expiration date. I hope I answer to your problem
- 0 kudos
- 62 Views
- 1 replies
- 0 kudos
extracting attachments from outlook
Can we fetch attachments from outlook in databricks?
- 62 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi s3You could use Microsoft graph for that. Here is an example: https://learn.microsoft.com/en-us/answers/questions/1631663/using-graph-api-to-retrieve-emailAnother way I have always done this is through the Logic App. It is pretty easy to set up an...
- 0 kudos
- 170 Views
- 2 replies
- 0 kudos
Update set in foreachBatch
I need to track codes of records that were ingested in foreachBatch function, and pass it as a task value, so downstream tasks can take actions based on this output. What would be the best approach to achieve that? Now, I have a following solution, b...
- 170 Views
- 2 replies
- 0 kudos
- 0 kudos
@skarpeck does your input df contain any filters? The empty codes variable could be due to empty microbatches maybe. Please check the numInputRows from your query's Stream Monitoring Metrics. I recommend you to check if there are input rows for the b...
- 0 kudos
- 29917 Views
- 10 replies
- 10 kudos
Failure starting repl. How to resolve this error? I got this error in a job which is running.
Failure starting repl. Try detaching and re-attaching the notebook.java.lang.Exception: Python repl did not start in 30 seconds. at com.databricks.backend.daemon.driver.IpykernelUtils$.startIpyKernel(JupyterDriverLocal.scala:1442) at com.databricks.b...
- 29917 Views
- 10 replies
- 10 kudos
- 10 kudos
I have had this problem many times, today I made a copy of the cluster and it got "de-saturated", it could help someone in the future
- 10 kudos
- 186 Views
- 1 replies
- 0 kudos
Resolved! How do Databricks notebooks differ from traditional Jupyter notebooks
Can someone please explain the key difference between a Databricks notebook and a Jupyter notebook.
- 186 Views
- 1 replies
- 0 kudos
- 0 kudos
The key differences between a Databricks notebook and a Jupyter notebook are as follows: Integration and Collaboration: Databricks Notebooks: These are integrated within the Databricks platform, providing a unified experience for data science and ma...
- 0 kudos
- 482 Views
- 5 replies
- 1 kudos
Resolved! Does column masking work with job clusters
Hi,We are trying to experiment with the column masking feature.Here is our use case:We have added a masking function to one of the columns of a tablethe table is part of a notebook with some transformation logicthe notebook is executed as part of a w...
- 482 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello, the shared cluster on a job will act the same as in an all purpose cluster, basically means that the cluster will be available for any user with permissions to it, in a job there will not be much actions to be done but when an action you are r...
- 1 kudos
- 166 Views
- 1 replies
- 0 kudos
Resolved! cloning the data between two catalogs
Hello community,I was writing this piece of code to do the data migration between two catalogs: # Read data and partitioning print(f"Loading {table_name} from production catalog...") prod_df_table_name = f"prod_catalog.`00_bro...
- 166 Views
- 1 replies
- 0 kudos
- 0 kudos
FYI,I did it increasing the size of the cluster using much cores and directly written:prod_df_table.write \.format("delta") \.mode("overwrite") \.saveAsTable(stg_df_table_name)
- 0 kudos
- 449 Views
- 6 replies
- 2 kudos
Resolved! use include property specified for a particular workspace using DABs
Hello, community,Is there a field in the YAML file used with DABs to specify files based on the workspace in use? For example, if I want to deploy notebooks and workflows for staging, they need to be a set of resources that differ from those in produ...
- 449 Views
- 6 replies
- 2 kudos
- 2 kudos
Yes, you can specify different sets of resources for different environments (such as staging and production) in the YAML file used with Databricks Asset Bundles (DABs). This is achieved using the targets mapping in the databricks.yml file.https://doc...
- 2 kudos
- 5451 Views
- 2 replies
- 3 kudos
Unity Catalog + Medallion Architecture
I am looking for a reference architecture or an example on how to organize unity catalog while adhering to the medallion architecture.What are some common naming conventions and methods?How to you isolate environments (dev/prod)?I was thinking of som...
- 5451 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi Shane,Could you share what you ended up doing in your scenario. I have similar requirements and would like to understand how you implemented yours Saurabh
- 3 kudos
- 330 Views
- 2 replies
- 0 kudos
Resolved! querying snowflake database using databricks query federation: no active warehouse
Hello Databricks community,I'm confused right now because I was able to query snowflake table using query federation 2 days ago. But now it's giving me error about no active warehouse:Status of query associated with resultSet is FAILED_WITH_ERROR. No...
- 330 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello@Alberto_Umana Thank you very much for you response.I was able to solve it on my side. The issue was lying on snowflake side. I realized I had to not only grant USAGE permission but also OPERATE permission to my snowflake account. Also added sfR...
- 0 kudos