cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Upendra_Dwivedi
by New Contributor II
  • 103 Views
  • 3 replies
  • 0 kudos

ON-Prem SQL Server Direct Connection with Azure Databricks

Hi All,I have SSMS installed and have some data there which i want to export to Databricks and process it there. Databricks is hosted on Azure Cloud and I am wondering if it is possible. I have tested it using JDBC connection but i am getting error: ...

  • 103 Views
  • 3 replies
  • 0 kudos
Latest Reply
MariuszK
Contributor III
  • 0 kudos

If you want to play with SQL Server the easy way is to use Azure SQL that will be visuable for Databricks. Alternativly you can use ADF with slef-hosted run time to extract data to Azure from SQL Server.Network configuration requires many steps and i...

  • 0 kudos
2 More Replies
Mumrel
by Contributor
  • 6472 Views
  • 5 replies
  • 1 kudos

Resolved! Usage of forum: How to find the threads where I commented on and my bookmarks

Hi quick questions, 1) the other day I commented on a thread and cannot find it. Is there a feature to find all my posts? I cannot find it under my profile.2) I find interesting threads and want to watch (in the sense: email me when updates occure) t...

  • 6472 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Jan St.​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...

  • 1 kudos
4 More Replies
Prasanna_N
by New Contributor
  • 36 Views
  • 0 replies
  • 0 kudos

Inference table Monitoring

i have data from march1 to march 14 in the final inference table and i have given 1 week granularity. after that profile and drift table is generated and i see the window start time as like this objectstart: "2025-02-24T00:00:00.000Z"end: "2025-03-03...

  • 36 Views
  • 0 replies
  • 0 kudos
Danny_Lee
by Valued Contributor
  • 62 Views
  • 0 replies
  • 1 kudos

re: Welcoming Bladebridge to Databricks!

Hi @Sujitha and Databricks team,Congrats on the acquisition of Bladebridge.  We used this tool a couple years back to migrate an important ETL process from Informatica.  I'm glad to see its part of the Data Intelligence Platform and have already take...

  • 62 Views
  • 0 replies
  • 1 kudos
databicky
by Contributor II
  • 93 Views
  • 1 replies
  • 0 kudos

How to Copy the notebooks from one environment to another environment

I have one requirement to copy the notebooks from one environment to another environment by one notebook automatically , how can I achieve it.  

  • 93 Views
  • 1 replies
  • 0 kudos
Latest Reply
Isi
Contributor
  • 0 kudos

Hey @databicky ,You can automate the process of copying notebooks from one Databricks environment to another using the Databricks REST API within a notebook. I show you the easiest way I found to do itimport json import requests import base64 # ====...

  • 0 kudos
johschmidt42
by New Contributor II
  • 73 Views
  • 1 replies
  • 0 kudos

Autoloader cloudFiles.maxFilesPerTrigger ignored with .trigger(availableNow=True)?

Hi, I'm using the Auto Loader feature to read streaming data from Delta Lake files and process them in a batch. The trigger is set to availableNow to include all new data from the checkpoint offset but I limit the amount of delta files for the batch ...

  • 73 Views
  • 1 replies
  • 0 kudos
Latest Reply
johschmidt42
New Contributor II
  • 0 kudos

It works when changing "cloudFiles.maxFilesPerTrigger" to "maxFilesPerTrigger". But this is unexpected..

  • 0 kudos
Kabil
by Visitor
  • 38 Views
  • 0 replies
  • 0 kudos

useing dlt metadata as runtime parameter

i have started using DLT pipeline, and i have common code which is used by multiple DLT pipeline. now i need to read metadata information like name of the pipeline and start time of the pipeline during run time, but since im using common code and pip...

  • 38 Views
  • 0 replies
  • 0 kudos
797646
by New Contributor II
  • 370 Views
  • 5 replies
  • 2 kudos

Resolved! Calculated measures not working in Dashboards for queries with big result

Queries with big result are executed on cluster. If we specify calculated measure as something like cal1 ascount(*) / count(distinct field1) it will wrap it in backticks as `count(*) / count(distinct field1) ` as `cal1`functions are not identified in...

  • 370 Views
  • 5 replies
  • 2 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 2 kudos

Hello Team, Could you all try with all caps? COUNT(DISTINCT xxx)

  • 2 kudos
4 More Replies
abelian-grape
by New Contributor III
  • 196 Views
  • 5 replies
  • 0 kudos

Trigger a Databricks Job When there is an insert to a Snowflake Table?

I need to automatically trigger a Databricks job whenever a new row is inserted to a Snowflake table. Additionally, I need the job to receive the exact details of the newly inserted row as parameters.What are the best approaches to achieve this? I’m ...

  • 196 Views
  • 5 replies
  • 0 kudos
Latest Reply
ashraf1395
Valued Contributor III
  • 0 kudos

I think lamba function/ event bridge would be a good way - You can query your snowflake table there and create logic for any new row insert mabe CDC etc and then you send a job trigger using databricks API / databricks SDK where you can pass your new...

  • 0 kudos
4 More Replies
ChristianRRL
by Valued Contributor
  • 127 Views
  • 1 replies
  • 0 kudos

Databricks UMF Best Practice

Hi there, I would like to get some feedback on what are the ideal/suggested ways to get UMF data from our Azure cloud into Databricks. For context, UMF can mean either:User Managed FileUser Maintained FileBasically, a UMF could be something like a si...

Data Engineering
Data ingestion
UMF
User Maintained File
User Managed File
  • 127 Views
  • 1 replies
  • 0 kudos
Latest Reply
ChristianRRL
Valued Contributor
  • 0 kudos

Hi there, checking back in here. Can someone help provide some feedback on my post?+ @NandiniN / @raphaelblg 

  • 0 kudos
YOUKE
by New Contributor II
  • 46 Views
  • 1 replies
  • 0 kudos

Connecting to SQL on Databricks Using SQLAlchemy or pyodbc

On Databricks, when I try to connect to SQL using SQLAlchemy or pyodbc to run delete queries on a specific table, I get this error: (pyodbc.Error) ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not ...

  • 46 Views
  • 1 replies
  • 0 kudos
Latest Reply
MariuszK
Contributor III
  • 0 kudos

This code will not work. You can try this:import osfrom sqlalchemy import create_engineaccess_token = os.getenv("DATABRICKS_TOKEN")server_hostname = os.getenv("DATABRICKS_SERVER_HOSTNAME")http_path = os.getenv("DATABRICKS_HTTP_PATH")catalog = os.gete...

  • 0 kudos
marcelhfm
by Visitor
  • 108 Views
  • 0 replies
  • 0 kudos

Internal GRPC errors when using databricks connect

Hey there, in our local development flow we heavily rely on databricks asset bundles and databricks connect. Recently, locally run workflows (i.e. just pyspark python files) have begun to frequently fail with the following grpc error:pyspark.errors.e...

  • 108 Views
  • 0 replies
  • 0 kudos
jdlogos
by Visitor
  • 57 Views
  • 0 replies
  • 0 kudos

apply_changes_from_snapshot with expectations

Hi,Question: Are expectations supposed to function in conjunction with create_streaming_table() and apply_changes_from_snapshot?Our team is investigating Delta Live Tables and we have a working prototype using Autoloader to ingest some files from a m...

  • 57 Views
  • 0 replies
  • 0 kudos
Trifa
by New Contributor II
  • 635 Views
  • 3 replies
  • 1 kudos

Resolved! Override DLT Fille Refresh using a Job parameter

HelloI have a Job with a DLT pipeline as a first task. From time to time, I want to execute this Job with a Full Refresh of the DLT pipeline. How could I override my default "full_refresh = false" ?This was possible before using the Legacy parameters...

Trifa_0-1701170537015.png
  • 635 Views
  • 3 replies
  • 1 kudos
Latest Reply
adriennn
Valued Contributor
  • 1 kudos

@Trifa luckily, it's simple to implement. You can be the guys are going to release Pipeline Parameters® a week after you have deployed your solution though 

  • 1 kudos
2 More Replies
Kumarn031425
by New Contributor
  • 104 Views
  • 1 replies
  • 0 kudos

Automating Migration of Delta Live Tables Pipelines Across Environments Using Azure DevOps CI/CD

I am seeking guidance on automating the migration of Delta Live Tables (DLT) pipelines across various environments—specifically from development to testing, and ultimately to production—utilizing Azure DevOps for Continuous Integration and Continuous...

  • 104 Views
  • 1 replies
  • 0 kudos
Latest Reply
ashraf1395
Valued Contributor III
  • 0 kudos

Hi there @Kumarn031425 , I guess, This video tutorial will answer most of your questions : https://youtu.be/SZM49lGovTg?si=X7Cwp0Wfqlo1OnuSHere , deployment of workspace resources using databricks azure devops and databeicks asset bundles tutorial is...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels