cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
NathanE
by • New Contributor II
  • 1985 Views
  • 1 replies
  • 1 kudos

Time travel on views

Hello,At my company, we design an application to analyze data, and we can do so on top of external databases such as Databricks. Our application cache some data in-memory and to avoid synchronization issues with the data on Databricks, we rely heavil...

Data Engineering
Reply
  • 1985 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@NathanE As you said, based on below article it may not support currenlty https://docs.databricks.com/en/sql/user/materialized-views.html, but at the same time looks as Materialized View is built on top of table and It is synchronous operation ( when...

  • 1 kudos
DatabricksNIN
by • New Contributor II
  • 1264 Views
  • 1 replies
  • 0 kudos

Pulling data from Azure Boards (Specifically 'Analytics Views' into databricks

Building upon a previous post/topic from one year ago.. I am looking for best practises/examples on how to pull data  from Azure Boards and specifically from 'Analytics Views' into databricks for analysis.I have succeeded in doing so with 'Work Items...

Data Engineering
Reply
  • 1264 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Building upon a previous post/topic from one year ago.. I am looking for best practises/examples on how to pull data  from Azure Boards and specifically from 'Analytics Views' into databricks for analysis.I have succeeded in doing so with 'Work Items...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
VtotheG
by • New Contributor
  • 5614 Views
  • 0 replies
  • 0 kudos

Problem Visual Studio Plugin with custom modules

We are using the Databricks Visual Studio Plugin to write our python / spark code.We are using the upload file to databricks functionality because our organisation has turned unity catelog off. We are now running into a weird bug with custom modules....

Data Engineering
databricks visual studio plug in
visual studio code
Reply
  • 5614 Views
  • 0 replies
  • 0 kudos
aseufert
by • New Contributor III
  • 5070 Views
  • 7 replies
  • 4 kudos

Dynamic Value References Not Working

I can't get the dynamic value references to work in my jobs. I can use the deprecate references (e.g. job_id) but not the new references (e.g. job.id). As a test, I set a text widget called MyJobID following the example that will receive the dynamic ...

Data Engineering
Workflows
Reply
  • 5070 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
themattmorris
New Contributor III
  • 4 kudos

For what it's worth, it looks like job-level parameters were added with this update as well. I was wondering why I was unable to use those, but those are also working for me now.

  • 4 kudos
6 More Replies
thibault
by • Contributor II
  • 2802 Views
  • 1 replies
  • 0 kudos

Resolved! Import notebook content into a python file

Hi, I have a workflow based on python scripts. How can I import the content of a notebook where a class and functions are defined?I know how to import python files into notebooks, but the other way around doesn't seem as straight forward.

Data Engineering
Reply
  • 2802 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
thibault
Contributor II
  • 0 kudos

Found a solution executing a notebook, using the databricks api to download the notebook content as bytes :1. set environment variables DATABRICKS_HOST and DATABRICKS_TOKEN2. w = WorkspaceClient()with w.workspace.download(notebook_path) as n: note...

  • 0 kudos
pgruetter
by • Contributor
  • 3081 Views
  • 1 replies
  • 1 kudos

Help me understand streaming logic with Delta Tables

Hello allI have a delta table in bronze layer, let's call it BRZ. It contains 25B rows and many duplicates. It has a version 0 and a version 1, nothing else yet. I then create a silver table SLV by running one deduplication batch job. This creates ve...

Data Engineering
Reply
  • 3081 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
pgruetter
Contributor
  • 1 kudos

Thanks for the confirmation. Not sure I see everything as your text gets truncated, but it basically confirms that it should work.Anyway: It looks like the incremental load is working. The problem here is, that we receive late arriving facts that tou...

  • 1 kudos
alexiswl
by • Contributor
  • 9789 Views
  • 3 replies
  • 0 kudos

Resolved! Create a UDF Table Function with DLT in UC

Hello, I am trying to generate a DLT but need to use a UDF Table Function in the process.  This is what I have so far, everything works (without e CREATE OR REFRESH LIVE TABLE wrapper)```sqlCREATE OR REPLACE FUNCTION silver.portal.get_workflows_from_...

Data Engineering
Reply
  • 9789 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
shan_chandra
Databricks Employee
  • 0 kudos

@alexiswl - could you please use CREATE OR REPLACE function instead of CREATE OR REFRESH LIVE table?

  • 0 kudos
2 More Replies
Nathant93
by • New Contributor III
  • 3300 Views
  • 1 replies
  • 0 kudos

Resolved! Date formatting

Does anyone know how to change the format of a date like this Dec 17 2016 8:22PMinto yyyy-MM-dd hh:mm:ss?Thanks

Data Engineering
Reply
  • 3300 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Krishnamatta
New Contributor III
  • 0 kudos

 Convert to timestamp first and then format to stringselect  date_format(to_timestamp('Dec 17 2016 8:22PM', 'MMM dd yyyy h:ma'), "yyyy-MM-dd HH:mm:ss")Here is the documentation for this:https://docs.databricks.com/en/sql/language-manual/sql-ref-datet...

  • 0 kudos
Chris_sh
by • New Contributor II
  • 1116 Views
  • 0 replies
  • 0 kudos

Enhancement Request: DLT: Infer Schema Logic/Merge Logic

Currently when DLT runs it observes NULL values in a column and infers that that column should be a string by default. The next time that table runs numeric values are added and it infers that it is now a numeric column. DLT tries to merge these two ...

Data Engineering
Reply
  • 1116 Views
  • 0 replies
  • 0 kudos
Randy
by • New Contributor III
  • 1348 Views
  • 1 replies
  • 0 kudos

Resolved! Unable to Write Table to Synapse 'x' has a data type that cannot participate in a columnstore index.

We have a process that creates a table in Synapse then attempts to write the Data generated in Databricks to it. We are able to create the table no problem but when we go to copy the data we keep getting an error that the column has a data type that ...

Data Engineering
Reply
  • 1348 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Randy
New Contributor III
  • 0 kudos

Resolved

  • 0 kudos
learnerbricks
by • New Contributor II
  • 5910 Views
  • 4 replies
  • 0 kudos

Unable to save file in DBFS

I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :" No such file or directory: 'No such file or directory: '/dbfs...

Data Engineering
Reply
  • 5910 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
pardosa
New Contributor II
  • 0 kudos

Hi,after some exercise you need to aware folder create in dbutils.fs.mkdirs("/dbfs/tmp/myfolder") it's created in /dbfs/dbfs/tmp/myfolderif you want to access path to_csv("/dbfs/tmp/myfolder/mytest.csv") you should created with this script dbutils.fs...

  • 0 kudos
3 More Replies
knutasm
by • New Contributor III
  • 5778 Views
  • 5 replies
  • 7 kudos

Run Delta Live Tables as service principal

How to run a delta live tables pipeline in production? It uses the owner's (creator's) permissions for writing to tables, and I can't change the owner of a UC-enabled pipeline after creation. I don't want regular users to have write access to prod ta...

Data Engineering
Reply
  • 5778 Views
  • 5 replies
  • 7 kudos
View Replies
Latest Reply
js54123875
New Contributor III
  • 7 kudos

same!

  • 7 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels