cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
MartinIsti
by • New Contributor III
  • 1961 Views
  • 1 replies
  • 0 kudos

Python UDF in Unity Catalog - spark.sql error

I'm trying to utilise the option to create UDFs in Unity Catalog. That would be a great way to have functions available in a fairly straightforward manner without e.g. putting the function definitions in an extra notebook that I %run to make them ava...

Data Engineering
function
udf
Reply
  • 1961 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
MartinIsti
New Contributor III
  • 0 kudos

I can see someone has asked a very similar question with the same error message:https://community.databricks.com/t5/data-engineering/unable-to-use-sql-udf/td-p/61957The OP hasn't yet provided sufficient details about his/her function so no proper res...

  • 0 kudos
Kingston
by • New Contributor II
  • 1688 Views
  • 3 replies
  • 0 kudos

Unable to overwrite table to Azure sql db

Hi I have a requirement to read table from azure sql db and update the table in azure databricks with transformations and overwrite updated table to the azure sql db but due to lazy evaluation of pyspark im unable to overwrite the table in azure sql ...

Data Engineering
Reply
  • 1688 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
YuliyanBogdanov
New Contributor III
  • 0 kudos

 Hi @Kingston Make sure that you have the proper permissions on the SQL server for the user you do the authentication through JDBC with, i.e. database reader / database writer. Then your approach can go in two directions, push the data from Databrick...

  • 0 kudos
2 More Replies
SenthilJ
by • New Contributor III
  • 3009 Views
  • 0 replies
  • 1 kudos

Databricks Deep Clone

Hi,I am working on a DR design for Databricks in Azure. The recommendation from Databricks is to use Deep Clone to clone the Unity Catalog tables (within or across catalogs). My design is to ensure that DR is managed across different regions i.e. pri...

Data Engineering
Disaster Recovery
Unity Catalog
Reply
  • 3009 Views
  • 0 replies
  • 1 kudos
Vsleg
by • Contributor
  • 1910 Views
  • 3 replies
  • 0 kudos

Enabling enableChangeDataFeed on Streaming Table created in DLT

Hello, Can I enable Change Data Feed on Streaming Tables? How should I do this? I couldn't find this in the existing documentation https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed .

Data Engineering
Reply
  • 1910 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Vsleg
Contributor
  • 0 kudos

@Retired_mod ?

  • 0 kudos
2 More Replies
Khaled75
by • New Contributor
  • 1343 Views
  • 0 replies
  • 0 kudos

Connect databricks

I discovered recently mlflow managed by Databricks so I'm very new to this and I need some help.Can someone explain for me clearly the steps to do to be able to track my runs into the Databricks API.Here are the steps I followed :1/ Installing Databr...

Capture d’écran 2024-03-30 à 01.20.12.png Capture d’écran 2024-03-31 à 16.54.55.png
Data Engineering
Data
tracking_ui
Reply
  • 1343 Views
  • 0 replies
  • 0 kudos
Debi-Moha
by • New Contributor II
  • 2558 Views
  • 0 replies
  • 0 kudos

Unable to write to S3 bucket from Databricks using boto3

I am unable to write data from Databricks into an S3 bucket. I have set up the permissions both on the bucket policy level, and the user level as well (Put, List, and others are added, have also tried with s3*). Bucket region and workspace region are...

Data Engineering
Reply
  • 2558 Views
  • 0 replies
  • 0 kudos
hossein_kolahdo
by • New Contributor II
  • 2371 Views
  • 2 replies
  • 0 kudos

Accessing data from a legacy hive metastore workspace on a new Unity Catalog workspace

Hello,For the purposes of testing I'm interested in creating a new workspace with Unity Catalog enabled, and from there I'd like to access (external - S3) tables on an existing legacy hive metastore workspace (not UC enabled). The goal is for both wo...

Data Engineering
hivemetastore
unitycatalog
Workspaces
Reply
  • 2371 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
MichTalebzadeh
Valued Contributor
  • 0 kudos

Your aim is to access  external S3 tables from a Unity Catalog workspace without data duplication and keeping data updates synchronized. Configure external location permissions. This ensure that both your Unity Catalog and Hive metastore workspaces h...

  • 0 kudos
1 More Replies
sgupta
by • New Contributor II
  • 2961 Views
  • 2 replies
  • 0 kudos

Select from a dynamic table name returned by databricks function

I have a databricks function that returns a table_nameCREATE OR REPLACE FUNCTION test_func()  RETURNS string  READS SQL DATA  RETURN    'table_name'I want to select from the table that is returned by this function. How can I make it work in SQL, some...

Data Engineering
Reply
  • 2961 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
sgupta
New Contributor II
  • 0 kudos

I looked at this posthttps://stackoverflow.com/questions/77475436/in-databricks-workbook-using-spark-sql-how-to-pass-parameters-thru-sql-udf-func What I want is to replace the static table name with the table name passed as parameter (param_table_nam...

  • 0 kudos
1 More Replies
toolhater
by • New Contributor II
  • 1182 Views
  • 0 replies
  • 0 kudos

Self Bootstrap Failure Community Edition

Trying to start a new compute this morning and I get the "Self Bootstrap Error." I saw some people with similar error but their fixes involved DNS settings. I've haven't made any dns changes and was working fine without any problem last night. I real...

Data Engineering
Reply
  • 1182 Views
  • 0 replies
  • 0 kudos
pSdatabricks
by • New Contributor II
  • 3689 Views
  • 3 replies
  • 0 kudos

Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise

I am trying to evaluate options for Monitoring and Alerting tools like New Relic, Datadog, Grafana with Databricks on Azure . No one supports when reached out to them. I would like to hear from the databricks team on the recommended tool / framework ...

Data Engineering
Reply
  • 3689 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Sruthivika
New Contributor II
  • 0 kudos

I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, makin...

  • 0 kudos
2 More Replies
FlexException
by • New Contributor II
  • 9923 Views
  • 4 replies
  • 1 kudos

Dynamic Number of Tasks in Databricks Workflow

Do Databricks workflows support creating a workflow with a dynamic number of tasks?For example, let's say we have a DAG like this:T1 ->    T2(1) ->             T2(2) ->              .....                 -> T3             T2(n-1) ->             T2(n)...

Data Engineering
Reply
  • 9923 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
tanyeesern
New Contributor II
  • 1 kudos

@FlexException Databricks API supports job creation and execution Task Parameters and Values in Databricks Workflows | by Ryan Chynoweth | MediumOne possibility is after running earlier job, process the output to create a dynamic number of tasks in s...

  • 1 kudos
3 More Replies
superspan
by • New Contributor II
  • 1689 Views
  • 1 replies
  • 0 kudos

How to access Spark UI metrics in an automated way (API)

I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from loca...

Data Engineering
metrics
Reply
  • 1689 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
superspan
New Contributor II
  • 0 kudos

Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.

  • 0 kudos
Geoff123
by • New Contributor III
  • 4803 Views
  • 7 replies
  • 0 kudos

Trouble on Accessing Azure Storage from Databricks (Python)

I used the same accessing method shown in https://community.databricks.com/t5/data-engineering/to-read-data-from-azure-storage/td-p/32230 but kept get the error below.org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient pr...

Data Engineering
Reply
  • 4803 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

Hi,you can find storage account firewall information by accessing resource in azure portal Please mind that if you are using Unity Catalog you should NOT mount Storage Account, you should rather use abstraction of Storage Creadentials and External Lo...

  • 0 kudos
6 More Replies
databrick53
by • New Contributor II
  • 3048 Views
  • 6 replies
  • 0 kudos

can't execute the code

When I was executing the code, I was getting this error:"Notebook detached×Exception when creating execution context: java.net.SocketTimeoutException: Connect Timeout"Can someone help me?

Data Engineering
Reply
  • 3048 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
toolhater
New Contributor II
  • 0 kudos

as of last night 3/27 it looks like it was working again

  • 0 kudos
5 More Replies
Marinagomes
by • New Contributor
  • 1470 Views
  • 0 replies
  • 0 kudos

raise Py4JJavaError while changing data type of a column

HiI'm using Azure databricks 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12). I'm trying to convert 2 columns from string data type to timestamp data type . My date columns are in below format2/18/2021 7:20:12 PMSo I wrote following commandfrom py...

Data Engineering
Reply
  • 1470 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels