Databricks

erigaud

Hello,I have a Job A that runs a Job B, and Job A defines a globalTempView and I would like to somehow access it in the child job. Is that in anyway possible ? Can the same cluster be used for both jobs ? If it is not possible, does someone know of a...

erigaud

Hello community !Here's my problem : I had a workspace with Unity Catalog enabled, and a catalog stored on an external location (Azure storage account). All my schemas and tables were storage on this storage. However we recently deleted the metastore...

erigaud

Hello,As the title says, I am trying to call an function from an Azure Function App configured with access restrictions from a python notebook in my Databricks workspace. The Function App resource is in a different subscription as the Databricks work...

erigaud · 12-13-2023

Hello everyone !I am looking for a way to dynamically specify pivot columns in a SQL query, so it can be used in a view. However we don't want to hard code the values that need to become columns, and would rather extract it from another table.I've se...

erigaud · 11-27-2023

Hello everyone,I have a workflow that starts by reading the CDF data for a change data feed.The syntax is exactly the following : (spark.readStream .format("delta") .option("readChangeFeed", "true") .option("startingVersion", 10) .table("my.str...

erigaud

Hello @Kaniz ,thank you for the very detailed answer. If I understand correctly there is no way to do this using temp views and using a Job Cluster ? I need in the case to use the same All-purpose for all my tasks in order to remain in the same spar...

erigaud

I have an issue when trying to create a new catalog based on the previous catalog location : Any idea ? Thanks Scott !

erigaud

Here's I how would do it, leveraging the medallion architecture : - Instead of a temporary table, load everything in a bronze table in append mode. This table should be liquid-clustered, and you should run OPTIMIZE on it regularly. It should also hav...

erigaud

Hello, I'll complement @Kaniz's answer, which is already very detailed. First of all if you're considering Liquid Clustering, you need to know on which columns you'll cluster your table. These need to be the columns on which you're gonna merge. Then,...

erigaud

Update : Problem was fixed ! The key was to set an VNET rule in the access restriction, giving access directly to the subnets used by Databricks.It seems like for Microsoft to Microsoft connections, the IP addresses are not used, so adding the IP ran...

Databricks

User Stats

User Activity

Pass Dataframe to child job in "Run Job" task

Create a catalog with external location from different Metastore

Call an Azure Function App with Access Restrictions from a Databricks Workspace

Dynamically specify pivot column in SQL

The operation CHANGE DATA FEED is not allowed on Streaming Tables.

Re: Pass Dataframe to child job in "Run Job" task

Re: Create a catalog with external location from different Metastore

Re: Liquid Clustering With Merge

Re: Liquid Clustering With Merge

Re: Call an Azure Function App with Access Restrictions from a Databricks Workspace