- 2183 Views
- 3 replies
- 1 kudos
Resolved! How to Read and Wrire Data between 2 seperate instance of Databricks
How to Read and Wrire Data between 2 seperate instance of Databricks.I want to have bi-directional data read and write between Databricks A and Databricks B. Both are not in same instance.Please help
- 2183 Views
- 3 replies
- 1 kudos
- 1 kudos
Here are some patterns that you can utilize:1. If the workspaces are in different Databricks Accounts or different Azure regions. The recommended approach is Delta sharing.The simplest, most governed way to let A read B’s tables and B read A’s tables...
- 1 kudos
- 1030 Views
- 6 replies
- 3 kudos
Dynamic value input to a job
How do I pass dynamic value to a databricks job?I created a notebook which will extract the names of files in the catalog, I want to pass these names as parameter to another notebook task in a job.What are the ways I can do this?
- 1030 Views
- 6 replies
- 3 kudos
- 3 kudos
Hello, I think you have posted wrong picture for notebook1. Could you please verify once.I made some changes and its working now. Thanks a lot.
- 3 kudos
- 1365 Views
- 3 replies
- 1 kudos
Resolved! Getting NO_TABLES_IN_PIPELINE error in Lakeflow Declarative Pipelines
Yesterday (10/1) starting around 12 PM EST we starting getting the following error in our Lakeflow Declarative Pipelines (LDP) process. We get this in environments where none of our code has changed. I found some info on the serverless compute abou...
- 1365 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @cpollock Check the “Event log” and “Pipeline logs” in the Databricks UI for any clues.also can you please share screenshot as pasted in window, attachment is not really working and only scanning
- 1 kudos
- 567 Views
- 3 replies
- 1 kudos
Resolved! Dynamic value input to a job in community free edition
How do i pass dynamic value to a databricks job?I created a notebook which will extract the names of files in the catalog, I want to pass these names as parameter to another notebook task in a job.
- 567 Views
- 3 replies
- 1 kudos
- 5370 Views
- 1 replies
- 3 kudos
Resolved! How to send automated emails from Databricks notebooks based on conditions or events?
Hi everyone,I’m currently exploring how to replicate something similar to Alteryx Email Activity within Databricks.Basically, I want to send automated emails to specific users when certain conditions or events occur in a notebook workflow for exampl...
- 5370 Views
- 1 replies
- 3 kudos
- 3 kudos
Hey @Akshay_Petkar ,This is something a lot of people try to do when they move workflows from Alteryx or SSIS into Databricks. There isn’t a direct “Email Activity” node like in Alteryx, but you candefinitely set up automated email notifications in a...
- 3 kudos
- 7358 Views
- 7 replies
- 5 kudos
COPY INTO command can not recognise MAP type value from JSON file
I have a delta table in Databricks with single column of type map<string, string> and I have a data file in JSON format created by Hive 3 for the table with thecolumn of same type. And I want to load data from file to Databricks's table using COPY IN...
- 7358 Views
- 7 replies
- 5 kudos
- 5 kudos
Usefrom_json(to_json({struct column}),{your schema definition})For exampleCOPY INTO {table} FROM (select from_json(to_json()), 'MAP<STRING, STRING>' FROM {path}) ...
- 5 kudos
- 1743 Views
- 3 replies
- 0 kudos
DLT Runtime Values
When my pipeline runs, I have a need to query a table in the pipeline before I actually create another table. I need to know the target catalog and target schema for the query. I figured the notebook might run automatically in the context of the cata...
- 1743 Views
- 3 replies
- 0 kudos
- 0 kudos
can you set up notebook parameters and pass them in the DLT pipeline? https://docs.databricks.com/en/jobs/job-parameters.html
- 0 kudos
- 10138 Views
- 3 replies
- 2 kudos
Resolved! use DeltaLog class in databricks cluster
I need to use DeltaLog class in the code to get the AddFiles dataset. I have to keep the implemented code in a repo and run it in databricks cluster. Some docs say to use org.apache.spark.sql.delta.DeltaLog class, but it seems databricks gets rid of ...
- 10138 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @pokus , You don't need to access via reflection. You can Access DeltaLog with spark._jvm:Unity Catalog and DeltaLake tables expose their metadata and transaction log via the JVM backend. Using spark._jvm, you can interact with DeltaLog Thanks!
- 2 kudos
- 1995 Views
- 3 replies
- 2 kudos
Resolved! Accessing DeltaLog and OptimisticTransaction from PySpark
Hi community,I'm exploring ways to perform low-level, programmatic operations on Delta tables directly from a PySpark environment.The standard delta.tables.DeltaTable Python API is excellent for high-level DML, but it seems to abstract away the core ...
- 1995 Views
- 3 replies
- 2 kudos
- 2 kudos
For accessing the Databricks pre-installed package's use spark._jvm.com.databricks.sql.transaction.tahoe.DeltaLog org.apache.spark.sql.delta.DeltaLog would be the OSS jar's classname.
- 2 kudos
- 3333 Views
- 1 replies
- 1 kudos
Resolved! Unable to load org.apache.spark.sql.delta classes from JVM pyspark
Hello,I’m working on Databricks with a cluster running Runtime 16.4, which includes Spark 3.5.2 and Scala 2.12.For a specific need, I want to implement my own custom way of writing to Delta tables by manually managing Delta transactions from PySpark....
- 3333 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Nasd_, I believe you are trying to use OSS jars on DBR. (Can infer based on class package) org.apache.spark.sql.delta.DeltaLog The error ModuleNotFoundError: No module named 'delta.exceptions.captured'; 'delta.exceptions' is not a package can be...
- 1 kudos
- 2041 Views
- 2 replies
- 0 kudos
Databricks Asset Bundles fails deploy but works on the GUI with same parameters
I'm running into an issue when running databricks bundle deploy when using job clusters.When I run databricks bundle deploy on a new workspace or after destroying previous resources, the deployment fails with the error: Error: cannot update job: At l...
- 2041 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @LeoRickli Are you setting apply_policy_default_values? https://docs.databricks.com/en/administration-guide/clusters/policies.html#:~:text=Default%20values%20don't%20automatically,not%20needed%20for%20fixed%20policies. After you update a polic...
- 0 kudos
- 2301 Views
- 1 replies
- 1 kudos
Resolved! JDBC driver uses Unsafe API, which will be completely deprecated in a future release of Java
Using JDBC driver (2.7.3) in OpenJDK 24 gives the following warning:WARNING: A terminally deprecated method in sun.misc.Unsafe has been called WARNING: sun.misc.Unsafe::arrayBaseOffset has been called by com.databricks.client.jdbc42.internal.apache.a...
- 2301 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @benesq , For JDBC driver 2.7.4 https://www.databricks.com/spark/jdbc-drivers-download should be used with Java Runtime Environment (JRE) 8.0, 11.0 or 21.0. As mentioned in the installation doc "Each machine where you use the Databricks JDBC Dri...
- 1 kudos
- 6352 Views
- 7 replies
- 3 kudos
Resolved! Azure Databricks Unity Catalog - cannot access managed volume in notebook
We have set up Azure Databricks with Unity Catalog (Metastore).Used Managed Identity (Databricks Access Connector) for connection from workspace(s) to ADLS Gen2ADLS Gen2 storage account has Storage Blob Data Contributor and Storage Queue Data Contrib...
- 6352 Views
- 7 replies
- 3 kudos
- 3 kudos
@AlbertWang @VAMSaha22 Since you want private connectivity I assume you have a vnet and a PE associated with the gen2 account. That PE needs to have a sub-resource of type dfs when the storage account is gen2/hierarchical namespace. You might want to...
- 3 kudos
- 2399 Views
- 1 replies
- 0 kudos
Resolved! Parameter "expand_tasks" on List job runs request seams not to be working (databricsk api)
I'm setting it as True, but it doesn't return the cluster_instance info. Here is the function I'm using:def get_job_runs(job_id): """ Fetches job runs for a specific job from Databricks Jobs API. """ headers = { "Authorization...
- 2399 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mildred The way you passed the data for the expand_tasks parameter is wrong: data = { data = { "job_id": job_id, "expand_tasks": "true" } It should not be passed as Python boolean values, but as a string "true" or "false" Once you do that will...
- 0 kudos
- 471 Views
- 1 replies
- 0 kudos
DLT Flow Redeclaration Error After Service Upgrade
Hi, our delta live tables(Lakeflow declarative pipelines) pipeline started failing after the Sep 30 / Oct 1 service upgrade with the following error :AnalysisException: Cannot have multiple queries named `<table_name>_realtime_flow` for `<table_name>...
- 471 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @DiskoSuperStar IT seems you’ve run into a recently enforced change in Databricks DLT/Lakeflow:Multiple flows (append or otherwise) targeting the same table must have unique names. actually it looks correct on your code. Check if your table_info ...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
