- 2554 Views
- 1 replies
- 0 kudos
incremental loads without date column
Hi All,We are facing a situation where our data source is Snowflake, and the data is saved in a storage location(adls) in parquet format. However, the tables or data lack a date column or any incremental column for performing incremental loads to Dat...
- 2554 Views
- 1 replies
- 0 kudos
- 0 kudos
Ideally you would have some change tracking system (cdc f.e.) on the source tables (Streams in the case of Snowflake, Introduction to Streams | Snowflake Documentation).But that is not the case.So I think you approach is ok. You cannot track what is...
- 0 kudos
- 5613 Views
- 2 replies
- 1 kudos
How to Pass Dynamic Parameters (e.g., Current Date) in Databricks Workflow UI?
I'm setting up a job in the Databricks Workflow UI and I want to pass a dynamic parameter, like the current date (run_date), each time the job runs.In Azure Data Factory, I can use expressions like @utcnow() to calculate this at runtime. However, I w...
- 5613 Views
- 2 replies
- 1 kudos
- 1 kudos
As szymon mentioned, dynamic parameter values exist, but the functionality is still far from what Data Factory has to offer.I am pretty sure though that this will be extended.So for the moment I suggest you do the value derivation in data factory, an...
- 1 kudos
- 3466 Views
- 7 replies
- 1 kudos
Databricks bundle
Hey, I am new to Databricks, and I am trying to test the mlops-stack bundle. Within that bundle there is a feature-engineering workflow and I have a problem to make it run. The main problem is the following.the bundle specified the target to be $bund...
- 3466 Views
- 7 replies
- 1 kudos
- 2366 Views
- 1 replies
- 0 kudos
Oracle -> Oracle Golden Gate ->Databricks Delta lake
Hi All,We have a situation where we are collecting data from different Oracle instances.The customer is using Oracle GoldenGate to replicate this data into a storage location.From there, we can use Auto Loader or Delta Live Tables to read Avro files ...
- 2366 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Phani1 ,In my opinion this is really good setup. You have push scenario where Oracle GoldenGate is responsible for delivering data into storage, so you don't have to bother about extraction part. And autoloader is the best choice when it comes t...
- 0 kudos
- 689 Views
- 0 replies
- 0 kudos
Delta Lake to Oracle Essbase
Hi All,How can we connect Databricks Delta Lake to Essbase in OCI? We know that Essbase supports JDBC/ODBC. Is it possible to use Python or PySpark to read from Delta Lake and load the data into Essbase? I think using JDBC/ODBC might affect performan...
- 689 Views
- 0 replies
- 0 kudos
- 1678 Views
- 0 replies
- 0 kudos
Denodo Connection Parameters.
Hi All,We are establishing a connection from Denodo to Databricks. During the development phase, we utilized a personal access token associated with developer account. However, this approach is not considered a best practice for production environm...
- 1678 Views
- 0 replies
- 0 kudos
- 2204 Views
- 2 replies
- 0 kudos
Restrict access of user/entity to hitting only specific Databricks Rest APIs
Hi community,Assume I generate a personal access token for an entity. Post generation, can I restrict the access of the entity to specific REST APIs? In other words, consider this example where once I use generate the token and setup a bearer token b...
- 2204 Views
- 2 replies
- 0 kudos
- 0 kudos
@Surajv You have to rely on access control settings on resources and entities (users or service principals or create some cluster policies), rather than directly restricting the API endpoints at the token level.Note: API access based on fine-grained ...
- 0 kudos
- 2195 Views
- 1 replies
- 2 kudos
Feature Request: GUI: Additional Collapse options
When you're using a very large notebook sometimes it gets frustrating scrolling through all the code blocks. It would be nice to have a few additional options to make this easier. 1) Add a collapse all code cells button to the top.2) Add a collapse a...
- 2195 Views
- 1 replies
- 2 kudos
- 2 kudos
this feature please
- 2 kudos
- 1737 Views
- 1 replies
- 0 kudos
Resolved! Does a queued databricks job incur cost?
Does a queued databricks job incur cost?
- 1737 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @qwerty3 ,No, it does not. When a job is queued (waiting for an available cluster or resources), there is no compute usage, so no charges are incurred for Databricks units (DBUs) or cloud infrastructure (VMs). The queued state is essentially a wai...
- 0 kudos
- 5343 Views
- 11 replies
- 1 kudos
Resolved! databricks Asset Bundle
i have come accross a documentation on asset bundles long back whcih states that when you typedatabricks bundle initit gives us option to choose a project type. But i see the below error when i do that i see the below erroris there a way, i can take ...
- 5343 Views
- 11 replies
- 1 kudos
- 1935 Views
- 1 replies
- 2 kudos
column mask on <tinyint>Y columns gives error
My table breaks when I try to mask a column with a name like `<tinyint>Y` -- Create a table with a masked column> CREATE FUNCTION mask_int_col(col_val INTEGER) RETURN CASE WHEN is_member('HumanResourceDept') THEN col_val ELSE CAST(NULL as INTEGER) EN...
- 1935 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @DW ,I have replicated your scenario and encountered the same error when applying a column mask to a column named 1Y in Databricks SQL.In short, it makes sense simply to follow Databricks documentation and use the SQL naming conventions, so that c...
- 2 kudos
- 2551 Views
- 1 replies
- 0 kudos
Databricks Pyspark filter several columns with similar criteria
I am querying a table from the Databricks Catalog which I have to filter several columns with the same criteria. below is what I have created so far. I have 10 columns that I have filter with a set of criteria from (dx_list1) and another 10 that I ...
- 2551 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @abueno ,As I understand the logic you want to implement is:1. For every pair of columns:First Column (DX_i): Must be in dx_list1Second Column (DX_{i+1}): Must be in dx_list22. The condition for each pair is:col('DX_i').isin(dx_list1) OR col('DX_{...
- 0 kudos
- 3376 Views
- 3 replies
- 1 kudos
How to retrieve Spark Session inside java jar library installed on Cluster
I have a java app in form of jar package. This jar is installed on a Databricks cluster. This jar package reads and writes to few tables in databricks. In order to achieve that, I need SparkSession available in the code. Given that spark session is a...
- 3376 Views
- 3 replies
- 1 kudos
- 2171 Views
- 0 replies
- 0 kudos
Course registration issue
Hi All,I'm trying to register for "Spark Developer course" which is free course but I don't see this course under the course catalogue and if I try to directly access using a link I get 403 forbidden error. Can anyone help me figure out the issue.
- 2171 Views
- 0 replies
- 0 kudos
- 1542 Views
- 2 replies
- 1 kudos
How to delete object permissions using REST API
We have REST API methods to set,update,Get object permissions documented here https://docs.databricks.com/api/workspace/permissionsIs there a way to revoke the permissions which are granted through REST API
- 1542 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you for your response!We have a situation where we assigned CAN_BIND permissions to a group for a service principal using the following:$ vim grant-service-principal.json{"access_control_list": [{"user_name": "{username}","permission_level": "C...
- 1 kudos
