- 2451 Views
- 2 replies
- 0 kudos
How can I pass job parameters to a dbt task?
I have a dbt task that will use dynamic parameters from the job: {"start_time": "{{job.start_time.[timestamp_ms]}}"}My SQL is edited like this:select 1 as idunion allselect null as idunion allselect {start_time} as idThis causes the task to fail. How...
- 2451 Views
- 2 replies
- 0 kudos
- 0 kudos
Also, you need to pass the parameters using the --vars flag like that: dbt run --vars '{"start_time": "{{job.start_time.[timestamp_ms]}}"}' You will need to modify the 3rd dbt command in your job.
- 0 kudos
- 1009 Views
- 2 replies
- 0 kudos
create_feature_table returns error saying database does not exist while it does
Hi, I am new on databricks and I am taking the training course on databricks machine learning: https://www.databricks.com/resources/webinar/azure-databricks-free-training-series-asset4-track/thank-you. When executing the code to create a feature tabl...
- 1009 Views
- 2 replies
- 0 kudos
- 275 Views
- 2 replies
- 1 kudos
Resolved! Are lambda functions considered bad practice?
As the title suggests I have a bunch of lambda functions within my notebooks and I wanted to know if it is considered to be "bad" to have them in there.output_list = json_files.mapPartitions(lambda partition: iter([process_partition(partition)])) \.f...
- 275 Views
- 2 replies
- 1 kudos
- 1 kudos
Using lambda functions within notebooks is not inherently "bad," but there are some considerations to keep in mind. While this code is functional, chaining multiple lambda functions can reduce readability and debugging capabilities in Databricks note...
- 1 kudos
- 212 Views
- 1 replies
- 0 kudos
Is there a way to analyze/monitor WRITE operations in a Notebook
I have user input as a Notebook, which process data and save it to a global temp view. Now I have my caller notebook to execute the input Notebook with dbutils.notebook API. Since the user can do anything in their notebook, I would like to analyze...
- 212 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @lauraxyz, I think you can use the system table and audit logs to achieve that monitoring: https://docs.databricks.com/en/admin/account-settings/audit-logs.html
- 0 kudos
- 1342 Views
- 1 replies
- 0 kudos
Resolved! Databricks Asset Bundle deploy failure
Hello,I have deployed successfully a Databricks Job that contains one task of type DLT using Databricks Asset Bundle.First deployment works well. For this particular Databricks job, I have clicked on "disconnect from source" to do some customization....
- 1342 Views
- 1 replies
- 0 kudos
- 0 kudos
To fix this, you need to set the allow_duplicate_names parameter to true in your deployment configuration. This parameter allows the deployment to proceed even if the pipeline name conflicts with an existing pipeline.
- 0 kudos
- 941 Views
- 1 replies
- 0 kudos
Resolved! not using defined clusters when deploying workflows in development mode by asset bundle
Hi, I'm using databricks asset bundle to deploy workflows.but when i deploy in development mode. the workflows do not use the new clusters, just using existing clusterscan i deploy with the defined new clusters in development mode??
- 941 Views
- 1 replies
- 0 kudos
- 0 kudos
You could use mode: development and then deploy with --compute-id and specify the ID of your personal compute cluster to replace the existing clusters. Only with mode: development will the compute ID replace existing, or per-task cluster specs.
- 0 kudos
- 509 Views
- 5 replies
- 0 kudos
Unable to view hive_metastore schemas although I have the same permissions as co-workers who can
Hello! I'm having trouble accessing the schemas of the hive_metastore. I have the same level of permissions as my fellow coworkers who don't have any trouble viewing the schemas. Please I would really appreciate it if you could help me with this beca...
- 509 Views
- 5 replies
- 0 kudos
- 0 kudos
Where you able to get this issue resolved after looking at the permissions level on your schema and tables?
- 0 kudos
- 465 Views
- 2 replies
- 0 kudos
Limitations When Using Instance Profiles to Connect to Kinesis
I encountered an issue where I couldn’t successfully connect to Kinesis Data Streams using instance profile authentication while working with Delta Live Tables (DLT) in a Unity Catalog (UC)-enabled environment.According to the documentation, instance...
- 465 Views
- 2 replies
- 0 kudos
- 0 kudos
@Alberto_Umana Thank you for sharing. However, I have already followed the steps mentioned in the article, and I’m still unable to establish a connection.When using AWS access keys, the connection is successful, which confirms that there are no issue...
- 0 kudos
- 431 Views
- 4 replies
- 0 kudos
UDF java can't access files in Unity Catalog - Operation not permitted
I am using Databricks on Azure.in pyspark I register UDF java functionspark.udf.registerJavaFunction("foo", "com.foo.Foo", T.StringType())Foo tries to load a file, using Files.readAllLines(), located in the Databricks unity catalog .stderr log:Tue J...
- 431 Views
- 4 replies
- 0 kudos
- 0 kudos
To address the issue of needing to run initialization code that reads file content during the load of a UDF (User Defined Function) in Databricks, you should avoid performing file operations in the constructor due to security restrictions. Instead, y...
- 0 kudos
- 9187 Views
- 14 replies
- 8 kudos
Parameterized spark.sql() not working
Spark 3.4 introduced parameterized SQL queries and Databricks also discussed this new functionality in a recent blog post (https://www.databricks.com/blog/parameterized-queries-pyspark)Problem: I cannot run any of the examples provided in the PySpark...
- 9187 Views
- 14 replies
- 8 kudos
- 8 kudos
option 2 can be done with TEMPORARY LIVE VIEWs (or TEMPORARY STREAMING TABLE) over a unity catalog table, so not "permanent" I guess.> for the gold layer is to save that spark SQL code into .py files for each table and import them in the DLT pipeline...
- 8 kudos
- 972 Views
- 7 replies
- 3 kudos
Migrating logic from Airflow DAGs to Databricks Workflow
Hello community,I'm planning to migrate some logics of Airflow DAGs on Databricks Workflow. But, I was facing out to some doubts that I have in order to migrate (to find the respective) the logic of my actual code from DAGs to Workflow.There are two ...
- 972 Views
- 7 replies
- 3 kudos
- 3 kudos
You can use Asset Bundles https://docs.databricks.com/en/dev-tools/bundles/index.html
- 3 kudos
- 987 Views
- 12 replies
- 5 kudos
Delta sharing service Issue making requests to Unity System Access tables
Hi all, We have been having an issue as of yesterday which I believe is related to queries against the system.access.table_linage in Unity Catalogs. This issue still persists todayWe get the following error:AnalysisException: [RequestId= ErrorClass=B...
- 987 Views
- 12 replies
- 5 kudos
- 582 Views
- 8 replies
- 1 kudos
Databricks single user compute cannot write to storage
I've deployed unrestricted single user compute for each developer in our dev workspace and everything works fine except for writing to storage where the cell will continuously run but seemingly not execute anything. If I switch to an unrestricted sha...
- 582 Views
- 8 replies
- 1 kudos
- 1 kudos
Adding to @saurabh18cs comments, also check if any instance profile attached to the cluster. What is the difference between the clusters, only access mode?
- 1 kudos
- 1856 Views
- 3 replies
- 10 kudos
Resolved! DELTA LIVE TABLES - MATERIALIZED VIEW DOES NOT INCREMENT NOTHING !
I'm very disappointed with this framework. The documentation is inadequate, and it has many limitations. I want to run materialized views with incremental updates, but DLT insists on performing a full recompute. Why is it doing this? Here is the log ...
- 1856 Views
- 3 replies
- 10 kudos
- 10 kudos
I'll try to clear up some misunderstanding surrounding the incremental load feature of materialized views:There isn't a feature to force materialized views to update incrementally. Instead, there is an optimizer called Enzyme that can selectively inc...
- 10 kudos
- 436 Views
- 1 replies
- 0 kudos
Resolved! Cannot create serverless sql warehouse, only classic and pro option available
Hey teamI am using databricks on Azure(East US region) and i have enabled serverless compute in Settings -> Feature Enablement. When i click on create sql workspace, i do not see serverless option.Any setting i am missing ?
- 436 Views
- 1 replies
- 0 kudos
- 0 kudos
I found the root cause for this issue, In Security and Compliance we had PCI-DSS selected and according to this doc we can not have that instead we can select HIPAA
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 347 | |
| 286 | |
| 252 |
