- 272 Views
- 1 replies
- 3 kudos
Resolved! Asset Bundles: Extending complex variables
Hi all,In my Asset Bundle I have some setting for a cluster like in the example at Substitutions and variables in Databricks Asset Bundles (section Define a complex variable).Now I want to add some additional attribute when using this variables, like...
- 272 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Daniel_dlh ,you can try to use YAML anchor, have a look at this example:before: databricks.yml variables: my_cluster: description: "Base cluster configuration" default: spark_version: "15.4.x-scala2.12" node_type_id: "Standa...
- 3 kudos
- 392 Views
- 1 replies
- 1 kudos
Resolved! Cost-Effective Databricks Pipeline for API Ingestion - Best Practices?
Hi Community,As a senior data engineer migrating ETL workloads to Databricks (with Unity Catalog and Delta Lake), I'm building a cost-effective pipeline to ingest data from a REST API. Goals: minimize DBU costs, handle incremental loads, ensure scala...
- 392 Views
- 1 replies
- 1 kudos
- 1 kudos
HI @TheDataMaverick ,The most efficient approach for your REST API ingestion pipeline on Databricks is to use an external service like Azure Functions (or AWS Lambda) to handle API calls, then land raw JSON/Parquet in ADLS/S3 for Auto Loader ingestio...
- 1 kudos
- 183 Views
- 1 replies
- 1 kudos
Resolved! How to manage shared parameters between different jobs?
I have few jobs that run serverless. Currently there is only one task for each job. These tasks have multiple parameters, some of which are shared between the jobs, some are unique to each jobs.Is it possible to define all the parameters for all the ...
- 183 Views
- 1 replies
- 1 kudos
- 1 kudos
It's weird, I have tried your definition and it works well when I repleace ${value1} and ${value2}. Not sure how you are using those, you should prefix with var. or use some static. values, like I did.
- 1 kudos
- 664 Views
- 6 replies
- 4 kudos
Resolved! How to use Row Level Security on a Sync table in Lakebase
Hi,Im trying to test some stuff out with the Databricks Lakebase, i want to connect the Lakebase to our custom frontend application. Using lakebase, since it is postgres based should result in faster query performance. I created a synced table to syn...
- 664 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi @Jeffrey- Short answer: you’re not doing anything wrong—this is expected with Lakebase synced tables. UC row filters/masks don’t apply to Lakebase synced tables yet; RLS must be defined on the Postgres side separately. Synced tables are owned in ...
- 4 kudos
- 260 Views
- 2 replies
- 0 kudos
Servicenow integration with AWS databricks
Hi Databricks Community,We’re running Databricks on AWS and would like to improve operational incident management for production workloads.Is there any official Databricks documentation or recommended approach to integrate with ServiceNow for automat...
- 260 Views
- 2 replies
- 0 kudos
- 0 kudos
@anshu_roy Hello, Thanks for the reply,Any sample code for rest API (that will fetch the job long along with name and time ) would be helpful.
- 0 kudos
- 13125 Views
- 2 replies
- 0 kudos
- 13125 Views
- 2 replies
- 0 kudos
- 0 kudos
These are action methods that return data -first() : Returns the very first row of the dataframe as a single row.head() : This does the same as first(), returns the first rowhead(n): Returns an array or list of the first n rowstake(n): Similar to hea...
- 0 kudos
- 163 Views
- 1 replies
- 1 kudos
Resolved! new to databricks
Hi this is phani can some one guide me with best resources to learn databricks as a fresherThank you,Phaneendra
- 163 Views
- 1 replies
- 1 kudos
- 1 kudos
Welcome to Databricks community. You should head to https://www.databricks.com/learn and practice them here on the Free edition https://www.databricks.com/learn/free-edition Start with the free, short fundamentals courses to build a strong base: tak...
- 1 kudos
- 1547 Views
- 3 replies
- 1 kudos
ERROR: Could not find a version that satisfies the requirement
I'm using Databricks workflows along with asset bundle to run my pipeline and recently we moved our pipeline from development to acceptance. We are also using JFrog as our package artifactory.Even though I do see the release candidate version under t...
- 1547 Views
- 3 replies
- 1 kudos
- 1 kudos
I do have init script install on the cluster level as well but still getting the same error, any other solutions for this error?And I have not made any changes to my asset bundle it was running well other day all of sudden I got this error.
- 1 kudos
- 289 Views
- 2 replies
- 0 kudos
Resolved! unable to create new catalog in databricks free edition
unable to create new catalog in databricks free edition
- 289 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @AbhiBange ,I had the same issue using UI approach. Open notebook and use SQL to create catalog. It will work CREATE CATALOG your_catalog_name
- 0 kudos
- 6873 Views
- 3 replies
- 4 kudos
Resolved! How do I track databricks cluster users ?
Hi, is there a way to find out/monitor which users has used my cluster, how long and how many times in an azure databricks workspace ?
- 6873 Views
- 3 replies
- 4 kudos
- 4 kudos
Do you have any query we can get the details from system table who all used the cluster in between this window
- 4 kudos
- 244 Views
- 1 replies
- 1 kudos
Resolved! Workspace Base Environment not applied when promoting DAB‑deployed notebooks to QA
Hi Team,I’m trying to understand how Workspace Base Environments interact with serverless compute when using Databricks Asset Bundles (DAB).According to the documentation:Workspace Base Environments are supported only for serverless Python, Python wh...
- 244 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, As you've correctly identified workspace base environments aren't currently supported by DABs as they're a relatively new feature. They are more meant to give a quick base environment to workspace users rather than used to deploy notebooks as jo...
- 1 kudos
- 192 Views
- 1 replies
- 1 kudos
Resolved! Lakeflow Ingestion Pipeline – Unable to Generate Event Log Table
For the DLT pipeline, we are successfully generating an event log table using the event_log configuration in the pipeline YAML.However, for the Salesforce pipeline, which is an ingestion pipeline (via Lakeflow Connector), we are unable to create or a...
- 192 Views
- 1 replies
- 1 kudos
- 1 kudos
Please read https://docs.databricks.com/aws/en/ldp/monitor-event-logsLakeflow ingestion pipelines do generate event logs, but they behave differently than classic Spark Declarative Pipelines. ✔ Lakeflow Connect pipelines include event logs as part o...
- 1 kudos
- 3982 Views
- 22 replies
- 18 kudos
Resolved! Copy text from the integrated Web terminal
How do I copy text from the integrated web terminal? The selection goes away as soon as I lift my finger from the mouse.
- 3982 Views
- 22 replies
- 18 kudos
- 18 kudos
An update here. I raised a ticket and found out this is because tmux mode is enabled by default in the web terminal. You can disable the tmux mode by adding `export DISABLE_TMUX=true` to the ~/.bashrc of the compute.When I asked them about the defaul...
- 18 kudos
- 228 Views
- 2 replies
- 2 kudos
Resolved! Service principle Personal Access Token permissions
Hello,I'm using a generated PAT from Service Principal to access Databricks from other tools. Now I've extended the permissions of the Service Principal. Should I re-generate the PAT? Or is the PAT used only for authentication, and authorisation is d...
- 228 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks for the great answer @anshu_roy Where can I check the token permissions?
- 2 kudos
- 210 Views
- 2 replies
- 2 kudos
Resolved! Automate Unity Access management Process at Schema, Tables, Cluster and Job
Hi Team,I’d like to automate the process of providing access to users and groups across:SchemasTablesJobsWorkflowsClustersCurrently, this is a manual, day-to-day process. Our goal is to implement an optimized, reliable solution.Proposed approach (con...
- 210 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @ameet9257 see if this psot can help you : https://www.linkedin.com/posts/ishashankshekhar_databricks-unitycatalog-datagovernance-activity-7334515730937229312-yVPg?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAlFpRcB2e5Ub33KKrsPHNgFLuA5WVk...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 791 | |
| 554 | |
| 349 | |
| 287 |
