- 889 Views
- 2 replies
- 0 kudos
Resolved! Which of my Lakeflow pipelines are using the same gateway?
Does anyone know of a way to see what Lakeflow pipelines are using the same gateway? We have a gateway connected to a SQL Server that serves multiple individual pipelines but I cannot find a way to see what those are. I've tried system tables. Any in...
- 889 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks @szymon_dybczak - this worked perfectly. This is what I ended up throwing together to see what I was looking for:from databricks.sdk import WorkspaceClient url = f"{DATABRICKS_HOST}/api/2.0/pipelines" headers = wc.config.authenticate() wc = ...
- 0 kudos
- 14027 Views
- 8 replies
- 7 kudos
Reading Iceberg table present in S3 from databricks console using spark given none error .
Hi Team , I am facing issue while reading iceberg table from S3 and getting none error when read the data . below steps I followed .Added Iceberg Spark connector library to your Databricks cluster. 2. Cluster Configuration to Enable Iceberg ...
- 14027 Views
- 8 replies
- 7 kudos
- 7 kudos
https://community.databricks.com/t5/technical-blog/secure-migration-of-aws-glue-iceberg-tables-to-databricks-unity/ba-p/148414
- 7 kudos
- 876 Views
- 4 replies
- 1 kudos
Resolved! Table listener
Hello Community,I would like to ask whether it’s possible to define a job that checks for updates in a table at a specified frequency.Here is my use case:Data is uploaded to a table located in Catalog A, Schema B, Table C (a.b.c).I need to transfer t...
- 876 Views
- 4 replies
- 1 kudos
- 1 kudos
Also I have a question about failure handling in case of triggered jobs. Let's say new data has come to the source table and job failed for any reason. If I rerun it or the next batch of data comes to source table, will the data from the failed job b...
- 1 kudos
- 1303 Views
- 1 replies
- 3 kudos
Resolved! Asset Bundles: Extending complex variables
Hi all,In my Asset Bundle I have some setting for a cluster like in the example at Substitutions and variables in Databricks Asset Bundles (section Define a complex variable).Now I want to add some additional attribute when using this variables, like...
- 1303 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Daniel_dlh ,you can try to use YAML anchor, have a look at this example:before: databricks.yml variables: my_cluster: description: "Base cluster configuration" default: spark_version: "15.4.x-scala2.12" node_type_id: "Standa...
- 3 kudos
- 693 Views
- 1 replies
- 1 kudos
Resolved! How to manage shared parameters between different jobs?
I have few jobs that run serverless. Currently there is only one task for each job. These tasks have multiple parameters, some of which are shared between the jobs, some are unique to each jobs.Is it possible to define all the parameters for all the ...
- 693 Views
- 1 replies
- 1 kudos
- 1 kudos
It's weird, I have tried your definition and it works well when I repleace ${value1} and ${value2}. Not sure how you are using those, you should prefix with var. or use some static. values, like I did.
- 1 kudos
- 1115 Views
- 2 replies
- 0 kudos
Servicenow integration with AWS databricks
Hi Databricks Community,We’re running Databricks on AWS and would like to improve operational incident management for production workloads.Is there any official Databricks documentation or recommended approach to integrate with ServiceNow for automat...
- 1115 Views
- 2 replies
- 0 kudos
- 0 kudos
@anshu_roy Hello, Thanks for the reply,Any sample code for rest API (that will fetch the job long along with name and time ) would be helpful.
- 0 kudos
- 13808 Views
- 2 replies
- 0 kudos
- 13808 Views
- 2 replies
- 0 kudos
- 0 kudos
These are action methods that return data -first() : Returns the very first row of the dataframe as a single row.head() : This does the same as first(), returns the first rowhead(n): Returns an array or list of the first n rowstake(n): Similar to hea...
- 0 kudos
- 689 Views
- 1 replies
- 1 kudos
Resolved! new to databricks
Hi this is phani can some one guide me with best resources to learn databricks as a fresherThank you,Phaneendra
- 689 Views
- 1 replies
- 1 kudos
- 1 kudos
Welcome to Databricks community. You should head to https://www.databricks.com/learn and practice them here on the Free edition https://www.databricks.com/learn/free-edition Start with the free, short fundamentals courses to build a strong base: tak...
- 1 kudos
- 2310 Views
- 3 replies
- 1 kudos
ERROR: Could not find a version that satisfies the requirement
I'm using Databricks workflows along with asset bundle to run my pipeline and recently we moved our pipeline from development to acceptance. We are also using JFrog as our package artifactory.Even though I do see the release candidate version under t...
- 2310 Views
- 3 replies
- 1 kudos
- 1 kudos
I do have init script install on the cluster level as well but still getting the same error, any other solutions for this error?And I have not made any changes to my asset bundle it was running well other day all of sudden I got this error.
- 1 kudos
- 987 Views
- 2 replies
- 0 kudos
Resolved! unable to create new catalog in databricks free edition
unable to create new catalog in databricks free edition
- 987 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @AbhiBange ,I had the same issue using UI approach. Open notebook and use SQL to create catalog. It will work CREATE CATALOG your_catalog_name
- 0 kudos
- 7736 Views
- 3 replies
- 4 kudos
Resolved! How do I track databricks cluster users ?
Hi, is there a way to find out/monitor which users has used my cluster, how long and how many times in an azure databricks workspace ?
- 7736 Views
- 3 replies
- 4 kudos
- 4 kudos
Do you have any query we can get the details from system table who all used the cluster in between this window
- 4 kudos
- 995 Views
- 1 replies
- 1 kudos
Resolved! Workspace Base Environment not applied when promoting DAB‑deployed notebooks to QA
Hi Team,I’m trying to understand how Workspace Base Environments interact with serverless compute when using Databricks Asset Bundles (DAB).According to the documentation:Workspace Base Environments are supported only for serverless Python, Python wh...
- 995 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, As you've correctly identified workspace base environments aren't currently supported by DABs as they're a relatively new feature. They are more meant to give a quick base environment to workspace users rather than used to deploy notebooks as jo...
- 1 kudos
- 1425 Views
- 1 replies
- 1 kudos
Resolved! Lakeflow Ingestion Pipeline – Unable to Generate Event Log Table
For the DLT pipeline, we are successfully generating an event log table using the event_log configuration in the pipeline YAML.However, for the Salesforce pipeline, which is an ingestion pipeline (via Lakeflow Connector), we are unable to create or a...
- 1425 Views
- 1 replies
- 1 kudos
- 1 kudos
Please read https://docs.databricks.com/aws/en/ldp/monitor-event-logsLakeflow ingestion pipelines do generate event logs, but they behave differently than classic Spark Declarative Pipelines. ✔ Lakeflow Connect pipelines include event logs as part o...
- 1 kudos
- 8160 Views
- 22 replies
- 18 kudos
Resolved! Copy text from the integrated Web terminal
How do I copy text from the integrated web terminal? The selection goes away as soon as I lift my finger from the mouse.
- 8160 Views
- 22 replies
- 18 kudos
- 18 kudos
An update here. I raised a ticket and found out this is because tmux mode is enabled by default in the web terminal. You can disable the tmux mode by adding `export DISABLE_TMUX=true` to the ~/.bashrc of the compute.When I asked them about the defaul...
- 18 kudos
- 1669 Views
- 2 replies
- 2 kudos
Resolved! Service principle Personal Access Token permissions
Hello,I'm using a generated PAT from Service Principal to access Databricks from other tools. Now I've extended the permissions of the Service Principal. Should I re-generate the PAT? Or is the PAT used only for authentication, and authorisation is d...
- 1669 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks for the great answer @anshu_roy Where can I check the token permissions?
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
