- 931 Views
- 5 replies
- 4 kudos
Centralized Location of Table History/Timestamps in Unity Catalog
Is there a centralized location in Unity Catalog that retains the table history, specifically the last timestamp, for managed delta tables?DESCRIBE HISTORY will provide it for a specific table, but I would like to get it for a number of tables.inform...
- 931 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @Eric_Kieft @noorbasha534 system.access.table_lineage includes a record for each read or write event on a Unity Catalog table or path. This includes but is not limited to job runs, notebook runs, and dashboards updated with the read or write even...
- 4 kudos
- 899 Views
- 1 replies
- 1 kudos
Resolved! Upsert from Databricks to CosmosDB
Hi guys,I'm adjusting a data upsert process from Databricks to CosmosDB using the .jar connector. As the load is very large, do you know if it's possible to change only the fields that have been modified?best regards
- 899 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, you can update only the modified fields in your Cosmos DB documents from Databricks using the Partial Document Update feature (also known as Patch API). This is particularly useful for large documents where sending the entire document for update...
- 1 kudos
- 1222 Views
- 1 replies
- 1 kudos
Resolved! What's the best way to get from Python dict > JSON > PySpark and apply as a mapping to a dataframe?
I'm migrating code from Python Linux to Databricks PySpark. I have many mappings like this: { "main": { "honda": 1.0, "toyota": 2.9, "BMW": 5.77, "Fiat": 4.5, },}I exported using json.dump, saved to s3 and was able to import with sp...
- 1222 Views
- 1 replies
- 1 kudos
- 1 kudos
For migrating your Python dictionary mappings to PySpark, you have several good options. Let's examine the approaches and identify the best solution. Using F.create_map (Your Current Approach) Your current approach using `F.create_map` is actually qu...
- 1 kudos
- 445 Views
- 1 replies
- 0 kudos
Resolved! Workflow job failing with source as Git Provider (remote github repo) with SP
Facing issue using Github App when running job with source as "Git provider" using Service Principle. Since we can't use PAT with SP on github, I am using Github app for authentication. Followed below documentation but still giving permission issue. ...
- 445 Views
- 1 replies
- 0 kudos
- 0 kudos
When running a Databricks workflow with a Git provider source using a Service Principal, you’re encountering permission issues despite using the GitHub App for authentication. This is a common challenge because Service Principals cannot use Personal ...
- 0 kudos
- 1187 Views
- 5 replies
- 0 kudos
Resolved! How to schedule workflow in python script
I saw how to schedule a workflow using UI but python script, can someone help me to find how to schedule workflow hourly in python script ? Thank you.
- 1187 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey @bricks3,Exactly, as far as I know you define the workflow configuration in the YAML file, and under the hood, DABS handles the API calls to Databricks (including scheduling).To run your workflow hourly, you just need to include the schedule bloc...
- 0 kudos
- 1956 Views
- 8 replies
- 6 kudos
CANNOT_UPDATE_TABLE_SCHEMA
I'm encountering a puzzling schema merge issue with my Delta Live Table. My setup involves several master tables on Databricks, and due to a schema change in the source database, one of my Delta Live Tables has a column (e.g., "reference_score") that...
- 1956 Views
- 8 replies
- 6 kudos
- 6 kudos
Dear Hung,Thank you so much for the kind words—I’m really glad the suggestions were helpful! You're absolutely doing the right thing by trying those options first before going for a full table drop. Testing with a new table and checking schema hints ...
- 6 kudos
- 439 Views
- 2 replies
- 0 kudos
How to acknowledge the incidents from the databricks
I want to connect to the service now from the databricks and i want to acknowledge the incidents like assigned to the person and i need to change the status of the incident and i want to update the worknotes as well, how can i achieve this by the hel...
- 439 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @databicky Just a quick clarification: this is the Databricks community forum, not official Databricks Support. Think of it like a place where users can share ideas, help each other, or sometimes just browse, responses are not guaranteed here.If y...
- 0 kudos
- 724 Views
- 1 replies
- 0 kudos
Issue with Delta Lake Table Optimization
Hi Databricks Community, I’m currently working on optimizing a Delta Lake table, but I’m encountering some performance issues during the vacuum process. I’ve attached a screenshot of the error message I’m receiving, along with the relevant code snipp...
- 724 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @whalebone711 Could you re-attach the screenshot? Looks like it doesn't appear in the post.Best,Isi
- 0 kudos
- 3417 Views
- 2 replies
- 2 kudos
Databricks Asset Bundle artifacts with module out of the bundle root (sync path)
Hello everyone!I’m currently working on a project with shared functionalities across different Databricks bundles. I have separate folders for each bundle, along with a common libs/ folder that holds some Python modules intended to be shared across b...
- 3417 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Jorge3, where you able to get this issue resolved? I believe your artifact build path points outside the synced directory structure, and after syncing ../libs, libs should be available within the bundle root, so the artifact path should be update...
- 2 kudos
- 44760 Views
- 26 replies
- 14 kudos
Resolved! How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?
Databricks Repos best-practices recommend using the Repos REST API to update a repo via your git provider. The REST API requires authentication, which can be done one of two ways:A user / personal access tokenA service principal access tokenUsing a u...
- 44760 Views
- 26 replies
- 14 kudos
- 14 kudos
For anyone coming here in the future, this should explain it: https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-repos#authorize-a-service-principal-to-access-git-foldersBasically:1. Go to your service in settings -> identity and access -...
- 14 kudos
- 9885 Views
- 5 replies
- 3 kudos
Resolved! Generating Personal Access Token to service principle databricks cli
Hi I am having issues generating personal access token to my service principle.I followed the steps from here my `~/.databrickscfg` has the following```[my-profile-name]host = <account-console-url>account_id = <account-id>azure_tenant_id = <azure-ser...
- 9885 Views
- 5 replies
- 3 kudos
- 3 kudos
I want something similar, to use a service principal token instead of a PAT, have you ever done this?https://community.databricks.com/t5/administration-architecture/use-a-service-principal-token-instead-of-personal-access-token/m-p/91629
- 3 kudos
- 333 Views
- 1 replies
- 0 kudos
Facing error which submitting lab assessment of Delivery Specialization: UC Upgrade
Hi,I attended the lab assessment of Delivery Specialization: UC Upgrade and while submitting my answers the grade is not getting updated successfully due to which the status is being showed as failed, although all the checkpoints got validated succes...
- 333 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ushnish_18, Can you share the user ID or email you used to access the lab so the team can take a look? Alternatively, you can send all the details, including screenshots, here for the team to investigate in detail and guide you appropriately. Tha...
- 0 kudos
- 518 Views
- 2 replies
- 0 kudos
how to pass secrets keys using a spark_python_task
Hello community,I was searching a way to pass secrets to spark_python_task. Using a notebook file is easy, it's only to use dbutils.secrets.get(...) but how to do the same thing using a spark_python_task set using serveless compute?Kind regards,
- 518 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @jeremy98. To securely access secrets in a spark_python_task with serverless compute in Databricks... first create a secret scope and add secrets (refer to this article). Then, pass secrets by injecting them into environment variables in the job c...
- 0 kudos
- 906 Views
- 1 replies
- 0 kudos
Acess to s3 in aws
My Problem:My Databricks workspace (Serverless Compute) is in AWS Account A, but my S3 bucket is in AWS Account B.Works in Shared Compute because i am manually setting access_key and secret_key.Does NOT work in Serverless Compute
- 906 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @HarryRichard08 ,I came across a similar thread of yours. Were you able to find a resolution for this?
- 0 kudos
- 2302 Views
- 10 replies
- 5 kudos
Performance issue with Spark SQL when working with data from Unity Catalog
We're encountering a performance issue with Spark SQL when working with data from Unity Catalog. Specifically, when I use Spark to read data from a Unity Catalog partition created by DLT and then create a view, the executor retrieval is very slow. ...
- 2302 Views
- 10 replies
- 5 kudos
- 5 kudos
you can read the physical parquet files with spark.read.parquet().The trick is to know which files are the current ones.
- 5 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
