- 3164 Views
- 2 replies
- 0 kudos
Resolved! Unable to generate account level PAT for service principle
I am trying to generate PAT for a service principle.I am following the documentation as shown below:https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html#create-token-in-accountI have prepared the below curl command:I am getting below error:Pl...
- 3164 Views
- 2 replies
- 0 kudos
- 0 kudos
I was able to generate the workspace level token using the databricks cli.I set the following details in the databricks cli profile(.databrickscfg) file: host = https://myworksapce.azuredatabricks.net/ account_id = (my db account id)client_id = ...
- 0 kudos
- 7491 Views
- 2 replies
- 1 kudos
[Delta live table vs Workflow]
Hi Community Members,I have been using Databricks for a while, but I have only used Workflow. I have a question about the differences between Delta Live Table and Workflow. Which one should we use in which scenario?Thanks,
- 7491 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, Delta Live Tables focuses on managing data ingestion, transformation, and management of Delta tables using a declarative framework. Job Workflows are designed to orchestrate and schedule various data processing and analysis tasks, including SQL q...
- 1 kudos
- 5174 Views
- 1 replies
- 0 kudos
Enable or disable Databricks Assistant in the Community Edition.
Hello,Good afternoon great people. I was following the step-by-step instructions to enable or disable Databricks Assistant in my Databricks Community Edition to enable the AI assistance. However, I couldn't find the option and was unable to enable it...
- 5174 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you @Retired_mod
- 0 kudos
- 5056 Views
- 0 replies
- 0 kudos
DAB template dbt-sql not working
Hi,We are trying to use the dbt-sql template provided for databricks asset bundles but getting error as follows: Looks like its regarding default catalog configuration. Has anyone faced this previously or can help with the same
- 5056 Views
- 0 replies
- 0 kudos
- 4268 Views
- 2 replies
- 0 kudos
How to overwrite git_source configuration in Asset Bundles
I'm using DAB to deploy a "jobs" resource into Databeicks and into two environments: "dev" and "prod". I pull the notebooks from a remote git repository using "git_resource", and defined the default job to use a tag to find which version to pull. Ho...
- 4268 Views
- 2 replies
- 0 kudos
- 0 kudos
I use target overrides to switch between branch and tags on different environments: resources: jobs: my_job: git_source: git_url: <REPO-URL> git_provider: gitHub targets: staging: resources: jobs: my_j...
- 0 kudos
- 5095 Views
- 1 replies
- 2 kudos
How to collect a thread dump from Databricks Spark UI.
If you observe a hung job, thread dumps are crucial to determine the root cause. Hence, it would be a good idea to collect the thread dumps before cancelling the hung job. Here are the Instructions to collect the Spark driver/executor thread dump: ​...
- 5095 Views
- 1 replies
- 2 kudos
- 2 kudos
Thank you for sharing @NandiniN
- 2 kudos
- 1598 Views
- 0 replies
- 0 kudos
Response code 400 received when using VSCode on Windows 10 but no issue while using Ubuntu
I use VSCode on Windows 10 for building and deploying a workflow from my system and always encounter response code 400 when trying to deploy it. I am able to deploy the workflows via Ubuntu, but not via Windows. Has anyone encountered this issue befo...
- 1598 Views
- 0 replies
- 0 kudos
- 919 Views
- 0 replies
- 0 kudos
jobs.python_wheel_task.enty_point can't find entry points defined in dependency packages
Let's say I have packageA with no entry points, packageA dependents on dependencyA package, which has entry-points.In order to be able to use those entrypoints, i.e.```yamlpython_wheel_task: package_name: packageA entry_point:dependencyA_entry```I ...
- 919 Views
- 0 replies
- 0 kudos
- 2724 Views
- 2 replies
- 0 kudos
child notebook is not displaying output.
I have built pipeline to execute databricks notebook having SQL scripts. It is executing notebook but not able to see output for each cell. I am executing child notebook through driver notebook.
- 2724 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Verr,In short it depends on how your child notebook is configured. But I would start with the following points -Output Logging Settings: Check the logging settings for your notebook cells. Ensure that the cells are configured to display output. In...
- 0 kudos
- 5622 Views
- 6 replies
- 2 kudos
Resolved! column "storage_sub_directory" is now always NULL in system.information_schema.tables
Hello,I am running a job that depends on the information provided in column storage_sub_directory in system.information_schema.tables .... and it worked until 1-2 weeks ago.Now I discovered in the doc that this column is deprecated and always null , ...
- 5622 Views
- 6 replies
- 2 kudos
- 1648 Views
- 1 replies
- 0 kudos
displaying unity catalog metadata and other information in sharePoint
Is there any connectors or api which we can use to display metadata information stored in Unity catalog to business users using SharePoint.
- 1648 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Lucifer As metadata are stored in system schema as table that means you can use databricks to extract the data from databricks and display it to Sharepoint Docs - Statement Execution API: Run SQL on warehouses | Databricks on AWS
- 0 kudos
- 9617 Views
- 2 replies
- 0 kudos
Create Team Git Repo in Azure Databricks
When I attach a Git repo to Databricks, it always puts the repo under my username/domain name:How can I create a "team" repo at the top level, so teammates don't have to drill into my username?
- 9617 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi,Interest of using a repo is to have a dedicated area for each of developers.If you want to have only a folder with the last version of the code, you should a CI/CD pipeline that will package the code and then delivered into a folder inside Workspa...
- 0 kudos
- 8893 Views
- 3 replies
- 1 kudos
Error when setting up OAuth profile for Databricks CLI
Hello.I'm currently trying to migrate a project from dbx to Databricks Asset Bundles. I have successfully created the required profile using U2M authentication with the command```databricks auth login --host <host-name>```I'm able to see the new prof...
- 8893 Views
- 3 replies
- 1 kudos
- 1 kudos
I ran into a similar error just now, and in my case, Pycharm was running some iPython startup scripts each time it opened a console. There was, for some reason, a file at `~/.ipython/profile_default/startup/00-databricks-init-a5acf3baa440a896fa364d18...
- 1 kudos
- 1413 Views
- 1 replies
- 0 kudos
Using Community Edition instance for Customer Academy’s DE Learning Plan
Hi!So I’ve been wondering since I started with the Data Engineering Learning Plan on the Customer Academy, should I go with my Community Edition Databricks, or I should go with creating a premium edition on either a cloud provider or the website.Than...
- 1413 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey, @Kaniz. May I recommend deleting the post if it is not in the right place or changing the forum so I get the proper response?
- 0 kudos
- 1830 Views
- 0 replies
- 0 kudos
DLT to push data instead of a pull
I am relatively new to Databricks, and from my recent experience it appears that every step in a DLT Pipeline, we define each LIVE TABLES (be it streaming or not) to pull data upstream.I have yet to see an implementation where data from upstream woul...
- 1830 Views
- 0 replies
- 0 kudos
