Databricks

Ajay-Pandey

Exciting news for Databricks users! The ability to view job details within the notebook workflow section, particularly for multithreaded jobs, is available now. Instead of manually inspecting each job for failures, this feature enables us to swiftly ...

Ajay-Pandey · 03-24-2024

Hi All, I just wanted to try the new Databricks Workflow private preview features (For Each Task).Can someone please guide me on how we can enable it in our workspace as I am having the same use case in my current project where this feature can help ...

Ajay-Pandey · 03-19-2024

Hi AllI just wanted to create PAT for Databricks Service Principle but getting below code while hitting API or using CLI - Please help me to create PAT for the same.#dataengineering #databricks

Ajay-Pandey · 02-13-2024

Hi Team,Is there any update on the Community Reward Store, as it's been discontinued from the old portal, and we still can't see the new portal for that.Is there any expected date when this will be available for community members?

Ajay-Pandey · 08-19-2023

I am not able to restart our streaming DLT pipeline as it's giving "__tmp_path_dir" number format error, I am using PUB-SUB as a source for streaming.Any solutions will help - #pusub #databricks

Ajay-Pandey

Hi @Brad Instead of passing the query you can read the postgres table and after that you can filter the dataframe with respective column this also use pushdown filter.Ex - remote_table = (spark.read .format("postgresql") .option("dbtable", "schem...

Ajay-Pandey

Hi @DataFarmer I Databricks I will advise you to use date type instead of int, this will make your life much simpler while working on the date type data.

Ajay-Pandey

Hi @Jorge3 Yes, you can use continues mode also.Please find syntax below - resources: jobs: dbx_job: name: continuous_job_name continuous: pause_status: UNPAUSED queue: enabled: true

Ajay-Pandey

Hi @Jorge3 Yes you can use file_arrival triger with DAB.Find below tag for your reference - resources: jobs: FileBasedJob: name: FileBasedJob trigger: pause_status: UNPAUSED file_arrival: url: abfss://test@...

Ajay-Pandey

Hi @ETLdeveloper You can use the multithreading that help you to run notebook in parallel.Attaching code for your reference - from concurrent.futures import ThreadPoolExecutor class NotebookData: def __init__(self, path, timeout, parameters = Non...

Databricks

User Stats

User Activity

Databricks Notebook Workflow

Databricks Private Preview Features

On-behalf-of token creation for service principals is not enabled for this workspace

Update regarding Community Reward Store

Issue while re-starting Streaming DLT pipeline - PUBSUB

Re: Pushdown in Postgres

Re: Data Warehouse in Databricks Date values as date or int: what is recommended?

Re: [Databricks Assets Bundles] Workflow trigger on file arrival

Re: [Databricks Assets Bundles] Workflow trigger on file arrival

Re: I have to run the notebook in concurrently using process pool executor in python