Databricks

naga_databricks · 08-23-2023

To read Bigquery data using spark.read, i'm using a query. This query executes and creates a table on the materializationDataset. df = spark.read.format("bigquery") \.option("query", query) \.option("materializationProject", materializationProject) \...

naga_databricks · 07-05-2023

I have a notebook where i want to use the workflow name and task name that it will be running under. How do i access these information?

naga_databricks · 06-14-2023

I am trying to insert a record into Delta table using notebook written in python. This record has a timestamp column that should be blank initially, later i have a plan to update the timestamp value.How am i inserting the record: stmt_insert_audit_r...

naga_databricks · 05-09-2023

I specify init scripts in my deployment.conf, as below: basic-static-cluster: &basic-static-cluster new_cluster: spark_version: "13.0.x-scala2.12" num_workers: 1 node_type_id: "n2-highmem-2" init_scripts: - worksp...

naga_databricks

I have seen this problem with Identity column causing concurrency issues. But you seem to be getting similar error when writing to files. I don't know completely know your use case completely here, but would advice retrying this operation by managing...

naga_databricks · 08-25-2023

Thanks Kaniz for the explanation. It sounds good to me that i'm working with tables here.For me the only cause of confusion was the underlined writing:

naga_databricks · 07-11-2023

Are there steps available for GCP?I have a service account on gcp, however adding this gcp service account as a collaborator on Github will keep the service account access as pending, as the invite has to be accepted. Something i might be missing her...

naga_databricks · 07-06-2023

Okay. Thanks.Instead i am now going to use base_parameters on deployment.conf and catch those parameters with dbutils.widget.get()

naga_databricks · 07-05-2023

Thanks. That looks like a promising option to access task level information. Is there a way to access workflow name?

Databricks

User Stats

User Activity

Reading bigquery data using a query

From within a Databricks Notebook, how do you get the workflow and task name

Set timestamp column to blank when inserting a record into delta table

Using Init scripts using DBX

Re: MetadataChangedException Exception in databricks

Re: Reading bigquery data using a query

Re: Service Principal to run Jobs that contain notebooks in Repos (GitHub)

Re: From within a Databricks Notebook, how do you get the workflow and task name

Re: From within a Databricks Notebook, how do you get the workflow and task name