Databricks Community

irfanaziz · 08-10-2022

The databricks notebook failed yesterday due to timestamp format issue. error:"SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '2022-08-10 00:00:14.2760000' in the new parser. You can set spark.s...

irfanaziz · 06-15-2022

So i have two partitions defined for this delta table, One is year('GJHAR') contains year values, and the other is a string column('BUKS') with around 124 unique values. However, there is one problem with the 2nd partition column('BUKS'), The values ...

irfanaziz · 02-08-2022

Continuing the above case, does that mean if i have several like 5 ADF pipelines scheduled regularly at the same time, its better to use an existing cluster as all of the ADF pipelines would share the same cluster and hence the cost will be lower?

irfanaziz · 01-17-2022

One of the source systems generates from time to time a parquet file which is only 220kb in size.But reading it fails."java.io.IOException: Could not read or convert schema for file: 1-2022-00-51-56.parquetCaused by: org.apache.spark.sql.AnalysisExce...

irfanaziz · 01-13-2022

I have observed a very strange behavior with some of our integration pipelines. This week one of the csv files was getting broken when read with read function given below.def ReadCSV(files,schema_struct,header,delimiter,timestampformat,encode="utf8...

irfanaziz · 12-22-2022

You can read any table from MSSQL. You would need to authenticate to the db, so your would need the connection string:def dbProps(): return { "user" : "db-user", "password" : "your password", "driver" : "com.microsoft.sqlserver.jdbc.SQLServerD...

irfanaziz · 12-12-2022

yes.

irfanaziz · 12-12-2022

Is it not the linked service that defines the kind of cluster created or used for any job?So i believe you could control the configuration via the linked service settings.

irfanaziz · 12-01-2022

This worked for me:import pandas as ps def merge_dataframes(left=None, right=None, how='inner', on=None, left_on=None, right_on=None, left_index=False, right_index=False): merging_left = left.copy() merging_left['we...

irfanaziz · 02-28-2022

Since all the pipelines are orchestrated via ADF, so we are using mostly dedicated clusters but the sizes are small. So the idea to run multiple notebooks via single notebook is not an optimal solution in this case. So i think if you have several pi...

Databricks Community

User Stats

User Activity

TimestampFormat issue

How to make a string column with numeric and alphabet values use as partition?

If two Data Factory pipelines are run at the same time or share a window of execution do they share the Databricks spark cluster(if both have the same linked service)? ( job clusters are those that are create on the go, defined in the linked service).

Issue in reading parquet file in pyspark databricks.

What is the difference between passing the schema in the options or using the .schema() function in pyspark for a csv file?

Re: How to import a data table from SQLQuery2 into Databricks notebook

Re: Is there a way to tell if a created job is not compliant against configured cluster policies before it runs?

Re: Is there a way to tell if a created job is not compliant against configured cluster policies before it runs?

Re: Pyspark Pandas column or index name appears to persist after being dropped or removed.

Re: If two Data Factory pipelines are run at the same time or share a window of execution do they share the Databricks spark cluster(if both have the same linked service)? ( job clusters are those that are create on the go, defined in the linked service).