@Manjunath Shettar​ :

It seems that the issue is related to the fact that the records in the HBase table have not been flushed to HDFS and are still stored in the Memstore. Spark's newAPIHadoopRDD API reads data from the HBase table through HBase's TableInputFormat, which only reads data that has been persisted (flushed) to HDFS.

The reason why the records become visible after a forced flush via HBase shell or a system-triggered flush is that the data is now persisted to HDFS and can be read by TableInputFormat. Similarly, any data written after the initial flush is immediately visible in Spark because it is persisted to HDFS and can be read by TableInputFormat.

The HBase-client's scan API can read data from the Memstore before flush because it directly accesses the HBase region server's Memstore, bypassing the need for data to be persisted to HDFS first.

Regarding the attempt to force HBase Scan API to read from replica-0, it is not clear how the replica-0 configuration is set up in your HBase cluster, and whether it is properly configured and running. If replica-0 is not properly configured, then setting the replicaId to 0 in the Scan object may not have any effect. If replica-0 is properly configured, then it may be necessary to also configure the HBase client to read from replica-0, as described in the HBase documentation.

As for the issue of not being able to use the Spark-HBase connector because the schema is unknown beforehand, one option is to dynamically infer the schema from the first few rows of data using Spark's DataFrame API. Once the schema is inferred, it can be used to read the remaining data from the HBase table. Another option is to manually specify the schema using Spark's StructType API, based on the known structure of the HBase table.