Can i get 2 different type of Source file (csv and Json) in a single build

[Abishek_317176]

Need to understand this 

i got 2 sources which currently writes to the Single S3 bucket in csv and json format daily is it possible to ingest both the file types in a single build to ingest in Databricks, file size <300 mb ?

or Alternately i should create 2 onboarding job from same directly to pick json by one and csv- Is it possible not sure...

or Convert the file type to same format then try ingesting 

thanks Abhisek

[Kaniz]

Hi @Abishek_317176,

Ingest Both File Types in a Single Build:

• Auto Loader: Databricks recommends using Auto Loader for incremental data ingestion from cloud object storage. Auto Loader efficiently processes new data files as they arrive in cloud storage without additional setup. It provides a Structured Streaming source called cloudFiles, which automatically processes new files as they arrive, with the option to process existing files in the same directory.
• If your daily file sizes are less than 300 MB, Auto Loader should be suitable for ingesting both CSV....

Create Two Onboarding Jobs:

• You can create two separate onboarding jobs—one for ingesting JSON files and another for ingesting CSV files. Each job can be configured to pick the relevant file type from the S3 bucket.
• While this approach is feasible, it might lead to more maintenance overhead, especially if the number of sources or file types increases over time.

Convert File Types to a Common Format:

• If you prefer a unified approach, consider converting one of the file types (CSV or JSON) to the other format before ingestion. For example:
  • Convert JSON files to CSV format (or vice versa) using a script or tool.
  • Once both file types are in the same format, you can ingest them together using Auto Loader or other methods.

COPY INTO (SQL):

• Databricks provides the COPY INTO command, which allows SQL users to idempotently and incrementally ingest data from cloud object storage into Delta tables. You can use it in Databricks SQL, notebooks, and Databricks Jobs.
• If you’re dealing with thousands of files, COPY INTO can be a good choice. However, if you expect millions of files over time, Auto Loader is more suitable.

[Kaniz]

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀