Cont...
Q: Do generated columns in Delta Live Tables include IDENTITY columns?
A: My understanding is that generated columns in Delta Live Tables do not contain IDENTITY columns. Here is more on generated columns in DLT.
Q: We store raw data for each customer in a separate folder on S3. Even though it has some pros, reading data folder by folder for processing data is taking a lot of time. Is it recommended to save data this way, or you suggest a better way? (maybe having one parquet file and append new data to it? or...?)
A: In this case I think you are spot on. So when you are basically trying to analyze the data, query the data, this methodology can speed up results. You can go with this current format or you can store it in delta format because delta is more optimized and it gives you the ability to go back into time and time travel. It also has many other capabilities like z ordering as well as optimization.
Q: I'm a contractor who expects to use Databricks for customers. How should I think about user and data management so that I can smoothly hand off projects to the client once they are completed?
A: You should use Unity Catalog which allows you to govern all your data and all your data assets like notebooks machine learning models. In addition, you can use Delta Sharing, which works hand in hand with Unity Catalog and lets you share model artifacts, notebooks, tables or underlining files with customers - Databricks and non-Databricks.
Q: customer wants to use Databricks and synapse together. Any advice? Can synapse read the delta tables generated from Databricks?
A: You can use Databricks and Synapse together. But I don't see a reason why you would use synapse along with Databricks. Unless there is a very specific requirement because everything that synapse does Databricks can do and more and at a cheaper cost. If you are keen on doing so however, feel free to read this doc for additional details.
Q: Do you have recommended tutorials for building an image classifier based on images on S3 (including perhaps transfer learning or using existing models)? What would the main benefits of Databricks for this application? I've been looking at the SageMaker options which look pretty straightforward, and I'm not sure how deep to dig in on Databricks as an alternative.
A: Great question! We have a host of different type of libraries that you can deploy to build image classifiers. So, whenever you are creating a cluster, just select the ml run times for that cluster and choose the latest long term support run time, which would have a lot of these libraries. You can also install custom scripts on the cluster if you want by using in its scripts.
Q: Do you have any information on the usage of the Delta Live Table? For example, the success stories. This will be useful for us to make a decision on whether or not to use delta live tables.
A: We have a lot of customers that are using Delta Live Tables. I have customers who are basically creating pipelines using Delta Live Tables and for instance we have one of our biggest customers who is using Delta Live Tables to simplify pipeline creation. This gives you data quality expectations as well as makes data engineering very simple for your data engineers and even accessible for your data and analytics people. It also makes streaming and batch processing very easy.
Q: What is the solution to process a big json file like 100 GB file and complex array?
A: Databricks is well equipped to handle large files. I suggest you use a larger cluster like a i3.4xlarge. Also, once you get this data into your storage you should definitely use commands like vacuum as well as optimized to distribute the data and vacuum to remove the older and unused files.
Q: I was a little confused on the use of the "pathGlobFilter" for Auto Loader, could you explain what this filter is used for?
A: These filters basically help you to filter the data. If all raw files are in one directory, you can pick certain files for each how to load a job. If the filter is not specified correctly then it will not pick up that pattern. So basically that means that your data that you do not want might sometimes flow into um into your delta lake as well as it will leading to less performance. Here is a community post with more details.
Q: How to pull from another branch while you are in a different branch,? For example, if I am working on a branch, and master branch contains changes ahead of my branch, and I want to incorporate them in my branch, how to rebase my branch with master branch in databricks?
A: Say for example your colleague is working on line number 10 to 15 but you are working on line number 20 to 25. Then once they push their changes you push their changes to the master branch they get merged and then appear created where you will review and approve those changes.
But if you want that, say for example your colleague has worked on line number 10 to 15 of a code and you want those changes to be reflected in your code as well. You will have to wait for those changes to be approved first merged into the master branch and then start working off it.
Q: How can I use Delta Table together with S3 Glacier (for old dates records)?
A: To retain some of those old records for your reference and all tables, then I suggest that you can push this data into S3 glacier as well. Having said that the retrieval times will still be the Sls that AWS suggests. So if you need to query this data for any reason then you might have to wait for those times
