cancel
Showing results for 
Search instead for 
Did you mean: 
Dan_Z
Databricks Employee
Databricks Employee
since ‎08-28-2021
‎11-07-2024

User Stats

  • 60 Posts
  • 15 Solutions
  • 23 Kudos given
  • 61 Kudos received

User Activity

Echoing what we said in Part 1: Test Data Curation, when your team is migrating script code and pipeline code to Databricks, there are three main steps: Look at the original code, understand what it's doingConvert the code to run on Databricks (conve...
Introduction to the SeriesOverview of Part 1: Test Data CurationSolution ApproachRequiredConsiderationsScript ModificationParsing all touched tablesParsing modified tablesAugment ScriptsJob script modificationEdge cases handledRe-use of scriptsDealin...
IntroductionRequirements of a great historical data loadOptionsSolution OverviewTypes of ActivitiesPipeline ParametersPerformanceActivity DetailsCopy activityLoad to tablesValidate tablesOptimize tablesGeneral Considerations Introduction When migrati...
IntroductionLogging in Azure Data Factory and Databricks NotebooksSolution RequirementsProposed SolutionCustom Logging PackageSet UpPrerequisitesStepsUsageADF Activity ParametersDatabricks Notebook WidgetsExample Log Analytics QueryConclusion Introdu...
mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:https://spa...